Skip to content

Testing¤

This document explains how to run and manage the project's tests. We use pytest as our testing framework.

Running the Tests¤

Run all tests¤

To run all tests in the project, use the following command:

poetry run pytest

Run tests in a specific file or directory¤

You can specify a file or directory to run only its tests:

poetry run pytest ./path/to/test --capture=no

The --capture=no parameter is useful for seeing print outputs during test execution.

Watch mode with pytest-watch¤

pytest does not include a native "watch" mode. To run tests automatically when file changes are detected, use pytest-watch (ptw).

poetry run ptw ./path/to/watch ./path/to/test --capture=no
  • ./path/to/watch: The path to the directory you want to monitor for changes.
  • ./path/to/test: The path to the file or directory containing the tests you want to run.

Run tests with markers (@pytest.mark)¤

If you want to run a specific subset of tests, you can mark them with the @pytest.mark.your_mark decorator.

# test.py
import pytest

@pytest.mark.my_mark
def test_check_something():
    ...

To run only the tests with a specific marker, use the -k parameter:

poetry run ptw ./path/to/watch ./path/to/test --capture=no -k=my_mark

Run tests that not connect to LLM¤

If you want to run tests that NOT connect to the LLM provider, you can use the key "llm_evaluation" to skip these tests:

poetry run pytest --capture=no -m "not llm_evaluation"