Skip to content

Testing¤

This document explains how to run and manage the project's tests. We use pytest as our testing framework.

Running the Tests¤

Run all tests¤

To run all tests in the project, use the following command:

poetry run pytest

Run tests in a specific file or directory¤

You can specify a file or directory to run only its tests:

poetry run pytest ./path/to/test --capture=no

The --capture=no parameter is useful for seeing print outputs during test execution.

Watch mode with pytest-watch¤

pytest does not include a native "watch" mode. To run tests automatically when file changes are detected, use pytest-watch (ptw).

poetry run ptw ./path/to/watch ./path/to/test --capture=no
  • ./path/to/watch: The path to the directory you want to monitor for changes.
  • ./path/to/test: The path to the file or directory containing the tests you want to run.

Run tests with markers (@pytest.mark)¤

If you want to run a specific subset of tests, you can mark them with the @pytest.mark.your_mark decorator.

# test.py
import pytest

@pytest.mark.my_mark
def test_check_something():
    ...

To run only the tests with a specific marker, use the -k parameter:

poetry run ptw ./path/to/watch ./path/to/test --capture=no -k=my_mark

Run tests that not connect to LLM¤

If you want to run tests that NOT connect to the LLM provider, you can use the key "llm_evaluation" to skip these tests:

poetry run pytest --capture=no -m "not llm_evaluation"

Testing without an API key using FakeDriver¤

CrewMaster v2.0.0 provides a FakeDriver implementation of RuntimeDriver that simulates LLM responses without making real API calls. This is ideal for:

  • CI/CD pipelines where no API key is available.
  • Fast unit tests that don't depend on network or external services.
  • Tutorials and how-to guides that should work out of the box.
from crewmaster.execution.runtime import FakeDriver
from crewmaster import execute
from crewmaster.operations.operation import Operation
from crewmaster.agents.context.store import ContextStore

# Use FakeDriver instead of PydanticAIDriver to avoid API calls
fake_driver = FakeDriver()

result = await execute(
    operation=my_operation,
    context_store=context_store,
    default_runtime=fake_driver,
    block_store=block_store,
)

To switch to a real LLM, simply replace FakeDriver with PydanticAIDriver — the rest of the code stays the same.