| MODULE | DESCRIPTION |
|---|---|
datasets |
Dataset loader for CrewMaster v2.0.0 evaluation system. |
datasets_test |
Tests for DatasetLoader protocol and YAMLDatasetStore implementation. |
protocols |
Observer protocols for CrewMaster v2.0.0 evaluation system. |
report |
Evaluation report types for CrewMaster v2.0.0 evaluation system. |
report_test |
Tests for EvaluationReport, CriterionResult, and RowResult. |
reporters |
Reporters for the CrewMaster v2.0.0 evaluation system. |
run |
EvaluationRun orchestrator for CrewMaster v2.0.0 evaluation system. |
run_test |
Tests for EvaluationRun orchestrator. |
score |
|
snapshots |
Snapshot store for CrewMaster v2.0.0 evaluation system. |
snapshots_test |
Tests for SnapshotStore protocol and FileSnapshotStore implementation. |
Dataset loader for CrewMaster v2.0.0 evaluation system.
Provides the Dataset model, DatasetLoader protocol, and YAMLDatasetStore implementation. Datasets are collections of input rows used as evaluation context, supporting pluggable backends (YAML, Langfuse, LangSmith, CSV).
| CLASS | DESCRIPTION |
|---|---|
Dataset |
A collection of input rows for evaluation. |
DatasetLoader |
Protocol for loading evaluation datasets. |
YAMLDatasetStore |
DatasetLoader implementation backed by YAML files on disk. |
A collection of input rows for evaluation.
Each row is a dictionary of context variables that get injected as template variables when executing an evaluation against the dataset. Rows are processed independently, one per evaluation run iteration.
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
Dataset name.
TYPE:
|
rows |
List of context dictionaries, one per evaluation input. |
source |
Source identifier (e.g.,
TYPE:
|
Protocol for loading evaluation datasets.
Implementations are responsible for loading datasets from their storage medium (local YAML files, Langfuse, LangSmith, CSV, etc.) and providing a list of available dataset names.
The default implementation is :class:YAMLDatasetStore, which
reads YAML files from a configured directory.
| METHOD | DESCRIPTION |
|---|---|
load |
Load a dataset from a URI. |
list_datasets |
List all available dataset names. |
Load a dataset from a URI.
| PARAMETER | DESCRIPTION |
|---|---|
|
A dataset URI (e.g.,
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Dataset
|
The loaded Dataset. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If the dataset cannot be found. |
ValueError
|
If the URI format is invalid. |
DatasetLoader implementation backed by YAML files on disk.
Reads dataset files from a configured root directory. Each dataset
is stored as {name}.yaml. The YAML file can be either a mapping
with name and rows keys, or a plain list of row dictionaries.
| PARAMETER | DESCRIPTION |
|---|---|
|
Root directory from which dataset files are resolved.
TYPE:
|
| METHOD | DESCRIPTION |
|---|---|
load |
Load a dataset from a |
list_datasets |
List all available dataset names. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
DATASET_PREFIX |
|
Load a dataset from a dataset:// URI.
The YAML file at {root}/{name}.yaml is parsed. It may be:
- A mapping with
name(optional) androwskeys. - A list of row dictionaries (name defaults to the filename).
| PARAMETER | DESCRIPTION |
|---|---|
|
URI in the form
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Dataset
|
The loaded Dataset. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the URI does not start with |
FileNotFoundError
|
If the file does not exist. |
Tests for DatasetLoader protocol and YAMLDatasetStore implementation.
| CLASS | DESCRIPTION |
|---|---|
TestDataset |
Tests for the Dataset model construction and defaults. |
TestYAMLDatasetStoreLoad |
Tests for YAMLDatasetStore.load(). |
TestYAMLDatasetStoreList |
Tests for YAMLDatasetStore.list_datasets(). |
TestDatasetLoaderProtocol |
Tests verifying YAMLDatasetStore conforms to DatasetLoader protocol. |
Tests for the Dataset model construction and defaults.
| METHOD | DESCRIPTION |
|---|---|
test_construct_from_fields |
Dataset can be constructed with all fields. |
test_default_source_empty |
Dataset source defaults to empty string. |
test_rows_can_contain_complex_values |
Dataset rows can contain nested dicts and lists. |
test_construct_from_fields() -> None
Dataset can be constructed with all fields.
test_default_source_empty() -> None
Dataset source defaults to empty string.
test_rows_can_contain_complex_values() -> None
Dataset rows can contain nested dicts and lists.
Tests for YAMLDatasetStore.load().
| METHOD | DESCRIPTION |
|---|---|
test_load_dict_format |
load() parses YAML mapping with name and rows. |
test_load_list_format |
load() handles YAML files that are plain lists of rows. |
test_load_without_name_field |
load() uses the filename as name when YAML has no name key. |
test_load_empty_dataset |
load() returns Dataset with zero rows for empty YAML. |
test_load_raises_for_invalid_uri_prefix |
load() raises ValueError when URI doesn't have dataset:// prefix. |
test_load_raises_for_missing_file |
load() raises FileNotFoundError for nonexistent datasets. |
test_load_handles_non_dict_non_list |
load() handles scalar YAML gracefully with empty rows. |
test_load_dict_format(tmp_path) -> None
load() parses YAML mapping with name and rows.
test_load_list_format(tmp_path) -> None
load() handles YAML files that are plain lists of rows.
test_load_without_name_field(tmp_path) -> None
load() uses the filename as name when YAML has no name key.
test_load_empty_dataset(tmp_path) -> None
load() returns Dataset with zero rows for empty YAML.
test_load_raises_for_invalid_uri_prefix(tmp_path) -> None
load() raises ValueError when URI doesn't have dataset:// prefix.
test_load_raises_for_missing_file(tmp_path) -> None
load() raises FileNotFoundError for nonexistent datasets.
test_load_handles_non_dict_non_list(tmp_path) -> None
load() handles scalar YAML gracefully with empty rows.
Tests for YAMLDatasetStore.list_datasets().
| METHOD | DESCRIPTION |
|---|---|
test_list_datasets_returns_names |
list_datasets() returns all YAML file stems in root. |
test_list_datasets_empty_directory |
list_datasets() returns empty list when no YAML files exist. |
test_list_datasets_returns_sorted |
list_datasets() returns names in alphabetical order. |
test_list_datasets_returns_names(tmp_path) -> None
list_datasets() returns all YAML file stems in root.
test_list_datasets_empty_directory(tmp_path) -> None
list_datasets() returns empty list when no YAML files exist.
test_list_datasets_returns_sorted(tmp_path) -> None
list_datasets() returns names in alphabetical order.
Tests verifying YAMLDatasetStore conforms to DatasetLoader protocol.
| METHOD | DESCRIPTION |
|---|---|
test_instance_matches_protocol |
YAMLDatasetStore is structurally compatible with DatasetLoader. |
test_protocol_is_runtime_checkable |
DatasetLoader can be checked with isinstance at runtime. |
Observer protocols for CrewMaster v2.0.0 evaluation system.
Defines two segregated observer interfaces following the Interface Segregation Principle:
ExecutionObserver: hooks into the DAG execution lifecycle. Useful for LLM call tracing, logging, and metrics.EvaluationObserver: hooks into the evaluation run lifecycle. Useful for progress reporting, score submission to external services, and final report handling.
Both protocols are @runtime_checkable so that isinstance
checks work structurally. All methods are async and provide default
no-op implementations so that implementors only override what they
need.
| CLASS | DESCRIPTION |
|---|---|
ExecutionObserver |
Observer for the DAG execution lifecycle. |
EvaluationObserver |
Observer for the evaluation run lifecycle. |
Observer for the DAG execution lifecycle.
Hooks are called at key points during :func:crewmaster.api.execute.
Implementors can use these hooks for logging, tracing LLM calls,
collecting metrics, or streaming progress.
All methods are optional — the default implementations are no-ops.
Usage::
class MyTracer:
async def on_node_start(self, node: Any) -> None:
print(f"Starting {node.operation_name}")
async def on_node_complete(self, node: Any, output: Any) -> None:
print(f"Completed {node.operation_name}")
observer: ExecutionObserver = MyTracer()
assert isinstance(observer, ExecutionObserver)
| METHOD | DESCRIPTION |
|---|---|
on_plan_start |
Called when execution of a plan begins. |
on_node_start |
Called before a node begins execution. |
on_node_complete |
Called after a node completes execution successfully. |
on_plan_complete |
Called when execution of a plan finishes successfully. |
Observer for the evaluation run lifecycle.
Hooks are called at key points during an :class:EvaluationRun.
Implementors can use these hooks for progress reporting, score
submission to external services (e.g. Langfuse), or final report
formatting.
All methods are optional — the default implementations are no-ops.
Usage::
class MyReporter:
async def on_criterion_complete(
self, row_index: int, criterion_name: str, score: Any
) -> None:
print(f"[{row_index}] {criterion_name}: {score.points}%")
async def on_run_complete(self, report: Any) -> None:
print(f"Global score: {report.global_score}%")
observer: EvaluationObserver = MyReporter()
assert isinstance(observer, EvaluationObserver)
| METHOD | DESCRIPTION |
|---|---|
on_run_start |
Called when an evaluation run begins. |
on_criterion_complete |
Called when a single criterion evaluation completes. |
on_row_complete |
Called when all criteria for a single dataset row are complete. |
on_run_complete |
Called when the full evaluation run completes. |
Evaluation report types for CrewMaster v2.0.0 evaluation system.
Defines RowResult, CriterionResult, and EvaluationReport — the typed output of an EvaluationRun. The report maintains two pivot views of the same data:
- By criterion (CriterionResult): averages scores for each criterion across all dataset rows. Answers "how well did the operation do on coherence?"
- By row (RowResult): all criterion scores for a single dataset input. Answers "for this particular input, how did all the criteria fare?"
The global_score is the average of all criterion averages.
There is no weighting.
| CLASS | DESCRIPTION |
|---|---|
RowResult |
Evaluation results for a single dataset row. |
CriterionResult |
Aggregated results for a single criterion across all rows. |
EvaluationReport |
Final evaluation report with two pivot views. |
Evaluation results for a single dataset row.
Contains all criterion scores for one input from the dataset.
| ATTRIBUTE | DESCRIPTION |
|---|---|
index |
Zero-based row index in the dataset.
TYPE:
|
context |
The input context dictionary injected for this row. |
output |
The raw output produced by the operation for this row.
TYPE:
|
scores |
Mapping from criterion name to its |
average |
Average of all criterion scores for this row (computed).
TYPE:
|
Final evaluation report with two pivot views.
The report is the typed result of an :class:EvaluationRun. It
provides two complementary pivot views of the same data and a
single global_score (average of criterion averages).
Use :meth:from_results to build a report from raw run data.
| ATTRIBUTE | DESCRIPTION |
|---|---|
snapshot |
Name of the evaluated snapshot.
TYPE:
|
dataset |
Name of the dataset used.
TYPE:
|
criteria |
Pivot by criterion — averages across rows.
TYPE:
|
rows |
Pivot by row — all criteria for one input. |
global_score |
Average of criterion averages (computed).
TYPE:
|
completed_rows |
Number of rows that completed without errors.
TYPE:
|
total_rows |
Total number of dataset rows processed.
TYPE:
|
errors |
List of error dicts, one per failed row. |
| METHOD | DESCRIPTION |
|---|---|
from_results |
Build an EvaluationReport from raw row results. |
property
¤
global_score: int
Average of all criterion averages.
Returns 0 when there are no criteria.
classmethod
¤
from_results(snapshot_name: str, dataset_name: str, rows_data: list[dict[str, Any]]) -> EvaluationReport
Build an EvaluationReport from raw row results.
This is the canonical constructor. It takes per-row data
and builds both pivot views (criteria and rows), counts
completed vs. errored rows, and computes global_score.
Each element in rows_data must contain:
index: zero-based row index (int)context: input context dictoutput: the operation output for this rowscores:dict[str, Score]mapping criterion names to their :class:Scoreinstances
Rows where any score is a :class:ScoreError are counted
as errors and excluded from completed_rows.
| PARAMETER | DESCRIPTION |
|---|---|
|
Name of the evaluated snapshot.
TYPE:
|
|
Name of the dataset used.
TYPE:
|
|
Per-row result dicts as described above. |
| RETURNS | DESCRIPTION |
|---|---|
EvaluationReport
|
A fully-populated :class: |
Tests for EvaluationReport, CriterionResult, and RowResult.
| CLASS | DESCRIPTION |
|---|---|
TestRowResult |
Tests for RowResult model. |
TestCriterionResult |
Tests for CriterionResult model. |
TestEvaluationReport |
Tests for EvaluationReport model and from_results factory. |
Tests for RowResult model.
| METHOD | DESCRIPTION |
|---|---|
test_average_single_score |
RowResult.average returns the score when there is only one. |
test_average_multiple_scores |
RowResult.average is the mean of all scores. |
test_average_no_scores |
RowResult.average returns 0 when there are no scores. |
test_average_with_error_score |
ScoreError.points=0, so it contributes 0 to the average. |
test_average_single_score()
RowResult.average returns the score when there is only one.
test_average_multiple_scores()
RowResult.average is the mean of all scores.
test_average_no_scores()
RowResult.average returns 0 when there are no scores.
test_average_with_error_score()
ScoreError.points=0, so it contributes 0 to the average.
Tests for CriterionResult model.
| METHOD | DESCRIPTION |
|---|---|
test_average_single_score |
CriterionResult.average with one score returns that score. |
test_average_multiple_scores |
CriterionResult.average is the mean of its scores list. |
test_average_no_scores |
CriterionResult.average returns 0 when scores list is empty. |
test_average_mixed_scores |
CriterionResult with mixed values. |
test_name_preserved |
CriterionResult preserves the criterion name. |
test_average_single_score()
CriterionResult.average with one score returns that score.
test_average_multiple_scores()
CriterionResult.average is the mean of its scores list.
test_average_no_scores()
CriterionResult.average returns 0 when scores list is empty.
Tests for EvaluationReport model and from_results factory.
| METHOD | DESCRIPTION |
|---|---|
test_global_score_with_multiple_criteria_and_rows |
global_score is the average of all criterion averages. |
test_global_score_with_no_criteria |
Report with no criteria has global_score of 0. |
test_single_row_single_criterion |
Simplest possible report: one row, one criterion. |
test_rows_with_score_error_do_not_break_aggregation |
Rows with ScoreError contribute 0 points but don't break averages. |
test_all_rows_errored |
When all rows have ScoreError, completed_rows is 0. |
test_context_and_output_preserved_in_rows |
RowResult preserves context and output from row data. |
test_integer_division_truncation |
Averages use integer division (truncation, not rounding). |
test_disparate_criteria_across_rows |
Different rows may have different criterion sets. |
test_global_score_with_multiple_criteria_and_rows()
global_score is the average of all criterion averages.
3 rows, 2 criteria.
Row 0: coherence=80, specificity=60 Row 1: coherence=90, specificity=50 Row 2: coherence=70, specificity=40
coherence average: (80+90+70)/3 = 80 specificity average: (60+50+40)/3 = 50 global_score: (80+50)/2 = 65
test_global_score_with_no_criteria()
Report with no criteria has global_score of 0.
test_single_row_single_criterion()
Simplest possible report: one row, one criterion.
test_rows_with_score_error_do_not_break_aggregation()
Rows with ScoreError contribute 0 points but don't break averages.
Row 0: coherence=100, specificity=80 → completed Row 1: coherence=ScoreError, specificity=ScoreError → errored Row 2: coherence=80, specificity=ScoreError → errored (one is error)
coherence average: (100 + 0 + 0) / 3 = 33 specificity average: (80 + 0 + 0) / 3 = 26 global_score: (33 + 26) / 2 = 29
test_context_and_output_preserved_in_rows()
RowResult preserves context and output from row data.
test_integer_division_truncation()
Averages use integer division (truncation, not rounding).
test_disparate_criteria_across_rows()
Different rows may have different criterion sets.
This shouldn't happen in normal evaluation runs (all rows use the same evaluator operations), but the report should handle it gracefully.
Reporters for the CrewMaster v2.0.0 evaluation system.
Provides observer implementations for visibility into execution and evaluation runs:
ReporterConsole: structured logging with per-criterion progress.ReporterNull: no-op observer for tests and headless environments.ReporterLangfuse: Langfuse integration for tracing and dataset scores.
| MODULE | DESCRIPTION |
|---|---|
console |
ReporterConsole — structured logging observer. |
console_test |
Tests for ReporterConsole. |
langfuse |
ReporterLangfuse — Langfuse observer for execution and evaluation. |
langfuse_test |
Tests for ReporterLangfuse. |
null |
ReporterNull — no-op observer for both execution and evaluation. |
null_test |
Tests for ReporterNull. |
| CLASS | DESCRIPTION |
|---|---|
ReporterConsole |
Structured-console reporter for execution and evaluation runs. |
ReporterLangfuse |
Langfuse-based reporter implementing both observer protocols. |
ReporterNull |
No-op reporter that implements both observer protocols. |
ReporterConsole(total_rows: int | None = None, logger: BoundLogger | None = None)
Structured-console reporter for execution and evaluation runs.
Uses :mod:structlog for all output. The logger is configurable
via the constructor, defaulting to "crewmaster.eval.console".
| ATTRIBUTE | DESCRIPTION |
|---|---|
total_rows |
Total number of dataset rows (used to format
progress indicators). If not set explicitly,
the reporter will attempt to extract it from
the run object in
TYPE:
|
logger |
The structlog :class:
TYPE:
|
Usage::
reporter = ReporterConsole(total_rows=3)
await reporter.on_criterion_complete(0, "coherence", score)
# → [1/3] coherence ... 78%
| PARAMETER | DESCRIPTION |
|---|---|
|
Total number of dataset rows for progress
formatting. If
TYPE:
|
|
A structlog logger instance. Defaults to
TYPE:
|
| METHOD | DESCRIPTION |
|---|---|
on_plan_start |
Log the start of a plan execution. |
on_node_start |
Log the start of a node execution. |
on_node_complete |
Log the completion of a node execution. |
on_plan_complete |
Log the completion of a plan execution. |
on_run_start |
Log the start of an evaluation run. |
on_criterion_complete |
Log criterion completion with progress indicator. |
on_row_complete |
Log row completion with average score. |
on_run_complete |
Log the final evaluation report. |
async
¤
Log the completion of a node execution.
async
¤
Log the completion of a plan execution.
async
¤
on_run_start(run: Any) -> None
Log the start of an evaluation run.
If total_rows was not set explicitly, attempts to extract
it from the run's dataset.
async
¤
async
¤
ReporterLangfuse(*, secret_key: str, public_key: str, base_url: str = 'https://cloud.langfuse.com', enabled: bool = True)
Langfuse-based reporter implementing both observer protocols.
Uses the official langfuse client for:
- Tracing of plan/node execution (
ExecutionObserverhooks). - Sending evaluation scores to Langfuse (
EvaluationObserverhooks), including dataset run association.
When enabled=False all hooks are no-ops, which is useful in CI
environments where no Langfuse instance is available.
| ATTRIBUTE | DESCRIPTION |
|---|---|
enabled |
Whether to actually send data to Langfuse.
TYPE:
|
_client |
The underlying
TYPE:
|
| PARAMETER | DESCRIPTION |
|---|---|
|
Langfuse secret key (project-level).
TYPE:
|
|
Langfuse public key (project-level).
TYPE:
|
|
Langfuse API base URL. Defaults to the public cloud instance.
TYPE:
|
|
When
TYPE:
|
| PARAMETER | DESCRIPTION |
|---|---|
|
Langfuse secret key (project-level).
TYPE:
|
|
Langfuse public key (project-level).
TYPE:
|
|
Langfuse API base URL.
TYPE:
|
|
When
TYPE:
|
| METHOD | DESCRIPTION |
|---|---|
on_plan_start |
Create a root Langfuse trace for the plan execution. |
on_node_start |
Create a child span for a plan node. |
on_node_complete |
End the span for a completed node. |
on_plan_complete |
End the root trace for the plan execution. |
on_run_start |
Initialize dataset run context in Langfuse. |
on_criterion_complete |
Send a single criterion score to Langfuse. |
on_row_complete |
Log row completion (no Langfuse action). |
on_run_complete |
Finalize the dataset run and flush all pending data. |
class-attribute
instance-attribute
¤
__slots__ = ('enabled', '_client', '_root_span', '_active_spans', '_dataset_run_id', '_dataset_name', '_score_batch')
async
¤
async
¤
async
¤
End the span for a completed node.
Updates the span with output metadata and ends it. If the node is not tracked (e.g. started before the reporter was attached), this is silently ignored.
| PARAMETER | DESCRIPTION |
|---|---|
|
The :class:
TYPE:
|
|
The structured output artifact produced.
TYPE:
|
async
¤
End the root trace for the plan execution.
Flushes all pending spans and ends the root observation. Any remaining active spans are also ended to prevent orphaned traces.
| PARAMETER | DESCRIPTION |
|---|---|
|
The :class:
TYPE:
|
|
The final output of the last node.
TYPE:
|
async
¤
async
¤
Send a single criterion score to Langfuse.
Scores are sent as NUMERIC scores using
:meth:langfuse.Langfuse.create_score. Each score includes
the criterion name and the points value, plus any explanation
from the score object.
| PARAMETER | DESCRIPTION |
|---|---|
|
Zero-based dataset row index.
TYPE:
|
|
The evaluator operation name.
TYPE:
|
|
The :class:
TYPE:
|
async
¤
Log row completion (no Langfuse action).
The per-criterion scores are already sent individually in
on_criterion_complete. This hook is a no-op for Langfuse
but is included for protocol compliance.
| PARAMETER | DESCRIPTION |
|---|---|
|
Zero-based row index.
TYPE:
|
|
Dict mapping criterion names to :class: |
No-op reporter that implements both observer protocols.
All methods inherit their default no-op implementations from the
protocols. This class exists so that callers can instantiate a
concrete object that satisfies isinstance checks for both
ExecutionObserver and EvaluationObserver.
Usage::
reporter = ReporterNull()
assert isinstance(reporter, ExecutionObserver)
assert isinstance(reporter, EvaluationObserver)
await reporter.on_run_start(run) # does nothing
| METHOD | DESCRIPTION |
|---|---|
on_run_start |
Called when an evaluation run begins. |
on_criterion_complete |
Called when a single criterion evaluation completes. |
on_row_complete |
Called when all criteria for a single dataset row are complete. |
on_run_complete |
Called when the full evaluation run completes. |
on_plan_start |
Called when execution of a plan begins. |
on_node_start |
Called before a node begins execution. |
on_node_complete |
Called after a node completes execution successfully. |
on_plan_complete |
Called when execution of a plan finishes successfully. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
__slots__ |
|
ReporterConsole — structured logging observer.
Implements both :class:ExecutionObserver and :class:EvaluationObserver
using structlog for structured output. Designed for development and
debugging workflows.
Progress format per criterion::
[1/3] coherence ... 78%
Row completion format::
[1/3] row average ... 75%
| CLASS | DESCRIPTION |
|---|---|
ReporterConsole |
Structured-console reporter for execution and evaluation runs. |
ReporterConsole(total_rows: int | None = None, logger: BoundLogger | None = None)
Structured-console reporter for execution and evaluation runs.
Uses :mod:structlog for all output. The logger is configurable
via the constructor, defaulting to "crewmaster.eval.console".
| ATTRIBUTE | DESCRIPTION |
|---|---|
total_rows |
Total number of dataset rows (used to format
progress indicators). If not set explicitly,
the reporter will attempt to extract it from
the run object in
TYPE:
|
logger |
The structlog :class:
TYPE:
|
Usage::
reporter = ReporterConsole(total_rows=3)
await reporter.on_criterion_complete(0, "coherence", score)
# → [1/3] coherence ... 78%
| PARAMETER | DESCRIPTION |
|---|---|
|
Total number of dataset rows for progress
formatting. If
TYPE:
|
|
A structlog logger instance. Defaults to
TYPE:
|
| METHOD | DESCRIPTION |
|---|---|
on_plan_start |
Log the start of a plan execution. |
on_node_start |
Log the start of a node execution. |
on_node_complete |
Log the completion of a node execution. |
on_plan_complete |
Log the completion of a plan execution. |
on_run_start |
Log the start of an evaluation run. |
on_criterion_complete |
Log criterion completion with progress indicator. |
on_row_complete |
Log row completion with average score. |
on_run_complete |
Log the final evaluation report. |
async
¤
Log the completion of a node execution.
async
¤
Log the completion of a plan execution.
async
¤
on_run_start(run: Any) -> None
Log the start of an evaluation run.
If total_rows was not set explicitly, attempts to extract
it from the run's dataset.
async
¤
async
¤
Tests for ReporterConsole.
| CLASS | DESCRIPTION |
|---|---|
TestReporterConsoleProtocolCompliance |
Verify ReporterConsole satisfies both observer protocols. |
TestReporterConsoleCriterionComplete |
Tests for on_criterion_complete output format. |
TestReporterConsoleRowComplete |
Tests for on_row_complete output format. |
TestReporterConsoleRunComplete |
Tests for on_run_complete output format. |
TestReporterConsoleExecutionObserver |
Tests for ExecutionObserver hooks. |
Verify ReporterConsole satisfies both observer protocols.
| METHOD | DESCRIPTION |
|---|---|
test_is_execution_observer |
ReporterConsole passes isinstance check for ExecutionObserver. |
test_is_evaluation_observer |
ReporterConsole passes isinstance check for EvaluationObserver. |
Tests for on_criterion_complete output format.
| METHOD | DESCRIPTION |
|---|---|
test_format_with_total_rows |
Log contains '[row/total] criterion_name ... points%'. |
test_row_display_is_one_based |
Row index 0 displays as [1/N], index 2 displays as [3/N]. |
test_zero_points |
Score with 0 points is displayed as 0%. |
test_total_rows_unknown_fallback |
When total_rows is None, '?' is used as placeholder. |
test_logger_name_in_event_dict |
The logger name is included in the event dict as a key. |
async
¤
test_format_with_total_rows()
Log contains '[row/total] criterion_name ... points%'.
async
¤
test_row_display_is_one_based()
Row index 0 displays as [1/N], index 2 displays as [3/N].
async
¤
test_total_rows_unknown_fallback()
When total_rows is None, '?' is used as placeholder.
async
¤
test_logger_name_in_event_dict()
The logger name is included in the event dict as a key.
Tests for on_row_complete output format.
| METHOD | DESCRIPTION |
|---|---|
test_format_with_average |
Log contains '[row/total] row average ... avg%'. |
test_single_criterion |
Average of a single score is that score. |
test_empty_scores |
Empty dict of scores shows average of 0. |
test_with_error_score |
ScoreError.points=0, contributes 0 to average. |
Tests for on_run_complete output format.
| METHOD | DESCRIPTION |
|---|---|
test_formats_report_with_global_score |
on_run_complete prints the report summary with global score. |
test_multiple_criteria_in_report |
Report with multiple criteria logs each one separately. |
Tests for ExecutionObserver hooks.
| METHOD | DESCRIPTION |
|---|---|
test_logs_plan_start |
on_plan_start logs the plan name. |
test_logs_node_start |
on_node_start logs the node operation name. |
test_logs_node_complete |
on_node_complete logs the node operation name. |
test_logs_plan_complete |
on_plan_complete logs the plan name. |
test_node_without_name_falls_back |
Node without operation_name uses 'unknown'. |
ReporterLangfuse — Langfuse observer for execution and evaluation.
Implements :class:ExecutionObserver and :class:EvaluationObserver
using the official langfuse client. Covers two scopes:
- Tracing: via
ExecutionObserver, sends plan/node lifecycle events as Langfuse traces and spans. - Dataset scores: via
EvaluationObserver, sends criterion and row scores as Langfuse scores, optionally linked to traces.
Credentials and configuration are passed via constructor parameters, not environment variables, so the reporter can be instantiated programmatically without global side-effects.
Usage::
reporter = ReporterLangfuse(
secret_key="sk-lf-...",
public_key="pk-lf-...",
)
# Pass to api.execute() or EvaluationRun as observer
| CLASS | DESCRIPTION |
|---|---|
ReporterLangfuse |
Langfuse-based reporter implementing both observer protocols. |
ReporterLangfuse(*, secret_key: str, public_key: str, base_url: str = 'https://cloud.langfuse.com', enabled: bool = True)
Langfuse-based reporter implementing both observer protocols.
Uses the official langfuse client for:
- Tracing of plan/node execution (
ExecutionObserverhooks). - Sending evaluation scores to Langfuse (
EvaluationObserverhooks), including dataset run association.
When enabled=False all hooks are no-ops, which is useful in CI
environments where no Langfuse instance is available.
| ATTRIBUTE | DESCRIPTION |
|---|---|
enabled |
Whether to actually send data to Langfuse.
TYPE:
|
_client |
The underlying
TYPE:
|
| PARAMETER | DESCRIPTION |
|---|---|
|
Langfuse secret key (project-level).
TYPE:
|
|
Langfuse public key (project-level).
TYPE:
|
|
Langfuse API base URL. Defaults to the public cloud instance.
TYPE:
|
|
When
TYPE:
|
| PARAMETER | DESCRIPTION |
|---|---|
|
Langfuse secret key (project-level).
TYPE:
|
|
Langfuse public key (project-level).
TYPE:
|
|
Langfuse API base URL.
TYPE:
|
|
When
TYPE:
|
| METHOD | DESCRIPTION |
|---|---|
on_plan_start |
Create a root Langfuse trace for the plan execution. |
on_node_start |
Create a child span for a plan node. |
on_node_complete |
End the span for a completed node. |
on_plan_complete |
End the root trace for the plan execution. |
on_run_start |
Initialize dataset run context in Langfuse. |
on_criterion_complete |
Send a single criterion score to Langfuse. |
on_row_complete |
Log row completion (no Langfuse action). |
on_run_complete |
Finalize the dataset run and flush all pending data. |
class-attribute
instance-attribute
¤
__slots__ = ('enabled', '_client', '_root_span', '_active_spans', '_dataset_run_id', '_dataset_name', '_score_batch')
async
¤
async
¤
async
¤
End the span for a completed node.
Updates the span with output metadata and ends it. If the node is not tracked (e.g. started before the reporter was attached), this is silently ignored.
| PARAMETER | DESCRIPTION |
|---|---|
|
The :class:
TYPE:
|
|
The structured output artifact produced.
TYPE:
|
async
¤
End the root trace for the plan execution.
Flushes all pending spans and ends the root observation. Any remaining active spans are also ended to prevent orphaned traces.
| PARAMETER | DESCRIPTION |
|---|---|
|
The :class:
TYPE:
|
|
The final output of the last node.
TYPE:
|
async
¤
async
¤
Send a single criterion score to Langfuse.
Scores are sent as NUMERIC scores using
:meth:langfuse.Langfuse.create_score. Each score includes
the criterion name and the points value, plus any explanation
from the score object.
| PARAMETER | DESCRIPTION |
|---|---|
|
Zero-based dataset row index.
TYPE:
|
|
The evaluator operation name.
TYPE:
|
|
The :class:
TYPE:
|
async
¤
Log row completion (no Langfuse action).
The per-criterion scores are already sent individually in
on_criterion_complete. This hook is a no-op for Langfuse
but is included for protocol compliance.
| PARAMETER | DESCRIPTION |
|---|---|
|
Zero-based row index.
TYPE:
|
|
Dict mapping criterion names to :class: |
Tests for ReporterLangfuse.
| CLASS | DESCRIPTION |
|---|---|
FakeSpan |
Fake Langfuse span for mocking start_observation. |
FakeClient |
Fake Langfuse client capturing method calls for assertions. |
TestReporterLangfuseProtocolCompliance |
Verify ReporterLangfuse satisfies both observer protocols. |
TestReporterLangfuseTracing |
Verify ExecutionObserver hooks interact with Langfuse correctly. |
TestReporterLangfuseEvaluation |
Verify EvaluationObserver hooks interact with Langfuse correctly. |
TestReporterLangfuseDisabled |
Verify disabled reporter is a no-op for all hooks. |
TestReporterLangfuseRowComplete |
Verify on_row_complete behavior. |
TestReporterLangfuseInitialization |
Tests for ReporterLangfuse constructor behavior. |
FakeClient()
Fake Langfuse client capturing method calls for assertions.
| METHOD | DESCRIPTION |
|---|---|
create_score |
|
flush |
|
get_dataset |
|
create_dataset |
|
start_observation |
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
scores |
|
flushed |
|
datasets_created |
|
datasets_fetched |
|
started_observations |
|
Verify ReporterLangfuse satisfies both observer protocols.
| METHOD | DESCRIPTION |
|---|---|
test_is_execution_observer |
ReporterLangfuse passes isinstance check for ExecutionObserver. |
test_is_evaluation_observer |
ReporterLangfuse passes isinstance check for EvaluationObserver. |
Verify ExecutionObserver hooks interact with Langfuse correctly.
| METHOD | DESCRIPTION |
|---|---|
test_on_plan_start_creates_root_trace |
on_plan_start calls start_observation with chain type. |
test_on_node_start_creates_span_child |
on_node_start creates a span with trace_context from root. |
test_on_node_complete_ends_span |
on_node_complete calls update() and end() on the span. |
test_on_plan_complete_ends_root_trace |
on_plan_complete calls end() on the root span and clears it. |
test_on_plan_complete_clears_orphan_spans |
Any spans left in _active_spans are ended on plan complete. |
async
¤
test_on_plan_start_creates_root_trace()
on_plan_start calls start_observation with chain type.
async
¤
test_on_node_start_creates_span_child()
on_node_start creates a span with trace_context from root.
async
¤
test_on_node_complete_ends_span()
on_node_complete calls update() and end() on the span.
async
¤
test_on_plan_complete_ends_root_trace()
on_plan_complete calls end() on the root span and clears it.
async
¤
test_on_plan_complete_clears_orphan_spans()
Any spans left in _active_spans are ended on plan complete.
Verify EvaluationObserver hooks interact with Langfuse correctly.
| METHOD | DESCRIPTION |
|---|---|
test_on_run_start_initializes_dataset |
on_run_start creates or fetches the dataset from Langfuse. |
test_on_criterion_complete_sends_score |
on_criterion_complete calls create_score with correct params. |
test_on_criterion_complete_multiple_scores |
Multiple criterion completions send multiple scores. |
test_on_criterion_complete_with_error_score |
ScoreError (points=0) sends 0.0 value. |
test_on_run_complete_flushes |
on_run_complete calls flush() on the client. |
test_on_run_complete_flushes_after_scores |
Scores are sent before flush on run complete. |
async
¤
test_on_run_start_initializes_dataset()
on_run_start creates or fetches the dataset from Langfuse.
async
¤
test_on_criterion_complete_sends_score()
on_criterion_complete calls create_score with correct params.
async
¤
test_on_criterion_complete_multiple_scores()
Multiple criterion completions send multiple scores.
async
¤
test_on_criterion_complete_with_error_score()
ScoreError (points=0) sends 0.0 value.
async
¤
test_on_run_complete_flushes()
on_run_complete calls flush() on the client.
async
¤
test_on_run_complete_flushes_after_scores()
Scores are sent before flush on run complete.
Verify disabled reporter is a no-op for all hooks.
| METHOD | DESCRIPTION |
|---|---|
test_client_is_none_when_disabled |
When enabled=False, _client is None. |
test_execution_hooks_are_noops |
All ExecutionObserver hooks are no-ops when disabled. |
test_evaluation_hooks_are_noops |
All EvaluationObserver hooks are no-ops when disabled. |
test_disabled_does_not_call_client |
When disabled, no client methods are invoked. |
test_client_is_none_when_disabled()
When enabled=False, _client is None.
async
¤
test_execution_hooks_are_noops()
All ExecutionObserver hooks are no-ops when disabled.
async
¤
test_evaluation_hooks_are_noops()
All EvaluationObserver hooks are no-ops when disabled.
async
¤
test_disabled_does_not_call_client()
When disabled, no client methods are invoked.
Verify on_row_complete behavior.
| METHOD | DESCRIPTION |
|---|---|
test_on_row_complete_is_noop |
on_row_complete does nothing (scores already sent per-criterion). |
async
¤
test_on_row_complete_is_noop()
on_row_complete does nothing (scores already sent per-criterion).
Tests for ReporterLangfuse constructor behavior.
| METHOD | DESCRIPTION |
|---|---|
test_constructor_stores_parameters |
Constructor stores all parameters correctly. |
test_constructor_default_values |
Default values are applied when parameters are omitted. |
test_on_run_start_without_dataset_name |
When run has no dataset_name, defaults to 'evaluation'. |
test_on_criterion_complete_preserves_explanation |
Score's explanation is passed as comment. |
test_constructor_stores_parameters()
Constructor stores all parameters correctly.
test_constructor_default_values()
Default values are applied when parameters are omitted.
async
¤
test_on_run_start_without_dataset_name()
When run has no dataset_name, defaults to 'evaluation'.
async
¤
test_on_criterion_complete_preserves_explanation()
Score's explanation is passed as comment.
ReporterNull — no-op observer for both execution and evaluation.
Implements :class:ExecutionObserver and :class:EvaluationObserver
with all hooks as no-ops. Useful in tests and headless environments
where no output is desired.
Because both protocols already provide default no-op implementations,
this is a structural placeholder that satisfies isinstance checks.
| CLASS | DESCRIPTION |
|---|---|
ReporterNull |
No-op reporter that implements both observer protocols. |
No-op reporter that implements both observer protocols.
All methods inherit their default no-op implementations from the
protocols. This class exists so that callers can instantiate a
concrete object that satisfies isinstance checks for both
ExecutionObserver and EvaluationObserver.
Usage::
reporter = ReporterNull()
assert isinstance(reporter, ExecutionObserver)
assert isinstance(reporter, EvaluationObserver)
await reporter.on_run_start(run) # does nothing
| METHOD | DESCRIPTION |
|---|---|
on_run_start |
Called when an evaluation run begins. |
on_criterion_complete |
Called when a single criterion evaluation completes. |
on_row_complete |
Called when all criteria for a single dataset row are complete. |
on_run_complete |
Called when the full evaluation run completes. |
on_plan_start |
Called when execution of a plan begins. |
on_node_start |
Called before a node begins execution. |
on_node_complete |
Called after a node completes execution successfully. |
on_plan_complete |
Called when execution of a plan finishes successfully. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
__slots__ |
|
Tests for ReporterNull.
| CLASS | DESCRIPTION |
|---|---|
TestReporterNullProtocolCompliance |
Verify ReporterNull satisfies both observer protocols. |
TestReporterNullNoExceptions |
Verify ReporterNull methods do not raise exceptions. |
Verify ReporterNull satisfies both observer protocols.
| METHOD | DESCRIPTION |
|---|---|
test_is_execution_observer |
ReporterNull passes isinstance check for ExecutionObserver. |
test_is_evaluation_observer |
ReporterNull passes isinstance check for EvaluationObserver. |
Verify ReporterNull methods do not raise exceptions.
| METHOD | DESCRIPTION |
|---|---|
test_execution_observer_hooks_dont_raise |
All ExecutionObserver hooks complete without error. |
test_evaluation_observer_hooks_dont_raise |
All EvaluationObserver hooks complete without error. |
test_all_hooks_are_callable |
Every hook method exists and can be called. |
test_reporter_null_is_slotted |
ReporterNull uses slots for memory efficiency. |
async
¤
test_execution_observer_hooks_dont_raise()
All ExecutionObserver hooks complete without error.
async
¤
test_evaluation_observer_hooks_dont_raise()
All EvaluationObserver hooks complete without error.
async
¤
test_all_hooks_are_callable()
Every hook method exists and can be called.
test_reporter_null_is_slotted()
ReporterNull uses slots for memory efficiency.
EvaluationRun orchestrator for CrewMaster v2.0.0 evaluation system.
Provides EvaluationConfig, EvaluationSession, EvaluationRun, and EvaluationRunError — the central orchestration layer that ties together snapshots, datasets, criteria, and the API execution engine to produce an EvaluationReport.
| CLASS | DESCRIPTION |
|---|---|
EvaluationConfig |
Configuration for an evaluation run. |
EvaluationSession |
All dependencies needed to execute an evaluation run. |
EvaluationRunError |
Raised when an evaluation run fails its completion threshold. |
EvaluationRun |
Orchestrator for evaluating an operation configuration against a dataset. |
Configuration for an evaluation run.
| ATTRIBUTE | DESCRIPTION |
|---|---|
timeout_seconds |
Maximum time in seconds for each dataset row execution before it is considered failed.
TYPE:
|
min_completion_pct |
Minimum percentage of rows that must complete
successfully for the run to be considered valid. If the
actual completion percentage is below this threshold,
:class:
TYPE:
|
All dependencies needed to execute an evaluation run.
Bundles the stores, runtime, observers, and config into a single injectable session object.
| ATTRIBUTE | DESCRIPTION |
|---|---|
snapshot_store |
The SnapshotStore for resolving snapshots.
TYPE:
|
dataset_loader |
The DatasetLoader for loading datasets.
TYPE:
|
runtime |
The RuntimeDriver for executing operations.
TYPE:
|
block_store |
The BlockStore for resolving block URIs.
TYPE:
|
prompt_engine |
The PromptEngine for rendering prompts.
TYPE:
|
context_store |
The ContextStore for domain resolution.
TYPE:
|
execution_observer |
Optional ExecutionObserver for plan lifecycle.
TYPE:
|
evaluation_observer |
Optional EvaluationObserver for run lifecycle.
TYPE:
|
config |
Evaluation configuration (timeout, thresholds).
TYPE:
|
class-attribute
instance-attribute
¤
config: EvaluationConfig = Field(default_factory=EvaluationConfig)
Raised when an evaluation run fails its completion threshold.
Orchestrator for evaluating an operation configuration against a dataset.
An EvaluationRun binds a snapshot (versioned operation + agent + task
blocks) with a dataset of input rows and a list of criterion operations.
When executed, it runs the snapshot's operation against every row,
evaluates the output with each criterion, and aggregates the results
into an :class:EvaluationReport.
| ATTRIBUTE | DESCRIPTION |
|---|---|
snapshot_name |
Name of the snapshot to evaluate.
TYPE:
|
dataset_name |
Name of the dataset to use.
TYPE:
|
criteria |
List of evaluator Operations, each producing a Score. |
| METHOD | DESCRIPTION |
|---|---|
execute |
Execute the evaluation run end-to-end. |
class-attribute
instance-attribute
¤
async
¤
execute(session: EvaluationSession) -> EvaluationReport
Execute the evaluation run end-to-end.
Flow:
1. Load the snapshot from the snapshot store.
2. Load the dataset from the dataset loader.
3. Build the effective Operation via snapshot.as_operation().
4. Resolve the ExecutionPlan once.
5. For each dataset row:
a. Call :func:crewmaster.api.execute with the row's context
injected as context_overrides.
b. Execute each criterion operation against the output.
c. Emit on_criterion_complete and on_row_complete.
6. Aggregate results into an EvaluationReport.
7. Check the completion threshold. If below
min_completion_pct, raise :class:EvaluationRunError.
8. Emit on_run_complete and return the report.
| PARAMETER | DESCRIPTION |
|---|---|
|
The fully configured evaluation session.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
An
|
class:
TYPE:
|
EvaluationReport
|
the global score. |
| RAISES | DESCRIPTION |
|---|---|
EvaluationRunError
|
If the percentage of completed rows is
below |
FileNotFoundError
|
If the snapshot or dataset cannot be found. |
Tests for EvaluationRun orchestrator.
Covers: - Tracer bullet end-to-end (3 rows, no criteria) - With criteria operations producing Score - Timeout handling - Exception capture as ScoreError - Completion threshold (min_completion_pct) - EvaluationObserver integration
| CLASS | DESCRIPTION |
|---|---|
TextOutput |
|
FakeDriver |
Fake driver that returns canned responses. |
TestEvaluationRunTracerBullet |
End-to-end tracer bullet: 3 rows, no criteria. |
TestEvaluationRunWithCriteria |
Tests for evaluation runs with criterion operations. |
TestEvaluationRunTimeout |
Tests for timeout handling in evaluation runs. |
TestEvaluationRunException |
Tests for exception capture as ScoreError. |
TestEvaluationRunThreshold |
Tests for min_completion_pct threshold. |
TestEvaluationRunObserver |
Tests for EvaluationObserver integration. |
TestEvaluationConfig |
Tests for EvaluationConfig defaults and construction. |
TestEvaluationSession |
Tests for EvaluationSession construction. |
TestEvaluationRunModel |
Tests for EvaluationRun model construction. |
TestSnapshotAsOperation |
Tests for Snapshot.as_operation() method. |
FakeDriver(responses: list | None = None, fail_on_call: int | None = None, fail_on_calls: set[int] | None = None, persistent_fail_from_call: int | None = None, sleep_seconds: float | None = None)
Fake driver that returns canned responses.
Supports per-call canned output and configurable failure/behavior.
| METHOD | DESCRIPTION |
|---|---|
execute |
|
astream |
Not used in evaluation runs but required by protocol. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
responses |
|
fail_on_call |
|
fail_on_calls |
|
persistent_fail_from_call |
|
sleep_seconds |
|
calls |
TYPE:
|
instance-attribute
¤
persistent_fail_from_call = persistent_fail_from_call
async
¤
astream(request: RuntimeRequest)
Not used in evaluation runs but required by protocol.
End-to-end tracer bullet: 3 rows, no criteria.
| METHOD | DESCRIPTION |
|---|---|
test_three_rows_no_criteria |
EvaluationRun with 3 dataset rows and no criteria. |
test_context_overrides_injected |
Row context is injected as template variables during execution. |
async
¤
test_three_rows_no_criteria(tmp_path)
EvaluationRun with 3 dataset rows and no criteria.
Verifies the report has 3 rows, completed_rows=3, total_rows=3.
async
¤
test_context_overrides_injected(tmp_path)
Row context is injected as template variables during execution.
Tests for evaluation runs with criterion operations.
| METHOD | DESCRIPTION |
|---|---|
test_criteria_produce_scores |
Criteria operations that produce Score appear in the report. |
test_criterion_context_includes_output |
Criterion operations receive operation_output in context. |
Tests for timeout handling in evaluation runs.
| METHOD | DESCRIPTION |
|---|---|
test_timeout_produces_score_errors |
A row that times out has ScoreError for each criterion. |
test_timeout_other_rows_continue |
When one row times out, subsequent rows still execute. |
Tests for exception capture as ScoreError.
| METHOD | DESCRIPTION |
|---|---|
test_exception_becomes_score_error |
A runtime exception in a row is captured as an error row. |
test_criterion_exception_becomes_score_error |
If a criterion operation fails, it's captured as ScoreError. |
async
¤
test_exception_becomes_score_error(tmp_path)
A runtime exception in a row is captured as an error row.
async
¤
test_criterion_exception_becomes_score_error(tmp_path)
If a criterion operation fails, it's captured as ScoreError.
Tests for min_completion_pct threshold.
| METHOD | DESCRIPTION |
|---|---|
test_threshold_met |
When completion is above threshold, no error is raised. |
test_threshold_not_met_raises_error |
When completion is below threshold, EvaluationRunError is raised. |
test_threshold_exactly_met |
When completion equals threshold, no error. |
test_threshold_with_no_rows |
Empty dataset with any threshold should succeed. |
async
¤
test_threshold_met(tmp_path)
When completion is above threshold, no error is raised.
async
¤
test_threshold_not_met_raises_error(tmp_path)
When completion is below threshold, EvaluationRunError is raised.
async
¤
test_threshold_exactly_met(tmp_path)
When completion equals threshold, no error.
async
¤
test_threshold_with_no_rows(tmp_path)
Empty dataset with any threshold should succeed.
Tests for EvaluationObserver integration.
| METHOD | DESCRIPTION |
|---|---|
test_observer_called_for_each_row |
EvaluationObserver hooks are called for each row and criterion. |
test_null_observer_does_not_break |
Setting observer to None should work without errors. |
test_observer_sees_error_scores |
Observer on_criterion_complete receives ScoreError when row fails. |
async
¤
test_observer_called_for_each_row(tmp_path)
EvaluationObserver hooks are called for each row and criterion.
async
¤
test_null_observer_does_not_break(tmp_path)
Setting observer to None should work without errors.
async
¤
test_observer_sees_error_scores(tmp_path)
Observer on_criterion_complete receives ScoreError when row fails.
Tests for EvaluationConfig defaults and construction.
| METHOD | DESCRIPTION |
|---|---|
test_default_values |
EvaluationConfig has correct defaults. |
test_custom_values |
EvaluationConfig accepts custom values. |
Tests for EvaluationSession construction.
| METHOD | DESCRIPTION |
|---|---|
test_minimal_session |
EvaluationSession can be constructed with required fields. |
test_session_with_observers |
EvaluationSession accepts observer instances. |
Tests for EvaluationRun model construction.
| METHOD | DESCRIPTION |
|---|---|
test_construct_without_criteria |
EvaluationRun can be constructed with no criteria. |
test_construct_with_criteria |
EvaluationRun can be constructed with criteria operations. |
Tests for Snapshot.as_operation() method.
| METHOD | DESCRIPTION |
|---|---|
test_as_operation_creates_operation |
as_operation() creates a valid Operation from snapshot fields. |
test_as_operation_with_string_agent |
as_operation() works when agent is a string. |
| MODULE | DESCRIPTION |
|---|---|
score |
|
score_base |
|
score_test |
Tests for Score types in crewmaster/evaluation/score/. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
Score |
|
ScoreAdapter |
TYPE:
|
module-attribute
¤
__all__ = ['Score', 'ScoreAdapter', 'ScoreBase', 'ScoreBoolean', 'ScoreCategoricalBinary', 'ScoreBooleanDirect', 'ScoreBooleanInverse', 'ScoreCategorialInverse', 'ScoreCategorical', 'ScoreCategoricalDirect', 'ScoreError', 'ScorePercent', 'ScorePercentDirect', 'ScorePercentInverse']
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
points |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
points |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
correct_categories |
TYPE:
|
points |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
points |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
log |
Loger para el módulo
|
CategoryName |
|
Score |
|
ScoreAdapter |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
points |
TYPE:
|
name |
TYPE:
|
explanation |
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
points |
TYPE:
|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
points |
TYPE:
|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
max_categories_allowed |
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
max_categories_allowed |
TYPE:
|
correct_categories |
TYPE:
|
points |
TYPE:
|
name |
TYPE:
|
explanation |
|
type |
TYPE:
|
value |
TYPE:
|
Tests for Score types in crewmaster/evaluation/score/.
| CLASS | DESCRIPTION |
|---|---|
TestScoreBooleanDirect |
|
TestScoreBooleanInverse |
|
TestScorePercentDirect |
|
TestScorePercentInverse |
|
TestScoreCategoricalDirect |
|
TestScoreCategorialInverse |
|
TestScoreCategoricalBinary |
|
TestScoreError |
|
TestScoreAdapter |
Tests for ScoreAdapter serialization and discriminators. |
| METHOD | DESCRIPTION |
|---|---|
test_true_returns_100 |
|
test_false_returns_0 |
|
| METHOD | DESCRIPTION |
|---|---|
test_true_returns_0 |
|
test_false_returns_100 |
|
| METHOD | DESCRIPTION |
|---|---|
test_returns_value_unchanged |
|
test_zero_returns_zero |
|
test_hundred_returns_hundred |
|
| METHOD | DESCRIPTION |
|---|---|
test_returns_100_minus_value |
|
test_zero_returns_100 |
|
test_hundred_returns_0 |
|
| METHOD | DESCRIPTION |
|---|---|
test_single_category_with_max_3 |
|
test_all_categories_with_max_4 |
|
| METHOD | DESCRIPTION |
|---|---|
test_single_category_with_max_3 |
|
test_no_categories_with_max_5 |
|
| METHOD | DESCRIPTION |
|---|---|
test_always_returns_0 |
|
test_with_none_source |
|
Tests for ScoreAdapter serialization and discriminators.
The ScoreAdapter uses a discriminated union mapping type literals to the abstract base classes (ScorePercent, ScoreBoolean, ScoreCategorical, ScoreError). Deserialization resolves to the base class, not concrete subtypes. This matches the legacy behavior and is sufficient for structured dump/inspect workflows.
| METHOD | DESCRIPTION |
|---|---|
test_deserialize_boolean |
|
test_deserialize_percent |
|
test_deserialize_categorical |
|
test_deserialize_error |
|
test_serialize_dump |
Dumping a concrete subtype preserves the base discriminator. |
test_serialize_error |
|
test_serialize_dump()
Dumping a concrete subtype preserves the base discriminator.
Snapshot store for CrewMaster v2.0.0 evaluation system.
Provides the Snapshot model, SnapshotStore protocol, and FileSnapshotStore implementation. Snapshots are versioned configurations that bind an Operation, AgentConfig, and task blocks for evaluation and deployment.
| CLASS | DESCRIPTION |
|---|---|
Snapshot |
A versioned configuration binding an Operation, Agent, and task blocks. |
SnapshotStore |
Protocol for resolving and managing configuration snapshots. |
FileSnapshotStore |
SnapshotStore implementation backed by YAML files on disk. |
A versioned configuration binding an Operation, Agent, and task blocks.
Snapshots are the primary artefact for evaluation and deployment. Each snapshot captures a specific version of the configuration needed to execute a given operation with a particular agent and task blocks.
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
Unique snapshot name (e.g., "blueprint_v1").
TYPE:
|
operation_name |
The Operation this snapshot targets.
TYPE:
|
agent |
AgentConfig instance or agent name string. |
task_blocks |
List of |
| METHOD | DESCRIPTION |
|---|---|
as_operation |
Build an Operation from this snapshot's configuration. |
class-attribute
instance-attribute
¤
as_operation() -> 'Operation'
Build an Operation from this snapshot's configuration.
The resulting operation uses the snapshot's agent and task_blocks,
with produces=str and kind="artifact" as defaults suitable
for evaluation runs.
When the agent is stored as a plain dict (e.g. when loaded from
YAML), it is automatically reconstructed as an :class:AgentConfig.
| RETURNS | DESCRIPTION |
|---|---|
'Operation'
|
An Operation configured from this snapshot. |
Protocol for resolving and managing configuration snapshots.
Implementations are responsible for loading snapshots from their storage medium (local disk, remote, database) and managing which snapshot is active for each operation.
The default implementation is :class:FileSnapshotStore, which
reads YAML files from a configured directory.
| METHOD | DESCRIPTION |
|---|---|
resolve |
Resolve a snapshot by name. |
get_active |
Get the currently active snapshot for an operation. |
set_active |
Set the active snapshot for an operation (promote). |
list_snapshots |
List all snapshot names for a given operation. |
Resolve a snapshot by name.
| PARAMETER | DESCRIPTION |
|---|---|
|
Snapshot name (e.g., "blueprint_v1") or URI (e.g., "snapshot://blueprint_v1").
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Snapshot
|
The resolved Snapshot. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If the snapshot does not exist. |
get_active(operation_name: str) -> Snapshot
Get the currently active snapshot for an operation.
| PARAMETER | DESCRIPTION |
|---|---|
|
The operation name to look up.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Snapshot
|
The active Snapshot for the given operation. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If no active snapshot exists for the operation. |
set_active(operation_name: str, snapshot_name: str) -> None
SnapshotStore implementation backed by YAML files on disk.
Reads snapshot files from a configured root directory. Each snapshot
is stored as {name}.yaml containing the snapshot data. The active
state is managed in active.yaml which maps operation names to
their active snapshot names.
| PARAMETER | DESCRIPTION |
|---|---|
|
Root directory from which snapshot files are resolved.
TYPE:
|
| METHOD | DESCRIPTION |
|---|---|
resolve |
Resolve a snapshot by name from a YAML file. |
get_active |
Get the currently active snapshot for an operation. |
set_active |
Set the active snapshot for an operation. |
save |
Save a snapshot to a YAML file on disk. |
list_snapshots |
List all snapshot names for a given operation. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
SNAPSHOT_PREFIX |
|
ACTIVE_FILE |
|
Resolve a snapshot by name from a YAML file.
Supports both plain names and snapshot:// URIs.
| PARAMETER | DESCRIPTION |
|---|---|
|
Snapshot name or
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Snapshot
|
The resolved Snapshot. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If the YAML file does not exist. |
ValueError
|
If the YAML is malformed. |
get_active(operation_name: str) -> Snapshot
Get the currently active snapshot for an operation.
Reads the active.yaml file and resolves the snapshot name
mapped to the given operation.
| PARAMETER | DESCRIPTION |
|---|---|
|
The operation to look up.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Snapshot
|
The active Snapshot. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If |
set_active(operation_name: str, snapshot_name: str) -> None
Set the active snapshot for an operation.
Creates or updates active.yaml, mapping the operation name
to the given snapshot name. Existing entries for other operations
are preserved.
| PARAMETER | DESCRIPTION |
|---|---|
|
The operation to update.
TYPE:
|
|
The snapshot to promote as active.
TYPE:
|
Save a snapshot to a YAML file on disk.
The snapshot is written to {root}/{snapshot.name}.yaml.
The name field is excluded from the YAML content since it
is encoded in the filename.
| PARAMETER | DESCRIPTION |
|---|---|
|
The Snapshot to persist.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
OSError
|
If the file cannot be written. |
list_snapshots(operation_name: str) -> list[str]
List all snapshot names for a given operation.
Scans all .yaml files in the root directory (excluding
active.yaml), parses each to check its operation_name
field, and returns the matching names sorted alphabetically.
| PARAMETER | DESCRIPTION |
|---|---|
|
The operation to filter by.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
list[str]
|
Sorted list of snapshot name strings. |
Tests for SnapshotStore protocol and FileSnapshotStore implementation.
| CLASS | DESCRIPTION |
|---|---|
TestSnapshot |
Tests for the Snapshot model construction and defaults. |
TestFileSnapshotStoreResolve |
Tests for FileSnapshotStore.resolve(). |
TestFileSnapshotStoreActive |
Tests for FileSnapshotStore active state management. |
TestFileSnapshotStoreList |
Tests for FileSnapshotStore.list_snapshots(). |
TestSnapshotStoreProtocol |
Tests verifying FileSnapshotStore conforms to SnapshotStore protocol. |
Tests for the Snapshot model construction and defaults.
| METHOD | DESCRIPTION |
|---|---|
test_construct_from_fields |
Snapshot can be constructed with all fields. |
test_construct_with_agentconfig |
Snapshot accepts an AgentConfig instance as agent field. |
test_default_task_blocks_empty |
Snapshot task_blocks defaults to empty list. |
test_construct_from_fields() -> None
Snapshot can be constructed with all fields.
test_construct_with_agentconfig() -> None
Snapshot accepts an AgentConfig instance as agent field.
test_default_task_blocks_empty() -> None
Snapshot task_blocks defaults to empty list.
Tests for FileSnapshotStore.resolve().
| METHOD | DESCRIPTION |
|---|---|
test_resolve_loads_from_yaml |
resolve() loads a Snapshot from a YAML file. |
test_resolve_strips_snapshot_prefix |
resolve() strips snapshot:// prefix from URIs. |
test_resolve_raises_for_missing_file |
resolve() raises FileNotFoundError for nonexistent snapshots. |
test_resolve_raises_for_non_dict_yaml |
resolve() raises ValueError if YAML is not a mapping. |
test_resolve_preserves_extra_fields |
resolve() preserves additional fields from YAML in the model. |
test_resolve_loads_from_yaml(tmp_path) -> None
resolve() loads a Snapshot from a YAML file.
test_resolve_strips_snapshot_prefix(tmp_path) -> None
resolve() strips snapshot:// prefix from URIs.
test_resolve_raises_for_missing_file(tmp_path) -> None
resolve() raises FileNotFoundError for nonexistent snapshots.
test_resolve_raises_for_non_dict_yaml(tmp_path) -> None
resolve() raises ValueError if YAML is not a mapping.
test_resolve_preserves_extra_fields(tmp_path) -> None
resolve() preserves additional fields from YAML in the model.
Tests for FileSnapshotStore active state management.
| METHOD | DESCRIPTION |
|---|---|
test_get_active_returns_snapshot |
get_active() returns the active snapshot for an operation. |
test_get_active_raises_when_no_active_file |
get_active() raises FileNotFoundError if active.yaml is missing. |
test_get_active_raises_when_operation_not_found |
get_active() raises FileNotFoundError when operation has no entry. |
test_set_active_creates_file |
set_active() creates active.yaml when it doesn't exist. |
test_set_active_updates_existing_entry |
set_active() updates an existing entry while preserving others. |
test_set_active_adds_new_operation |
set_active() adds a new operation entry to existing active.yaml. |
test_get_active_returns_snapshot(tmp_path) -> None
get_active() returns the active snapshot for an operation.
test_get_active_raises_when_no_active_file(tmp_path) -> None
get_active() raises FileNotFoundError if active.yaml is missing.
test_get_active_raises_when_operation_not_found(tmp_path) -> None
get_active() raises FileNotFoundError when operation has no entry.
test_set_active_creates_file(tmp_path) -> None
set_active() creates active.yaml when it doesn't exist.
test_set_active_updates_existing_entry(tmp_path) -> None
set_active() updates an existing entry while preserving others.
test_set_active_adds_new_operation(tmp_path) -> None
set_active() adds a new operation entry to existing active.yaml.
Tests for FileSnapshotStore.list_snapshots().
| METHOD | DESCRIPTION |
|---|---|
test_list_snapshots_filters_by_operation |
list_snapshots() only returns snapshots for the given operation. |
test_list_snapshots_empty_for_unknown_operation |
list_snapshots() returns empty list for unknown operations. |
test_list_snapshots_ignores_malformed_yaml |
list_snapshots() skips files with invalid YAML instead of crashing. |
test_list_snapshots_filters_by_operation(tmp_path) -> None
list_snapshots() only returns snapshots for the given operation.
test_list_snapshots_empty_for_unknown_operation(tmp_path) -> None
list_snapshots() returns empty list for unknown operations.
test_list_snapshots_ignores_malformed_yaml(tmp_path) -> None
list_snapshots() skips files with invalid YAML instead of crashing.
Tests verifying FileSnapshotStore conforms to SnapshotStore protocol.
| METHOD | DESCRIPTION |
|---|---|
test_instance_matches_protocol |
FileSnapshotStore is structurally compatible with SnapshotStore. |
test_protocol_is_runtime_checkable |
SnapshotStore can be checked with isinstance at runtime. |