Evaluate with the Sandbox

This guide explains how to use the CrewMaster sandbox to evaluate operations using the evaluation system (snapshots, datasets, criteria, and evaluation reports).

What is the evaluation system?

CrewMaster v2.0.0 includes a complete evaluation framework (crewmaster/evaluation/) that lets you:

• Version your operations with snapshots (YAML files that capture expected inputs/outputs of an operation).
• Define datasets in YAML with rows of input data to test against.
• Write criteria as Operation nodes that produce a Score (ScoreBoolean, ScorePercentDirect, etc.) — these are the same Operation primitives you already know from the DAG system.
• Run evaluations with EvaluationRun, which executes your operation against every row in the dataset, evaluates the output with each criterion, and produces an EvaluationReport with per-criterion averages, per-row scores, and a global_score.

Starting the sandbox

cd /path/to/crewmaster

# Install with sandbox-ui extra
pip install -e ".[sandbox-ui]"

# Start the UI (FastAPI + NiceGUI)
python -m crewmaster.sandbox.ui.app

The web UI opens at http://localhost:8080. The FastAPI endpoints are mounted at /api/, so you can also use curl against the API directly.

If you only need the API (no UI), start the FastAPI server standalone:

uvicorn crewmaster.sandbox.main:app --host 0.0.0.0 --port 8000

Evaluate a single operation (evaluate-single)

The evaluate-single scenario demonstrates evaluating a simple product review summarizer. The operation produces a structured ReviewSummary (sentiment, key positives/negatives, rating, justification) and is evaluated with three criteria:

Criterion	Score type	What it measures
sentiment_accuracy	ScoreBoolean	Whether the sentiment label is correct
coverage	ScorePercentDirect	How many review points are covered
rating_justification	ScorePercentDirect	Quality of the rating justification

Using curl

curl -s -X POST http://localhost:8000/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "scenario": "evaluate-single",
    "snapshot": "summarizer_v1",
    "dataset": "mixed_reviews",
    "criteria": ["sentiment_accuracy", "coverage", "rating_justification"],
    "observer": "console"
  }' | python -m json.tool

Using the UI

1. Open http://localhost:8080 in your browser.
2. Find the Evaluate Single card (icon: assessment).
3. Click 🔍 Evaluar.
4. The result area shows the evaluation report with the global score highlighted, per-criterion averages, and per-row scores.

Evaluate a multi-node pipeline (evaluate-pipeline)

The evaluate-pipeline scenario reuses the DAG pipeline domain from dag_pipeline (BrandSeed → TypographyDesigner → Colorist → Designer → VisualIdentity) and evaluates it with three criteria:

Criterion	Score type	What it measures
font_industry_fit	ScorePercentDirect	How well fonts match the industry
color_harmony	ScorePercentDirect	Color palette cohesion and harmony
identity_coherence	ScorePercentDirect	Overall visual identity consistency

Using curl

curl -s -X POST http://localhost:8000/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "scenario": "evaluate-pipeline",
    "snapshot": "brand_identity_v1",
    "dataset": "brand_seeds",
    "criteria": ["font_industry_fit", "color_harmony", "identity_coherence"],
    "observer": "console"
  }' | python -m json.tool

Using the UI

1. Open http://localhost:8080.
2. Find the Evaluate Pipeline card (icon: account_tree).
3. Click 🔍 Evaluar.

Understanding the response

A successful evaluation returns a report like:

{
  "snapshot": "summarizer_v1",
  "dataset": "mixed_reviews",
  "criteria": [
    {
      "name": "sentiment_accuracy",
      "average": 1.0,
      "type": "ScoreBoolean"
    },
    {
      "name": "coverage",
      "average": 85.0,
      "type": "ScorePercentDirect"
    },
    {
      "name": "rating_justification",
      "average": 90.0,
      "type": "ScorePercentDirect"
    }
  ],
  "rows": [
    {
      "index": 0,
      "context": { "review": "Great product, ..." },
      "scores": {
        "sentiment_accuracy": { "name": "sentiment_accuracy", "value": true, "explanation": "..." },
        "coverage": { "name": "coverage", "value": 85, "explanation": "..." },
        "rating_justification": { "name": "rating_justification", "value": 90, "explanation": "..." }
      }
    }
  ],
  "global_score": 87,
  "completed_rows": 3,
  "total_rows": 3,
  "errors": []
}

• global_score: Aggregate score from 0 to 100 across all criteria and rows.
• criteria[].average: Average score for that criterion across all rows.
• rows[].scores: Per-criterion scores for that specific dataset row.
• errors[]: Any evaluation errors (timeouts, driver failures).

Available observers

The observer field controls how evaluation progress is reported:

Observer	Description
console	Logs structured progress to stdout (default).
null	No output — silent evaluation.
langfuse	Sends traces to Langfuse. Requires environment variables LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY.

Scenarios as a starting point

Both evaluate-single and evaluate-pipeline are self-contained modules in crewmaster/sandbox/scenarios/. Each one:

• Defines its own domain models (ReviewSummary, VisualIdentity, etc.).
• Includes template blocks under crewmaster/sandbox/blocks/.
• Includes snapshot YAML files and dataset YAML files in their block directory.
• Builds evaluator Operation nodes as criteria.

Use them as reference implementations when integrating the evaluation system into your own CrewMaster projects:

• Single operation: See scenarios/evaluate_single.py — simple operation + three criteria, no sub-operations.
• Multi-node pipeline: See scenarios/evaluate_pipeline.py — reuses an existing DAG pipeline with sub_operations, adds three evaluation criteria that assess the final VisualIdentity output.

Next steps

• Add your own snapshots with crewmaster snapshot create (see CLI docs).
• Create custom datasets in YAML using the YAMLDatasetStore format.
• Write domain-specific criteria as Operation nodes with produces=Score*.
• Integrate observers for production monitoring (Langfuse, Datadog, etc.).