Fleet run grader + Langfuse calibration
Every night, grades the unattended AI workers' runs against a rubric, and asks James to spot-check a sample so the grader stays honest.
how it works
grade_runs.py: deterministic safety checks over transcripts (forbidden commands, call caps, path scope) then a tool-less claude -p majority-of-3 judge with the policy as of run time (git); pushes traces/scores to Langfuse over OTLP; calibrate.py samples into a Langfuse annotation queue and pulls verdicts into labels.jsonl.
Automation: Something that runs on a schedule by itself.
The code for this one lives in a private repo. This page explains the idea so you can build your own.