James Washington
writing about archive utils free game ideas replays
Technique

Run a classifier in shadow mode before it can gate

Let a new AI judge watch silently until there's evidence it's right.

Letting AI make calls, safely

how it works

Log scores only; a hand rubric backtested at chance (AUC 0.514) and a tiny-sample refit had a wide CI, so thresholds wait for labels; contrast with fail-closed gates like agent-merge.

Technique: A pattern or hard-won lesson, written down so it isn't learned twice.

The code for this one lives in a private repo. This page explains the idea so you can build your own.

classifierevaluationtechnique

related

  • Jev shadow scorer for recruiter mail Automation Tests an AI scoring of recruiter emails quietly, logging results without letting it decide anything yet.
  • agent-merge: Jev-gated self-merge for agent branches Automation The one sanctioned way an AI may merge its own work: a script that reviews it first and refuses if it didn't show its work.
  • Fleet run grader + Langfuse calibration Automation Every night, grades the unattended AI workers' runs against a rubric, and asks James to spot-check a sample so the grader stays honest.
  • Two-layer inbox classifier (rules first, LLM judge second) Automation Automatically labels every incoming email as belonging to one of several businesses or a personal area, so every downstream tool sees the same answer.
  • Code-owns-workflow, models-supply-judgment outreach pipeline Automation A program that researches leads and writes draft sales emails for a person to approve, never sending anything itself.
← everything I've built
© 2026 James Washington
github twitter rss