← back
AI Evals Lab

AI Evals Lab

Live

Maven AI Evals & Analytics certified — Stella Liu & Amy Chen

Started: July 13, 2026 · Course + builds across July–August 2026, alongside the voice-agent eval work

AI:Claude CodeOpenAIClaude
Stack:PythonFastAPIOpenAI APIArize PhoenixSQLite vector storeClaude Artifacts

A deep-dive into AI evaluations, anchored by Stella Liu & Amy Chen's Maven AI Evals and Analytics Playbook course and its certificate. The lab has three layers: the course sandbox — a PDF-grounded chemistry tutor with refusal guardrails and Phoenix tracing (baseline imported from the instructors' repo, extended with a suggested-question chip UI and run against the course's 220-case testsets); three built study artifacts — an interactive 248-field Evals Playbook working document, a Field Manual, and a Study Deck that distill the methodology; and the applied payoff — the voice agent on this site now ships behind a full production eval stack: four split LLM judges, a 97% golden-set gate, Phoenix trace observability, and a hand-label calibration loop.

// screens

AI Evals Lab — screen 1
1 / 4

// highlights

  • Certified: Maven AI Evals and Analytics Playbook, taught by Stella Liu & Amy Chen (2026)
  • Interactive AI Evals Playbook — a 248-field working document covering scope → sniff test → quantify → experiment → launch gate → monitor, with browser persistence and Markdown export
  • Field Manual + Study Deck — two companion guides distilled from the course: a start-to-finish working checklist and a recite-cold retention deck
  • Course sandbox: PDF-grounded chemistry tutor with refusal guardrails and Phoenix OTel tracing — baseline imported from the instructors' repo, extended with a suggested-question chip UI and exercised against the course's 220-case testsets
  • Applied to production: the portfolio voice agent now ships behind four split LLM judges (binary + calibrated style), a 31-case golden set gating releases at 97%, Phoenix trace observability, and a judge-vs-human calibration report
  • Studied the error-taxonomy and test-scenario generator skills from the course's evals repo and folded them into the portfolio's own QA tooling

// takeaways

  • Write the bar before you see the results. Pre-committing thresholds (≥90% golden set, ≤$0.50/session) turns every eval run into a go/no-go decision instead of a rationalization.
  • Binary beats graded. Pass/fail judges are dramatically more stable than 0-10 scores — reserve graded scales for the one dimension where a trend line matters, and calibrate it against your own labels.
  • LLM judges need two things before you trust them: determinism (temperature 0 flipped verdicts stopped flipping) and an instruction-following floor — a small model inverted a negative-constraint rubric entirely.
  • Evals turn 'looks safe' into 'is safe.' The Deal Room's leak gate taught this once; wiring judges, golden sets, and calibration onto the voice agent made it a habit instead of a story.