The AI Runtime · FDE Lab
The AI Runtime FDE Lab is a public client for forward deployed AI work: realistic briefs, open labels, deterministic evaluators, and reference builds documented failure by failure.
Free · Open labels · No signup · Reproducible scores
The best deployments happen inside client environments, behind NDAs, under production pressure. The architecture is confidential. The data is protected. The postmortem is privileged. The result is a field with no commons: no shared problems, no comparable results, and no easy way to learn from each other's builds.
So the field keeps rediscovering the same lessons in private: where the model belongs, where deterministic policy must own the decision, what the harness must catch, what auditors will ask, what breaks when a model changes, and what the last few points of quality cost.
Open source solved this for software libraries. It has not solved it for deployment craft, because deployment craft is judgment under constraints. Judgment gets better only against comparable problems. Comparable problems require a shared client.
Every engagement is a problem at Meridian, a fictional financial-services and healthcare group with realistic deployment problems: auditability requirements, cost ceilings, escalation paths, compliance officers, and production failure modes. Data is public where possible, fully synthetic where regulation demands it.
Each engagement covers one core skill the field is building right now. The skill is what you practice. The Meridian constraint is what makes the practice transfer.
The evaluator turns "here is my approach" into "here is my result." When two builds score 74 and 81 on the same labels, the conversation is about the seven points between them.
Problem, labels, evaluator, and starter repo go public. The acceptance bar is stated up front.
The AI Runtime builds against the brief and publishes the debrief: what worked, what broke, what it cost, what changes at 10x scale. Every debrief is a newsletter long-form.
Fork the repo, run the evaluator, beat the reference score. File a field report as a GitHub issue: approach, score, what broke, cost. Notable builds get linked below and credited in the newsletter.
No registration. No cohort. No scheduled calls. The evaluator is always open, and every engagement stays live after it ships.
Engagement 01 is live. The remaining engagements show the planned arc and may change as the Lab learns from each cycle.
Skill focus: Structured outputs and classification agents at scale
Meridian's consumer-complaint backlog is a regulatory exposure. Build the triage system: route CFPB complaints to product category and escalation tier, scored against public golden labels. The reference build ships with a cost report per 1,000 complaints.
The debrief question What did the last five points of F1 cost you?
Skill focus: LLM-as-Judge and evaluation calibration
Meridian's risk team will not accept “the model is good.” Build an LLM-as-Judge over Engagement 01 outputs, then audit the judge itself: agreement with ground truth, position bias, verbosity bias, drift across model versions, and cases where the judge sounds confident and is wrong.
The debrief question When should a judge be trusted, and how do we know?
Skill focus: Document AI and structured extraction
Meridian's health plan is drowning in prior authorization requests. Extract structured requests from fully synthetic clinical documents and route them against payer rules. No PHI. No shortcuts. Scored per field.
The debrief question Where does the model belong in a compliance pipeline, and where must deterministic policy own the decision?
Skill focus: Tool use, MCP, and harness design
Classification was the easy half. Meridian now wants the triage agent to act: pull account context, file escalations, and write back to case systems through MCP-connected tools. Build the tool layer and the harness around it, then score the same agent under two harness configurations, head to head, including across a model upgrade.
The debrief question How much reliability lives in the harness rather than the model?
Skill focus: Context engineering and agent memory
A Meridian complaint lives for weeks across dozens of touches. Decide what the agent remembers, for how long, under what retention rules. Engineer the context so a long-lived case never blows the window or leaks across cases. Then instrument the agent so every run can prove it touched only authorized data and stopped when it should have.
The debrief question What should an auditor be able to reconstruct from an agent run?
Skill focus: Agent observability and LLMOps
The capstone. A degraded system meets a replay of live-shaped traffic. Detect it, diagnose it, remediate it, and file the postmortem in the format regulated teams actually use.
The debrief question What did you detect, what did you miss, and what would have prevented it?
Builds from the field, linked as they land. Beat the reference score or document an instructive failure, and your report goes here.
Readiness reports for AI systems, agent runtimes, and deployment surfaces assessed through an FDE lens.
The evaluator does not care about anyone's title. Clone it, score your build, and put a number next to your name.