Field Challenge 01
Financial Complaint Triage
Build an AI agent that routes consumer finance complaints and flags regulatory risk from messy public complaint narratives.
- Self-serve challenge
- One afternoon
- Real CFPB narratives
- Submit via GitHub issue
You will not leave with a demo. You will leave with a public, FDE-style case study: a triage agent, a deterministic eval run against 40 public labels, and a field report that shows how you scope, build, measure, and explain.
The customer problem
A regional bank receives hundreds of consumer complaints every week. The operations team needs to route each one to the right internal team and catch the complaints that carry regulatory, legal, or customer-harm risk.
Today, routing is inconsistent. Some complaints go to the wrong team. Some high-risk complaints get treated like routine service issues. And some complaints sound severe because of tone, but the underlying issue is operationally routine. Separating those is the job.
What this actually is
A fixed-list triage challenge against a frozen snapshot of real consumer complaint narratives, derived from the CFPB Consumer Complaint Database. You build an AI agent that makes exactly two scored calls per complaint: which team it routes to, and whether it carries a risk flag. Every decision must cite verbatim evidence from the narrative, and the evaluator checks that each quote actually appears in the text.
This is not live monitoring, not a hidden benchmark, and not a leaderboard. The 40 labels are public by design. The point is evaluation discipline, evidence grounding, and failure analysis under a customer-shaped constraint, the same skills FDE and applied AI teams run on.
The dataset
200 real complaint narratives, published by the CFPB with consumer opt-in and scrubbed of personal information before publication. The narratives are messy on purpose: redaction markers (XXXX), typos, all-caps passages, rambling structure. The mess is the challenge, not a bug to clean.
Decisions come only from the cached record. If the evidence is not in the record, it does not exist for this challenge. No outside knowledge about companies or regulators, no live browsing. Treat redacted content as unknown.
The six routing teams
- account_access
login problems, lockouts, closures, identity verification, funds blocked by an account restriction.
- payments_transfers
payments, transfers, wires, ACH, deposits, withdrawals, funds missing in transit.
- fraud_disputes
unauthorized transactions, scams, fraud claims, charge disputes, reimbursement disputes.
- credit_lending
credit cards, loans, mortgages, interest, fees, debt collection, loan servicing.
- customer_service
poor support, delays, unreachable reps, unresolved requests with no clearer product route.
- other
only when none of the above fit.
The risk flag
Flag a complaint when the record suggests serious potential harm or regulatory concern: repeated failure to resolve a fraud claim, inability to access essential funds, foreclosure or severe financial loss, discrimination or fair-lending concerns, privacy misuse, or a company allegedly ignoring required dispute processes.
Calibration rule one. Angry tone alone is not high risk. An emotional complaint can be operationally routine.
Calibration rule two. A short narrative is not automatically low risk. A short complaint can still describe fraud, loss of funds, discrimination, or inability to access money.
The bar
40 hand-labeled complaints with public labels, spread across all six teams and seeded with the hard cases: angry-but-routine, short-but-severe, multi-issue, vague, and heavily redacted narratives. A deterministic evaluator scores your committed predictions against them. Reruns verify, they do not regenerate.
| Metric | What it measures | Why it matters |
|---|---|---|
| High-risk recall | share of truly high-risk complaints your agent flagged | a missed high-risk complaint is the most serious failure |
| Routing accuracy | share of complaints routed to the correct team | wrong routes waste operations time |
| Valid evidence rate | share of evidence quotes that actually appear in the narrative | a decision without grounded evidence is a guess |
How it runs
Step 1: see the evaluator work, no API key, under two minutes
git clone https://github.com/ogkranthi/the-ai-runtime-lab cd the-ai-runtime-lab/ai-runtime-field-challenge-01 pip install jsonschema python run_eval.py --predictions examples/predictions.example.jsonl
Step 2: run the reference agent, then make it yours
Self-serve, in one afternoon. Run the reference agent, score it, then change the prompts, the decomposition, the evidence selection, or the whole approach. The agent is provider-agnostic with Anthropic as the default. AI must do the core reasoning; deterministic code handles orchestration, validation, and scoring.
python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt export ANTHROPIC_API_KEY=your_key_here # run the reference agent on ten complaints, then score it python agent/run_agent.py --input data/complaints.jsonl --output outputs/predictions.jsonl --limit 10 python run_eval.py --predictions outputs/predictions.jsonl # write your report from the template cp REPORT_TEMPLATE.md REPORT.md
- Run the reference agent and the evaluator
- Read your failures against the 40 labels
- Change the approach and rerun
- Write your Field Report from the template
A mini report takes 30 to 60 minutes: run the reference agent, change one thing, report your score and one failure you understood. A full Field Report takes 2 to 4 hours: your own approach, a decision log, a failure table with root causes, and what you would change in v2.
Submit your Field Report
Open a GitHub issue linking your fork, your committed predictions, and your REPORT.md; anyone can rerun the evaluator against them.
What you leave with
A working triage agent. A deterministic eval harness you ran yourself. Measured high-risk recall, routing accuracy, and evidence validity. A failure analysis with root causes. And a teardown that says, not “I built an agent,” but “I built a complaint-triage agent, scored it against a frozen labeled set, and can show you exactly where it fails and why.” That is a portfolio artifact, not a demo.