A note on timing: this happened in September 2026, on Inspect AI 0.3.259 through 0.3.269.

This is a companion to our series of posts, where Ryan Hansz and I walk through the eval behind our SNAP results. Partway through that work, a whole family of models stopped working, and the fix ended up being in Inspect itself.

Inspect is free, open source software from the UK government’s AI Security Institute for testing AI models: you give it a set of questions with known answers, it asks each model, and it scores how often the model gets them right.

Every request crashed

When I first ran the full set of frontier models through Amazon Bedrock, OpenAI’s GPT-5.6 models (Sol, Terra and Luna) crashed on every single request.

These models return their reasoning as an encrypted blob called redactedContent, never as readable text. That’s on purpose, and it’s a valid response shape in AWS’s API. But Inspect’s Bedrock provider only expected the readable shape, and its pydantic model marked the readable field as required. So every response failed validation before Inspect could even parse it:

ValidationError: 2 validation errors for ConverseResponse
output.message.content.0.reasoningContent.reasoningText
  Field required [type=missing, input_value={'redactedContent': b'rsn...'}, input_type=dict]

A raw aws bedrock-runtime converse call showed the API was fine. The bug was in Inspect. Reading the provider source (inspect_ai/model/_providers/bedrock.py) took about 10 minutes and saved me an afternoon of guessing.

Issue first, then the pull request

Inspect has an issue-first rule for outside contributors. You open a GitHub issue, a maintainer marks it accepted, and only then can you open a pull request. So I patched my local install to keep the eval moving, and then filed issue #5217 on September 3 with reproduction steps and a proposed fix.

One change became four

The fix ended up bigger than I planned. I’d scoped one change, and the merged pull request had four:

1. Parse the encrypted block instead of crashing.

2. Send the encrypted bytes back to the model on the next turn. Parsing alone fixed turn one, but it broke every multi-turn and tool-using eval at turn two. That’s most of what we run.

3. Do the same in the streaming path, so streamed and regular calls behave the same.

4. Swap in a placeholder for the bytes when writing the raw response to the log. The field is binary, and non-text bytes break the log’s text encoding.

It took three rounds of review with the maintainers. They caught things I had missed, including a crash I had found and then talked myself out of because my test data wasn’t realistic. Pull request #5299 merged on September 25 and shipped in Inspect 0.3.269. If you run OpenAI reasoning models through Bedrock in Inspect, they work now.

The review also turned up three related issues I filed afterward: #5300, #5301 and #5302.

Measuring AI on benefits questions only counts if the measuring tool is open and works for everyone, so we fixed the bug upstream instead of keeping a private patch.

What I’d take from it

• When a provider crashes, make the sample, raw API call yourself. It tells you in one command whether the bug is upstream or in your code.

• Read the provider’s source before you blame your config.

• Test the second turn too. A fix that parses a response can still break the conversation that follows.

• Patch locally to keep moving, but file the issue the same day while the repro is fresh.

It’s a small fix, and I’m still pretty happy that a SNAP eval led to a patch in the AI Security Institute’s own framework.

Through this work, we’ve scanned AI governance in all 50 U.S. states plus D.C., read 35 AI policies at different levels and scored them each against 25 policy provisions, built case studies of different states with AI policies, and held conversations with state agency leaders doing the actual work to find out what failed in practice. We landed on the finding that AI guardrails in public benefits should start at the agency level. By agency level, we mean two administrative layers: statewide IT or CIO policy and the benefits agency’s own policy. In the policies we reviewed, both layers were more specific than legislation or executive orders, and neither has to wait for a new law to act.

Here are three reasons we came to that conclusion.

1. Shadow AI is already happening, and the agency is held responsible for it

In December 2023, a child protection worker in Victoria, Australia discreetly used ChatGPT to draft a report for the Children’s Court. The generated report raised some concerns, as it was found to downplay the severity of potential harm to the child and had impacts on decisions about the child’s care. This sparked further investigation by the state’s privacy regulator who found that nearly 900 department employees – almost 13 percent of the workforce – had used ChatGPT in the prior six months.

The final investigation report blamed the agency, finding that the Department of Families, Fairness and Housing (DFFH) failed to “take reasonable steps to ensure the accuracy of personal information and to protect personal information from unauthorised disclosure.”

While that case is about child protection in Australia, the same pattern is already showing up in the US. One report, a survey of 3,335 public servants across 10 countries including the US, found that in “low-enablement environments,” 64% of enthusiastic AI workers report using personal logins, and 70% report using AI for work tasks without their manager knowing. Additionally, journalists in Washington State documented this pattern when they obtained thousands of pages of ChatGPT conversations from local officials: constituent replies, policy documents, and grant applications – none carrying the state-required disclosure, and many from personal accounts.

While we haven’t seen a records request on a SNAP or Medicaid eligibility office, we’d expect it to find a similar outcome: a caseworker using a personal login for an AI tool to transcribe their lengthy interview, draft their case notes, or answer a tricky policy question.

When staff don’t have clear guidance and approved tools, AI use doesn’t stop. It moves to where it can’t be tracked, evaluated, or made safe. We understand caseworkers are under immense pressure from changing policies, staffing shortages, and increased administrative burden, and we know why they’re turning to AI. But for tools that touch benefit decisions, this kind of risk can’t be tolerated. The Australia case made it clear that agencies can still be held responsible for AI outcomes, even when they don’t act.

Giving staff an approved alternative is part of that responsibility, but a list of tools alone isn’t enough: if no one owns the approval process or staff have to wait indefinitely for an answer, the list functions like a ban and pushes experimentation back toward personal accounts.

2. Agency policies can survive legislative churn

Legislation takes multiple cycles and can end up weaker than it started. Colorado passed the country’s first comprehensive state AI law in May 2024. Its original requirements were set to begin in February 2026, then delayed until June while lawmakers negotiated a compromise. In May 2026, lawmakers replaced it with a narrower framework: some provisions took effect immediately, while the remaining requirements begin in January 2027. That is two years of legislative revision during which the original law never took effect, while AI capabilities continued to change rapidly.

New York shows what the agency layer can do when a statute is weakened. The legislature rewrote the LOADinG Act in 2025, before it took effect, and dropped its human-review requirement. The state’s own AI policy, updated in June 2026, still says “Automated final decision systems are not permitted,” and names benefits eligibility in scope. It’s statewide IT policy rather than law, making enforcement penalties slightly weaker. However, that guardrail survived where the statute didn’t.

The federal litigation task force created in 2025 targets state AI laws, while the broader federal strategy expressly preserves requirements governing a state’s own procurement and use of AI. Agency procurement terms and internal-use rules therefore remain a separate layer of protection.

3. Agency policies contain specific rules and review requirements

Agency policies already carry the most specific rules. We scored 35 policies on how many of 25 common rules each one actually requires (note: these are selected policies, not a representative sample). The rules we looked for included things like required pre-deployment testing, prohibited use cases, and impact evaluations.

Statewide IT policies contained on average 9.2 out of 25 provisions, and SNAP agency policies 7.0, compared with 5.8 for state statutes and 1.0 for executive orders. Prohibited uses, data classification, and AI inventories all show up at the agency level more often than in statute – and these are some of the provisions we think are the most useful for protecting beneficiaries.

The agency layer is also where policies actually get revised. Of the 150 policies we checked, only 42 percent require their own review with a real cadence, owner, or trigger. Another 43 percent say something like “may be reviewed periodically,” which functions the same as saying nothing at all. State agency and IT policies carry a real review requirement more often than any other layer of policy.

But a feedback loop has two halves, and the second one is missing almost everywhere. None of the 10 state agency policies we scored requires an outcome evaluation of the AI it governs, or a way to feed evaluation results back into the policy. That’s one recommendation we added into the model agency policy we’re building.

The agency is where a definition of AI meets real tools. Defining “artificial intelligence” and deciding which tools need to be governed by the policy has been one of the biggest challenges we’ve seen states struggle with.

• Are we only governing generative AI?

• Does that include traditional machine learning or automated systems the agency already uses?

• Which use cases do we need to be tracking more closely?

• How do we prevent overburdening our staff with an approval process?

These are real questions states are grappling with, and ones we hope to help states think through in the model policy guidance we’re writing.

A statutory definition is not enough to produce a complete inventory. Agency policy can pair the definition with clear ownership, reporting procedures, and a current inventory of the tools staff actually use.

New York shows why. Its state law had one of the broadest definitions of AI of any policy we reviewed, but its first public inventory still returned no use cases from SNAP or several other agencies with high-profile AI tools already in production. The state law included a requirement to inventory all AI tools that “directly impact the public.” While they attempt to define the terminology, the law lacks useful operational detail, which may be one reason agencies failed to report their tools.

That is why a benefits agency still needs its own policy when a statewide one exists: the statewide policy sets the groundwork, and the benefits agency applies it more specifically to its programs, and keeps the inventory current.

Limitations of agency-level policy

Agency policy is not a substitute for law, particularly when it comes to claimant rights. The statewide policies in our latest research were much stronger on how agencies manage AI than on explanation, appeal, or opt-out rights. Agencies can test operating rules now; legislatures still have a role in codifying protections that should not depend on which administration is in office.

Agency-level policy as a legislative testing ground

We think states should start governing at the agency level, now. Not instead of legislation, but ahead of it. An agency can write and revise its guardrails on something closer to the timeline the technology actually moves on. The ineffective guardrails can be removed, and what works well can become a candidate for state legislation.

In the two states where we scored both a statewide policy and the benefits agency’s policy, the statewide policy was the more detailed of the two. This suggests a benefits agency doesn’t have to start from scratch; it can build on what its state IT office already requires and add the rules specific to its programs.

What’s next

We plan to present more of our thinking at Nava’s Open Source Summit: AI Futures on October 14.

We’re also using this research to build a model agency AI policy that agencies can adapt now and revise as the evidence changes. We especially want to hear what happens after a policy is adopted:

• Which rules hold up

• Which create work without reducing risk

• Which ones staff route around

• What would trigger a revision to your policy

If your agency is drafting an AI policy, or has one that isn’t doing what it was meant to, comment below or email us at labs@navapbc.com.

Intro

In our first post, we shared how models performed on a set of 500 SNAP (CalFresh) scenario questions. Here, I’ll look at the questions they got wrong most often and the experiments I tried to improve their answers.

In the original runs, GPT-6 Astra led the models we tested with an average accuracy of 92.2%. That still meant about 39 of 500 answers were wrong according to a key created by SNAP experts. Let’s dive in to the errors and how we might improve accuracy.

Our testing dataset focused on California questions, so you may see SNAP referred to as CalFresh (California’s name for the program.)

Error Categories

To begin, I wanted to see if any topics or types of questions caused the most errors. The dataset creators, Better Government Lab, tagged each question in the dataset with a SNAP quality control topic. This let us see where errors clustered in LLM responses. The categories with high errors were fairly consistent across models, though each model had its own highs and lows. I saw that four categories (self-employment, transitional benefits, social security numbers and citizenship and noncitizenship status) had the most errors.

Note: Some categories in the dataset only have a few questions, so one answer can cause a large swing in accuracy.

Why are LLMs getting questions wrong?

As LLMs improve, mistakes can be harder to untangle. In the frontier models we tested, we did not see clear examples of hallucinated policies or arithmetic errors. Instead, we saw subtle misreadings of wording and incorrect assumptions about households or income.

As an example, “for benefits,” “for the purpose of assistance,” and “application for assistance” can all lead to different correct responses to a question.

Other errors reflected outdated policy information (the models had no web access in our first tests) or reliance on general rules when more specific ones applied.

Error cause: not aware of policy updates

These were policies that had changed after the models had been trained. For example, non-citizen eligibility policies changed in the last year. Since we didn’t allow web-search in our initial testing, models had no way to see new information about the rules.

Error cause: relying on general policies when more specific rules applied.

• The model didn’t surface or prioritize California policies that override general policies.

  • For example, SSN requirements for newborns or verification requirements for students.

Error cause: not applying the correct role for a household member

Some questions relied on understanding the difference between being reported on an application, having income or resources counted, belonging to a CalFresh household, and receiving benefits. Assuming a different scenario led the AI to answer questions incorrectly.

Error cause: Nuances in income + benefit calculations

Determining how to count income and expenses for SNAP is a complex process. LLMs made errors in this category when they incorrectly assumed:

• Whether an amount already had the earned income deduction applied to it

• Whether an amount in a question was “gross” or “net” income

• How to count one time (excluded) vs. recurring payments (included)

• If an income amount was “reasonably anticipated” when calculating future income amounts

• How to apply rules about self-employment expense deductions. Like whether a meal at a business conference is a deductible expense.

Experiments to improve results

Experiment #1: Add web search and increase reasoning level

This is the most straightforward change I could make. The experiment assumed that if models had more reasoning budget and access to newer resources, then they would improve accuracy, especially on questions with recent policy changes.

• I tested two OpenAI models: GPT-6 Sol and GPT-6 Astra.

• Changing reasoning effort and adding web search without specialized guidance produced modest overall results. Citizenship and noncitizenship questions showed a clearer improvement.

  • As expected, this category benefited most from access to current sources because its policies had recently changed.

    

Experiment #2: Improve the system prompt

I reviewed questions GPT-6 Astra consistently got wrong in the dataset, wrote notes on the mistakes, and turned those notes into general guidance for later tests.

GPT-6 Astra with the revised prompt averaged 491.7/500 (98.3%) across three runs, up from 461/500 (92.2%) at baseline."

Baseline prompt

You are answering a SNAP scenario benchmark question.
Return only JSON: {"answer":"one multiple-choice letter","reasoning":"brief explanation","review_required":true|false}.
Use only facts in the scenario. Do not represent your response as a final benefits decision or legal advice.

Types of instructions I added

Many of the instructions I added clarified terms or expected processes that had led to errors. A few examples are:

• For questions determined to be about immigration or SSN requirements, the instructions told the model to use web search to find up to date policies since these topics have had recent changes.

  • For topics with California specific policies, I instructed the model to check current CDSS guidance

• For income and deductions:

  • Gross income is before business expenses or household deductions; net income is after applicable deductions.

  • A benefit-increasing income decrease or expense claim generally needs verification before raising benefits; one lower payment or higher expense does not by itself establish a lasting reduction.

  • Treat an amount described as earnings as gross unless called net or after deductions.

  • Over-the-counter medicines and vitamins require prescription or approval by a qualified health professional

  • Use submitted official records to resolve a conflicting verbal estimate when those records already establish the amount and frequency.

• For household composition:

  • Inclusion in the household assessment is distinct from an individual’s eligibility to receive benefits; an ineligible noncitizen’s income and resources can still matter.

  • Do not infer that co-residence makes unrelated people a food household.

• And many more pieces of guidance and clarification for each category of error

Limitations

• The changes I made to the prompt were designed specifically for this set of 500 questions.

  • This type of focus could lead to “overfitting” where the model will perform well on this set of questions, but do worse in the real world.

  • Further testing on new scenarios and different phrasings could show whether this guidance holds up.

• While I am familiar with CalFresh and SNAP policies, I am not a policy expert, so some of the guidance I gave the tool would need to be verified before it would be used broadly.

What might this mean when developing AI tools?

An overall score can hide errors in the questions that matter most to people. Having each question assigned to a quality control category allowed us to have a better picture of how models perform in areas that matter to beneficiaries and staff. Like checking eligibility, identifying who belongs in a household, verifying an expense, or calculating income.

Based on this experience, here are a few suggestions for teams building AI tools for public benefits:

• Review the trickiest errors with program experts. Some questions were straightforward to resolve, such as pointing the model to current citizenship rules. Others required understanding when staff request verification and when they can accept a client’s statement.

• Clarify ambiguous phrasing in people’s inputs and the tool’s output. If a detail changes the answer, such as whether income is gross or net, the tool should ask before proceeding.

• Make sure the model’s approach fits its audience. A public-facing tool should explain that people can apply for SNAP even if they are unsure whether they qualify. A caseworker-facing tool needs more precise guidance on process and verification to support staff decisions.

• Be ready for a “whack-a-mole” improvement process. Guidance that helps with one topic can cause mistakes elsewhere. When I added broad guidance to verify evidence before using it, the model did worse on questions where verification was needed only if a claim was questionable.

We built it on the Vercel AI SDK with a hand-rolled agent loop: streamText with stopWhen: [stepCountIs(500)], a custom context compressor, a 900-line composed system prompt, and a readReference tool for pulling in documentation on demand.

We swapped that hand-rolled setup for Vercel’s Eve, a pre-built agent harness, without changing what the prompt was achieving. The result: costs dropped anywhere from 44% to 85%, depending on the application and model, with fewer errors and faster runs along the way. Here’s what a harness actually is, what ours was missing, and what the numbers looked like once we made the switch.

What is a harness in an AI application, and why is it useful?

A harness is the core of an AI application. Think of it as scaffolding that wraps around a raw model and turns it into something that can actually get work done, not just answer questions in a chat box. It takes in a request and routes it through the right tools to complete it: running code or browsing files, keeping track of memory and context over a long task, sandboxing risky actions, pausing for human approval, and handling retries and logging along the way.

A handful of pre-built harnesses already do this well — Claude Code, Codex, Cursor, and Vercel’s Eve among them. We picked Eve to test against our own hand-rolled setup, and here’s what changed.

What we saw with a harness vs. without one

To see this play out, here’s what happened when we tested against real-world benefit applications — BenefitsCal, as one example: a large, multi-page SNAP benefits form — with a harness and without one.

Before Eve

The Form-Filling Assistant started with a single prompt to manage the agent; it grew past 900 lines as we ran into more types of forms it needed to handle. The prompt was very case-specific, tailored to make sure the benefit applications in Riverside, CA could be completed seamlessly. However, we could only reliably handle three applications, and even those only ran error-free most of the time.

On top of that prompt, we built a custom set of tools, memory storage, and compaction, and tested each skill individually to try to make the agent more reliable. It got messy fast: as the prompt grew, it got harder to predict what would break, or when. We also leaned heavily on Opus 4.7, which was expensive, and every model update risked breaking the prompt.

Why we added Eve

We brought in Vercel’s Eve harness to target three problems directly:

• Memory storage and compaction

• The ability to switch to less expensive models

• Error handling and testing

After Eve

The clearest change showed up in the bill, but it started with a structural fix: we no longer have to manually wire together skills, tools, and the main prompt — Eve handles that connection natively.

1. Cost dropped substantially. Smaller applications that used to run up to $5 now cost about $2. Larger applications, like BenefitsCal, went from roughly $50 down to $18.

2. Run time dropped too. On our largest applications, run time went from 34 minutes down to 13.

3. Memory got seamless. Built-in memory worked reliably and didn’t lose data mid-run. That held true for compaction too — Eve doesn’t need a custom compaction implementation, so the agent moves through memory and compaction without issue. Without it, we regularly saw applications stall or error out at exactly that step.

4. Skills became reusable. The agent reads skill files as needed instead of loading every skill into the first prompt. We used to struggle to manage skills effectively; with Eve, the agent pulls in each skill only when it’s appropriate. On our longest application, I ran it multiple times with Eve and it completed every time. The same application on our original, non-Eve prompt kept erroring out.

5. Modals stopped being a bottleneck. We used to get stuck on modals often — Eve moves through them cleanly, mostly because it knows when to call the right skill or tool.

The BenefitsCal numbers

Without Eve, we couldn’t get Sonnet 5 to complete a run at all — we had to fall back to Sonnet 4.6, and even then needed two forced continues to get past memory and compaction, at a cost of $52.99 and a runtime of 34 minutes. With Eve, that same Sonnet 4.6 run completed with no errors, at $18.67 and 13 minutes.

Smaller applications: the cost

BenefitsCal is one of our largest applications, so we also wanted to check whether the cost savings held up on smaller ones, across multiple models, before committing further to harness testing.

WIC (Women, Infants, and Children) — the smallest application we support:

• gpt-5.4-mini: $0.13 without Eve → $0.02 with Eve (about an 85% reduction)

• Sonnet 4.6: $2.76 without Eve → $1.10 with Eve (about a 60% reduction)

• Opus 4.8: $3.40 without Eve → $1.90 with Eve (about a 44% reduction)

• Haiku 4.5: $0.62 without Eve → $0.25 with Eve (about a 60% reduction)

WIC was the easiest application we tested — no errors, and no need for the memory tools — but the cost reduction held up across every model we tried.

IHSS (In-Home Supportive Services) — a one-page application, but an older one:

• Sonnet 5: $5.39 without Eve → $2.40 with Eve (about a 55% reduction)

• Haiku 4.5: $2.10 without Eve → $0.95 with Eve (about a 55% reduction) — though both runs completed with errors, not cleanly

Haiku’s IHSS result comes with a caveat worth calling out: with or without Eve, it couldn’t fill in masked or numbered fields — state, SSN, birthdate, phone number — so that specific gap looks like a model limitation, not a harness one. Eve still helped elsewhere on that same run: without it, Haiku also missed an entire “check any that apply” section and left Submit enabled despite the missing fields; with Eve, neither of those additional issues showed up. Sonnet 5 didn’t have this problem at all — with Eve, it took longer to analyze the page but filled out every field correctly, masked ones included.

What this could unlock for us

Eve took away a lot of the pain we hit while building the application. We spent a lot of time just refining the agent to handle our three main applications reliably. With Eve handling those core pain points, we can put more of that time into the user experience instead.

A few directions we’re now considering:

• Multi-form applications

• Audio AI interaction

• A possible Chrome extension

• Moving away from a chat-based interface

Also on our radar: Vercel’s AI SDK now has a HarnessAgent API that lets you swap between harnesses the same way you’d swap between models — write the agent once, and change the harness underneath it later. Still early, but worth testing.

The takeaways

Overall, a harness has proven to be the right way to build an AI application. Working with an AI model is powerful, but without a good harness, it comes with unexpected issues and flaws that are hard to predict — that’s just the nature of LLMs.

Once we let Eve take over the parts we’d been hand-rolling ourselves, the same models did the same work for 44% to 85% less, with fewer errors and faster runs. For an agentic application running at any real scale, that’s not a marginal win — it’s the difference between a project that’s worth scaling and one that quietly bleeds budget.

SNAP helps millions of people get food. The program is hard to navigate: rules about who is eligible, who belongs on an application, and which deductions a household can get add up to a lot of ways to get something wrong.

Applied well in this space, AI tools can increase application accuracy, cut administrative burden, and get people food faster. Done poorly, they can discourage people from applying, create more work for staff who have to fix AI’s mistakes, and drive up payment errors, which are now tied to how much funding states get for SNAP.

All of these use cases hinge on how well AI models perform on fundamental SNAP questions. This post walks through how we evaluated frontier and open source Large Language Models (LLMs) on 500 questions about California’s SNAP program, known as CalFresh. Many of the questions would be applicable to nationwide SNAP questions. Future posts will detail how prompt, harness, and context engineering can continue to increase precision.

The dataset is available on request and we used open-source tools, so anyone can repeat this approach. The question scenarios were developed by Better Government Lab in partnership with SNAP policy experts. See more about the dataset in the appendix.

Sample question (illustrative)

Preamble
Age: Young adult (below 24)
Gender: Male
Employment status: Employed by other
Healthcare/Disability status: Not legally disabled
Citizenship status: U.S. citizen

Question
The client is a 20-year-old enrolled full-time, year-round at a California community college. He lives with two roommates and buys and prepares his own food. He works an average of 22 hours per week at a grocery store. Does he meet the CalFresh student eligibility rules? [Select one correct answer.]

A. No, full-time students are not eligible for CalFresh
B. No, he must live with his parents and apply with them because he is under 22
C. Yes, because he works an average of at least 20 hours per week
D. Yes, but only if he is approved for work-study

Answer (reviewed by SNAP policy experts)
Answer: C

How the models performed

Last updated: Sept. 22, 2026

Accuracy

• GPT-6 Astra scored highest, averaging 92.24% across three test runs.

• That’s impressive! But this still means the LLM got around 39 out of 500 questions wrong. Depending on the question, one of those incorrect responses could have large impacts.

Costs

• Costs per 500-question run varied considerably (Fable’s $5 compared to Gemini’s $0.49).

• While costs were low for our runs of 500 questions, this costs would grow linearly as usage grows, unless questions are batched and cached. Costs would also be more expensive for back-and-forth conversations, as opposed to our test which constrained models’ responses to a single multiple-choice response.

Speed

• Time to respond varied as well: Sonnet 5 was fastest, with a 1.8-second median response, compared to over 20 seconds for some local models.

• Open-weight models took longer to respond, but speeds vary depending on the hardware running the model. The open models in this test were run on a Mac Studio M3 with ~819 GB/s unified memory bandwidth. Most of these local models could be run on a modest MacBook Pro or other consumer hardware.

Other observations

• Open-weight models performed well, with Gemma 4 31B scoring highest at 85.8%. But they didn’t match the accuracy or speed of the frontier models.

• Newer, larger models generally scored higher than their predecessors (GPT-6 outperformed GPT-5.6, for example), with one exception: Claude Opus did slightly better than Fable. A surprise was Gemini 3.8 Flash, which had a great balance of accuracy and low cost, but was slower to respond than other models.

AI made the most mistakes in 4 categories

The dataset creators, Better Government Lab, grouped their questions using the same error categories as SNAP quality-control reviewers. When we looked at how models did across those categories, we saw most struggled in the same four categories. Questions about citizenship status, self-employment, transitional benefits, and Social Security numbers had the most errors. In a future post, we’ll dive into these categories and questions to see what may be causing the higher error rates.

What’s next

Our next post will explore the high-error questions and categories to explore what types of questions cause LLMs issues, and what might be done to reduce those errors.

Multiple-choice questions are only a starting point. Building on our work with Form-Filling Assistants, we plan to measure the speed, accuracy, and cost of AI agents while completing government forms.

Appendix 1: The dataset

We used Better Government Lab’s SNAP Scenarios dataset to measure AI models. Better Government Lab is a Nava Labs partner and worked with us evaluating an assistive chatbot for caseworkers. They describe the dataset as:

…a detailed 500-question benchmarking dataset for measuring LLM or human-in-the-loop accuracy on realistic scenarios related to the U.S. Supplemental Nutrition Assistance Program (SNAP) in California...Questions were generated based on frequencies of types of errors from publicly available national quality control data, which pertains to the accuracy of SNAP benefit amounts.

Source: https://huggingface.co/datasets/cornellbgl/SNAP_scenarios

The questions are equally crucial to caseworkers and program staff who are under pressure to reduce their payment error rates in order to avoid cuts in already strained funding. Better Government Lab also worked with SNAP policy experts in California and Georgia to review and validate the set of questions to ensure they’re both accurate and relevant.

This round of testing was focused on California’s SNAP program. Many questions are consistent nationwide, but some rules vary by state. Results here shouldn’t be read as a measure of how models handle SNAP nationally.

Note: Any example questions and responses in this post are realistic, but not pulled directly from the test set, to avoid having the answers on the open internet.

Appendix 2: Full results

Appendix 3: Our testing setup

• Tools: We used Inspect, which is described as “an open-source framework for large language model evaluations…developed by the UK AI Security Institute and Meridian Labs.”

  • This let us set up test conditions and run LLMs against the scenario dataset reliably and quickly.

• Model setup: For all of the models we tested, we set reasoning to “low” or equivalent settings, and turned off web search and other tools.

  • Our aim was to provide a baseline measure of performance with settings similar to what may be used in chatbots or other tools for staff and the public.

  • We ran each frontier model 3 times to account for variability in the LLM’s responses.

  • Since local models had to run sequentially on someone’s machine, we only ran one test round for those.

In May 2026, we decided to open source our conglomeration of Caseworker Empowerment Tools.

With this work, Nava Labs maintains rigorous, multi-layered generative AI guardrails to ensure everything we ship meets the needs of caseworkers serving real people. We use strategies like automated schema enforcement (i.e. Zod), deterministic processing (simple function calls), observability with Open Telemetry standards, and persistent logging to ensure we can trace requests and responses from start to finish. Most critically: every tool allows for a “human in the loop” at various review gates to ensure safety and accuracy.

Our commitment to safety and technical integrity of rolling out these solutions is non-negotiable.

Why guardrails matter

Generative AI can fail convincingly. An answer can sound useful while creating or citing the wrong policy, exposing sensitive data, or taking an action the user never intended. We’ve all been there when we slightly push back against Claude or ChatGPT and receive a you are absolutely right message (see: Stochastic Parrot).

In public-benefit work, those failures could cause irreparable harm to caseworkers and the people they intend to serve. That is unacceptable.

Guardrails belong throughout the entire software development lifecycle.

How we embed AI guardrails across the software development lifecycle

Guardrails aren’t an afterthought or a quick prompt adjustment — they belong woven directly into the software development lifecycle. Here is how Nava Labs embeds safety, control, and observability from initial design to active maintenance:

1. Plan by setting hard boundaries

Before writing a single line of code, we define the system’s exact perimeter. Nava Labs does this by explicitly mapping what data the model can see. What state changes it can trigger. And which high-stakes decisions an automated tool is strictly forbidden from making.

2. Build by coding for least privilege

We don’t rely on the model to “behave” itself. We attempt to enforce this behavior in code. We do this by validating every input and output with strict type enforcement (like Zod schemas), sandboxing untrusted web content inside virtual machines, scoping retrieval strictly to vetted policy manuals and documents, and granting tools the absolute minimum access they need to operate.

3. Test by breaking things on purpose

Models fail in weird ways. So, we actively try to trip them up before caseworkers ever touch them. We can do this by running golden-set benchmarks, monitoring for model drift over time, and running aggressive security stress-tests. Specifically, targeting direct and indirect prompt injection attacks (including sneaky aria-tag injections hidden in scraped websites or documents).

4. Operate by watching the wires in real time

Shipping to production is just the start. We maintain total visibility by tracing every end-to-end request. Typically by using OpenTelemetry standards, tracking real-time failure modes, and keeping a fully tested, one-click rollback path ready if a model update acts up.

5. Review by keeping humans in the loop

At the end of the day, the machine is just a tool helping an individual. We review what is proposed by surfacing code-verified citations, confidence scores, and missing-field alerts directly to caseworkers, ensuring human navigators always have the context, time, and explicit authority to approve, correct, or shut down a workflow.

We released our Caseworker Empowerment Toolkit as open source on GitHub so that other teams can inspect, challenge, and improve these patterns with us. Feel free to dive in!

Questions to keep in mind when building with generative AI

These questions below are really for teams building with and implementing generative AI to think about throughout the entire delivery lifecycle.

• What can the model see, and what can it change?

• What happens when a retrieved document or webpage contains malicious instructions?

• Can we trace every important claim or action to its source?

• Which failures stop the workflow instead of triggering another model call?

• What tests run when the model, prompt, data, or tools change?

• Who can disable or roll back the system when something goes wrong?

• Does the human reviewer have the evidence, time, and authority to disagree?

• Etc.

Have you been doing any work on implementing LLM guardrails? We’d love to hear about it in the comments!

Government agencies and processes already work this way. Think about how a benefits application moves: from an intake caseworker to an eligibility technician to a supervisor. Or background checks: criminal records, credit, employment verification, and education verification — running concurrently then combined at the end.

The handoffs, the specialization, the escalation to a human with signature authority: that architecture already exists. Often it is written into statutes, regulations, standard operating procedures, and so on. Similar to legacy software being globbed onto over the decades.

I’ve found myself wondering about:

• what pattern(s) government employees, processes, and technologies already use

• what breaks when you swap a human with decades of experience for an AI agent

• how we effectively measure and evaluate the AI agent and outcomes

• how we combat automation bias and rubber-stamping outputs

• and so on…

Let’s explore the eight core patterns from the Google post and how to apply them in a government context:

1. Sequential Pipeline

2. Coordinator / Dispatcher

3. Parallel Fan-Out

4. Hierarchical Decomposition

5. Generator and Critic

6. Iterative Refinement

7. Human in the Loop

8. Composite Patterns

The legal constraints that change the architecture

Before getting into the AI agent patterns, I want to highlight six requirements rooted in U.S. federal law. These six requirements shape which patterns are safe where and how they can be applied. Each requirement has some architectural consequences.

1. Every handoff is a federal record. Under the Federal Records Act, agency decision-making must be preserved and auditable. In a multi-agent system, the state passed between AI agents is part of the decision record. If an eligibility agent writes a determination recommendation to session state and a downstream agent overwrites it, you have destroyed a record. This makes ADK’s output_key discipline part of compliance control.

2. Adverse determination recommendations require an explanation the claimant can appeal. A person that is denied benefits gets told why, in plain-language terms, specific enough to contest. An AI agent chain that produces a correct recommendation with no traceable reasoning is a liability — even when the answer it provides is correct. Every pattern below needs to be able to answer: can you reconstruct which AI agent concluded what, and on what evidence?

3. Plain language is a statutory requirement. The Plain Writing Act of 2010 requires covered documents to be written clearly. Any AI agent generating citizen-facing text needs a critic checking reading level.

4. Section 508 applies to generated output. If an AI agent produces a PDF, a form, or a web response, accessibility is a requirement. Generated output needs an accessibility check in the loop.

5. The Privacy Act limits what crosses AI agent boundaries. Records in a system of records have use limitations. When a coordinator routes a case to three specialist AI agents, you have potentially disclosed personally identifiable information (PII) to three contexts. AI agent boundaries are disclosure boundaries. I look at this as the Principle of Least Privilege.

6. ATO boundaries are system boundaries. An AI agent touching a FedRAMP-authorized system and an AI agent touching an unauthorized one cannot share state. The authorization boundary constrains the implementation.

Keep these six constraints in the back of your mind while reading the patterns below. They help determine which ones you can deploy and which need a human in the loop.

1. Sequential pipeline

Government processes are often sequential with defined handoffs.

Where this could fit:

• FOIA request handling: Request Intake → Classification → Records Search → Redaction → Review → Response Generation

• Permit applications: Application Parser → Code Compliance Checker → Environmental Impact Reviewer → Fee Calculator → Approval Generator

• Benefits application processing: Intake → Eligibility Recommendation → Documentation Review → Approval → Notification

ADK’s SequentialAgent runs sub-agents in order. Each writes to shared session.state under an output_key, and the next AI agent reads it.

2. Coordinator / dispatcher

A coordinator AI agent reads intent and routes to a specialist, then steps out of the way.

Where this could fit:

• 311 citizen service centers routing requests to Trash and Recycling, Permits, Tax Questions, Public Safety, Parks & Recreation

• Health system navigation routing to Benefits, Healthcare Scheduling, Prescription Management, Claims Status

• Taxpayer assistance routing to Filing Questions, Payment Plans, Audit Support, Tax Law Interpretation, Refund Status

As you can see, routing runs on LLM-driven delegation. You define a parent with sub_agents, and ADK’s AutoFlow transfers control based on each child’s description.

My gut tells me that the Privacy Act constraint mentioned above applies at the routing decision. The coordinator needs only enough information to route and no more.

3. Parallel fan-out and gather

Independent checks run at the same time, and a synthesizer combines them.

Where this could fit:

• Background checks: Criminal Records, Credit, Employment Verification, Education Verification, running concurrently

• Building permit review: Structural Safety, Fire Code, Electrical Code, Plumbing Code, Zoning Compliance, reviewed simultaneously

• Grant application screening: Eligibility, Conflict of Interest, Budget Review, Technical Merit, concurrent

ParallelAgent runs sub-agents at once, then a downstream AI agent reads all their outputs. Two things of note:

1. Having many parallel AI agents running in separate threads with a shared state will create race conditions.

2. Do not let the output consolidator flatten the findings. For example, do not remove cited fire code specifics; include why something passed or failed with grounded information.

4. Hierarchical decomposition

A parent delegates part of a task, waits for the result, and continues its own reasoning. While this may seem the same as the sequential routing example above, this one is a bit different. The hierarchical parent gets an answer back and keeps working.

Where this could fit:

• Environmental impact assessment: a project coordinator AI agent delegates to Air Quality, Water Resources, Wildlife Impact, and Cultural Resources specialists, then writes an integrated assessment that reasons across all four

• Federal procurement review: a procurement officer AI agent calls Federal Acquisition Regulation Compliance, Small Business Set-Aside Analysis, and Price Reasonableness, then makes an award recommendation informed by all three

• Policy research: a lead AI agent calls a research assistant that manages Federal Register search, case law lookup, and Government Accountability Office report retrieval, then drafts against what came back

My understanding is that you can wrap a sub-agent in AgentTool to make this work. The parent calls the sub-agent’s entire workflow as a single function call and receives the result.

5. Generator and critic

A good rule of thumb that I like to follow when creating agentic systems is to split creation from validation. A generator drafts output, a critic checks against explicit criteria. Failures route back with specific feedback until the draft passes. This is also known as a Ralph Loop or setting a Goal in Codex.

Where this could fit:

• Guidance documents: the critic validates against the Code of Federal Regulations, plain-language standards, and Section 508 accessibility requirements

• Public notices: the critic checks accuracy, required disclosures, and reading level

• Grant scoring: the critic validates against the published rubric and flags scoring inconsistent with it

This is the pattern that statutory constraints demand. The Plain Writing Act and Section 508 are “pass/fail” criteria, which is exactly what a critic is for.

Since this is essentially a where loop, you should have explicit and enumerated criteria for the critic to hit, along with escalation paths or failsafes (like max_iterations) to ensure that the loop does not run indefinitely.

6. Iterative refinement

The same loop shape aimed at improvement rather than correctness. Generator drafts, critic writes optimization notes, refiner rewrites.

Where this could fit:

• Rulemaking: the critic identifies legal ambiguity and economic impact and the refiner incorporates comments

• Congressional reports: the critic checks factual accuracy, citation quality, and balance

You should be cautious with this approach since the refiner will overwrite text on each pass. Ensure that you are tracking draft history and how the language evolved with a version/iteration key.

7. Human in the loop

This is arguably the most important pattern in this list as it relates to government services.

AI agents do the groundwork and a human authorizes anything irreversible or high-consequence.

Where this could fit:

• Benefits recommendation: it could surface other programs for individuals with a human caseworker approving before sending

• Security clearance recommendations: AI could flag patterns and a human adjudicator decides

• Regulatory recommendations: an officer approves before action

For a large share of government decisions this is not optional. Where a statute or delegation of authority assigns a determination to a named official, an AI agent can prepare the decision, but it cannot make it.

Two really important items to capture when implementing this are:

1. Who reviewed/approved the AI output

2. Whether or not they changed the AI recommendation

Also, being able to think about and combat Automation Bias in AI is critical.

8. Composite systems

I’d be remiss not to mention that no government system or process is singular. Most combine the patterns above. Take a few moments to think about a government process that you interacted with recently. This could be as mundane as renewing your passport or applying for a grant.

These flows appear simple on the surface, but have decades of processes built up behind the scenes. Both of these examples end the same way: with a human making an informed decision.

Final thoughts

For work in government and civic tech, we should be thinking about where can we implement automation to free us humans up to do more impactful work. How can we add AI agents to reason and act where mistakes are cheap and reversible?

FOIA request classification, permit intake parsing, and 311 routing could be good first targets. They’re high volume, the failure mode is a rerouted request rather than a denied benefit, and every one of them has a human already positioned to catch errors.

Adjudication is the last thing you automate — not the first — and even then the AI agent prepares the decision for a human who signs it.

Have you been exploring multi-agent architectures? Do you see opportunities or concerns?

But before we get to good, let’s talk about the current state.

The current state

There’s a lot of conversation about “AI readiness,” including what leadership, governance, and infrastructure should be in place for public-sector agencies to deploy AI tools in benefit programs like SNAP. At this stage, both federal and state legislative guidance is thin (only three states have AI laws that explicitly impact AI use in public benefit programs1), and will likely stay that way. Instead, we’ve seen many states turn to governors’ executive orders, AI task force reports, and administrative policies to add adequate controls and guardrails around how AI can be deployed, seeking to prevent harm.

Once the policies and infrastructure are in place, public benefit agencies often decide they’re ready for their first AI pilots, often starting with internal-facing tools that aim to reduce caseworker burden. Many states have begun piloting numerous AI tools including policy chatbots, case note generators, document quality verifiers, and the like — tools Nava has had a hand in delivering.

And if the state agency has the right infrastructure and capacity, or if they’re working with a trusted vendor, hopefully they’re also doing some version of the third step: AI evaluation.

With all three steps in place, the process would look something like this:

This is a good start, and agencies who’ve developed some form of this process should applaud their progress — then push a little further.

And while most states establish governance before adoption, it doesn’t always happen in a linear fashion. States who fail to govern, or who establish strict guardrails that don’t allow for safe experimentation, may see shadow AI adoption by frustrated employees looking to reduce workload burden behind the scenes.

In states who’ve established some form of generic AI governance as the first step, we’re also starting to see a pattern where AI adoption is increasing faster than the two things that should keep it accountable: evaluation and governance. AI technology is evolving faster than the other two can keep up.

Current gaps, and how states might address them

A few common problems we’ve identified across many states with this current process:

1. Evaluating technical performance but not impact

Many states are missing a critical step in the above process: AI impact evaluation. Any AI tool deployed for any reason is typically evaluated by the deployer for a certain period of time to ensure quality, safety, reliability, and other performance metrics.

But human service agencies don’t care solely about the performance of the tool — they care whether it’s making a difference in the program performance metrics they’re expected to meet: Did it reduce the SNAP Payment Error Rate? Did it decrease churn among participants subject to work requirements? Did it reduce denials for failure to return verifications? These are the questions state agency leaders are responsible for answering to their legislatures when it comes time to fund their programs.

Despite this need, we’ve found that states are often lacking in AI impact evaluations, which are the types of evaluations that actually answer those questions. This may be due to lack of data infrastructure, lack of appropriate staff and resources, or lack of understanding around how to conduct impact evaluations. Whatever the reason, this is an area we’ve seen for further growth in many states.

We believe evaluations — including impact evaluations — are a key component of AI adoption in the public sector that should be prioritized. These evaluations help ensure the tool not only works but it actually meets the outcomes set by the agency’s original goals. We believe at a minimum, all AI tools deployed in public assistance programs should be able to answer these three questions:

1. Does it reduce caseworker burden?

2. Does it improve client access?

3. Does it cause any unintended harm?

For states who are struggling to get started with impact evaluations, Nava has put together some great resources you can start with.

2. Not sharing learnings across deployments

Another common problem we’ve seen is that the different AI tools are being piloted by different vendors, across different programs, and by different agencies. What one agency learns in their user research and evaluations rarely makes it to the next, causing duplicated work and inconsistent deployments that don’t build on each other.

We believe states that want to build AI tools effectively, quickly, and consistently in their benefit programs should have a system in place to share learnings across different deployments. This ensures that the same mistakes don’t get repeated from program to program.

There are now approximately 14 states that are publishing or are required to publish inventories for either “high risk” or all AI use cases within government agencies.2 We think an AI inventory — at a minimum for “high risk” use cases — is a good start, but we think this can go even further.

First, states need to ensure that the AI use case inventory process isn’t so burdensome and restrictive that their agencies are afraid to populate it. This is a pattern we’ve seen repeated a few times across states: inventories that require detailed submission, review, and approval. They ultimately scare agencies away from disclosing certain use cases they think will take too long to approve, or won’t be approved at all. We would caution that a burdensome inventory process can be equivalent to no inventory at all (and uses significantly more resources to set up).

Second, we believe that good AI inventories don’t just include a list of the use cases, but also share their evaluation results and any lessons learned. If other agencies are using the inventory as a way to understand what exists and how they can deploy it, these results will be critical for helping them make informed decisions and ask potential vendors the right questions.

A good AI inventory might also include space for the following information:

• Agency owner of this deployment, or who to contact for more information

• User research and evaluation findings or how to access them

• Best practices from the deployment. For example:

  • “We found a good way to ask users about consent to using AI. Here’s the pattern and language we used…”

  • “We were able to safely engage users of this particular demographic by doing…”

  • “We developed a way of ensuring meaningful human review of the final output. Here’s what we did…”

• Negative outcomes or harms discovered that should be shared with others developing a similar tool

3. Missing mechanisms for policy revision

We’ve found that in states that have written AI policies, almost none have a process in place to review and revise that policy. Once published, there’s often no clear owner reviewing the AI policy to make sure it did what it was intended to do. Did the guardrails protect who they were meant to? Are there any additional ones needed? AI technology is changing rapidly. What once made sense at the time of publishing may no longer be sufficient to protect against the capabilities of newer models or agentic AI.

We believe that AI governance should be revisited regularly, especially early in a state’s experimentation with AI. While legislation is a lot harder to change, there are more flexible policy vehicles like statewide executive orders and government-wide policies. This is the level at which states can maneuver more quickly and effectively. Good AI governance will only get better over time, and this can be enhanced and expedited if it operates as a loop — where governance feeds adoption, which feeds evaluation, which feeds back to governance and adoption.

We believe forming a strong loop from the start is what will separate states that simply adopt AI from states that get better at it.

Next steps

If your state is just starting out on your journey to AI adoption, or if you have a process and you’re looking to refine it, consider how you’ll build a loop. This could look like many things, including:

1. Build up your agency’s data infrastructure to allow for better impact evaluations

2. Write evaluation criteria into your RFPs

3. Create a post-deployment review of every AI use case that helps identify best practices and negative outcomes that might warrant adjustments to policy

4. Develop an AI inventory that also includes evaluation and learning data that’s shareable at least across your agencies, if not publicly

5. Codify and share any best practices that have proven effective in your program

6. Set a regular review cadence of your AI policies

7. Designate one clear owner of an AI policy whose job is to revise based on learnings

8. Consider joining feedback channels and working groups with other states that are interested in cross-state sharing of learnings

If you’re also thinking about this topic, we’d love to hear from you about your experience. Is there anything we’re missing? Let us know in the comments, or email us at labs@navapbc.com.

Government teams want to experiment with AI. But two factors can make that challenging:

• Cost: Cloud AI inference APIs cost money, with pricing often very opaque

• Security: Extremely sensitive data can’t leave the building

Local language models can help solve both problems. No per-token bills. Nothing leaves your machine.

Local language models follow a pattern similar to the open source movement: free to use, free to tinker with, free to run on your own terms.

Local language models’ strategic advantages for public-sector AI

Beyond just data security, having sovereignty on your tech stack — including language models — allows you to not be beholden to a single frontier lab or cloud provider, with their accompanying price increases, deprecations, etc.

Especially if you’re not an expert in FedRAMP and other regulatory compliance, hosting language models locally simplifies compliance. Your data stays in your control.

Research shows that fine-tuned smaller language models outperform larger models on a series of tasks — even within government domains. This affords you flexibility to load models into memory quicker, generate responses faster, and continue to close the gap with frontier models.

I’d be remiss not to mention that there is a maintenance and update tradeoff. Fine-tuning, model versioning, and other MLOps is operational overhead if you are looking to expand beyond yourself.

What’s out there

Some tools that make this easy to experiment with local language models include Ollama, n8n, and LM Studio.

To keep it simple, we’ll use the following:

• Ollama is a way to serve language models locally, right from your own machine. Think of it as a local API server for open models.

• n8n is a no-code workflow automation layer on top. You wire together nodes instead of writing scripts.

Put these two together, and you get 100% private, self-hosted AI agents and automation workflows. No paid cloud APIs. Everything processed locally.

Note: n8n has a self-hosted-starter-kit repository, but if you are on a Mac you cannot expose your GPU to Docker (similar to vLLM). It just takes a little more setup.

What I built

Back in June of 2025, I started prototyping with local language models on a new machine: a MacBook M4 Pro with 24 GB of RAM.

Both Ollama and n8n run in Docker, so I’m not too worried about a model going rogue and taking my machine offline (e.g. 370+ GB of data; deleting production databases; and more).

Some of the first automations I built are:

1. An email drafter

2. A Slack channel summarizer

3. A local document search, using RAG

4. An offline plain-language checker

See the GitHub repo here.

Anecdotally, for small, narrow tasks — like summarization, extraction, and classification — a well-prompted local model can perform quite well.

Where it is now and what’s next

This is extremely nascent. These are quick experiments, not production tools.

A few open questions I’m still working through:

• How far local models can really go on more complex reasoning tasks with vector stores, multistep reasoning and acting, etc.

• How well a small local model performs when fine-tuned on domain-specific documents versus a frontier model

• Whether the setup holds up outside a single laptop (e.g. running on a server on-premise)

If you’re experimenting with local AI for public-sector work, I’d love to hear how it’s going. Comment below or reach us at labs@navapbc.com.

Resources:

• Self-hosted AI starter kit

• n8n.io

• ollama.com

• Sample n8n Workflows

When someone needs support for food, health insurance, veterans benefits, and other public programs, the experience is driven by documents. Someone uploads a proof of income, or medical expenses, or proof of enrollment in an educational program. Staff work with those documents to make sure they’re accurate, valid, and meet requirements.

Much of the work to improve the experience of these programs is about making that flow better: getting files uploaded correctly, making sure they’re readable, and letting people know as early as possible if they need to resubmit something, so they don’t wait weeks to hear back.

As part of my role on the Nava Labs program, I work on our Caseworker Empowerment Toolkit, which supports staff who connect people to benefits. In this toolkit, one of the questions we’re exploring is: how might document AI tools help staff classify documents, extract information, and figure out whether what someone’s shared will actually get them the benefits they’re looking for?

And as we develop this tool, I keep hitting a challenge of finding realistic documents to test with that don’t include people’s sensitive data. This is a challenge common to any project trying to improve benefit program experiences.

Why testing document workflows is hard

The main obstacle is creating or gathering documents that account for different scenarios applicants may be in. Many benefits require proof of income, which could come from salaried employment, self-employment, or the gig economy, to name a few.

To address this, teams either hand-create or find sample documents and hope they cover enough of the situations people are in. Another path is using real documents, but this requires securely handling people’s most sensitive data, so it’s not a good fit for early development and testing.

There are so many permutations of scenarios, document types, and input methods, and there’s a need to test as many as possible before anything is put in front of the public or staff. Synthetic documents add another testing method to the toolkit, one that doesn’t depend on real sensitive information.

Why now

Previously, ChatGPT and other image-generation models were unreliable. Many of us have seen examples of images where a person has six fingers, or text with typos and garbled characters. The quality was so low that the output was essentially useless for testing.

Gpt-image-2 was released this spring and now an entire document can now be generated. Not perfectly, but fairly consistently: full-page layouts, different document structures, text that refers to the same person throughout, and dates and amounts that line up well enough to support testing.

A quick comparison of models

Here’s the same prompt for a synthetic pay stub, run through the old model and the new one.

Test 1: Paystubs

gpt-image-1: Synthetic pay stub generated by gpt-image-1, showing garbled words like “Duringar” and “Gotalc” and numbers that don’t add up.

gpt-image-2: Synthetic pay stub generated by gpt-image-2, with a clean layout, itemized earnings and deductions, and year-to-date totals that reconcile (at least at first glance).

Test 2: SNAP award letters

The same jump shows up in a SNAP award letter. The gpt-image-1 version misspells “Effective” and “Caseworker” and signs off with “Sercrndy.” The gpt-image-2 version produces a full approval notice with a benefit summary, EBT card instructions, and fair hearing rights, all consistent with one made-up household.

The synthetic SNAP award letter from gpt-image-2 makes a much more internally consistent approval notice with a benefit summary and fair hearing rights.

Note: If anything, Image2 makes this document look too polished, since many real SNAP award letters are simple B&W text in a letter with an organization logo in the header

What this makes possible

This lets us start to more systematically test document experiences, rather than relying purely on ad-hoc examples from the internet or handmade documents from a few people on an internal team. If someone has income from self-employment or the gig economy, we can generate documentation from the different apps and services they use and see how our systems would handle it at a first pass. And if we hear about a scenario where experiences are poor, we can generate documents for that scenario specifically. So we get speed, we get depth of experimentation, and we can do all of that without using people’s real information.

This process also has potential applications in detecting, or training staff to detect, fake documents created with AI.

The flipside

The images are better, but they can still contain errors, and as the quality goes up, the errors become harder to notice. Fewer sixth fingers, more income amounts off by a few dollars. No synthetic document is a replacement for human-generated documents and human testing. It’s all a supplement and a way to expand the tools we have available to make getting and receiving benefits less onerous for the public and staff.

Further work

• Deeper evaluation: The images are great for coarse document classification and generally seem to follow instructions for household composition, dates, income amounts, and other details. But further human validation is needed before scaling up synthetic document creation as a testing method.

• Improving tools to support batch creation + sharing of documents

• Tailoring prompts: Continuing to improve layouts that don’t match real documents, increasing formats (phone photos vs. scans vs. handwritten notes) and quality (blurriness, stains, crumpling, etc.)

When they meet with clients, caseworkers have to make a choice over and over: pay attention to the person in front of them, or look away to type notes into a computer. Do the first, and they’re left trying to remember the conversation at the end of a long day. What the client said about their income, their rent, who lives in the household, and which documents they need to find. The alternative is spending most of the interaction with their focus on a screen instead of a person.

One client interaction can turn into a note in a case management system, required forms and other outputs. Often times different providers and funders want the same information reported in slightly different ways. Additionally, if the caseworker spoke with a client in another language, they’ll have to write an English record in their case management system, then write back to the client in their preferred language. All of this documentation burden adds stress to staff who support people who are often already in a crisis, leading to high rates of burnout and turnover.

We’ve been prototyping a tool for those interactions. With consent, the tool transcribes the conversation, pulls out the case facts, and drafts a case note for the worker to review and correct before anything gets saved. The tool does the first pass and the caseworker can correct and edit any outputs. This prototype is a way for us to explore the implications of voice AI on case note accuracy, administrative burden, privacy and consent.

What’s out there

“Ambient AI” has been maturing, with model providers like Assembly AI offering a Realtime Speech-to-Text API that can transcribe interactions and even assign speaker labels on the fly. There are many applications of models like these, especially in the healthcare space, where providers like Abridge offering documentation support for clinical providers.

Case workers we’ve talked to have been curious about the idea, with reservations about client privacy and the accuracy of the outputs. But so far, the only real application we’ve heard from case workers is them adding notes (with sensitive info manually removed) to ChatGPT to format them into the appropriate template.

Prototype walkthrough

To explore this concept, the prototype walks through five steps: set up the session, capture the conversation, review the facts, review the drafts, and finalize.

It opens by asking who’s in the meeting. The worker adds themselves and the client by name before any recording starts, and consent is confirmed here too.

Session setup. Add participants and confirm consent before the conversation begins.

During the conversation, the tool transcribes in real time. In the demo scenario, a SNAP renewal where the client speaks Spanish, the bilingual transcript shows each turn in both languages, color-coded by speaker. When the conversation ends, the transcript is ready to review before moving to fact extraction.

Live record. The bilingual transcript shows both languages side by side, speaker-labeled and timestamped.

After the conversation, the worker reviews extracted facts grouped into “Needs review” and “Accepted.” High-confidence details (address, household members, income, expenses) are pre-accepted but editable. Lower-confidence items surface at the top for explicit review.

Fact review. Lower-confidence items surface first. Every fact has Accept, Reject, and Source controls.

Every fact links back to the exact moment in the conversation it came from. The evidence dialog shows the bilingual transcript scroll alongside a usage panel: the source, confidence band, timestamp, transcript quote, and which note sections the fact was pulled into.

Evidence. The transcript quote, confidence, timestamp, and the note sections it feeds into, all in one place.

Only after the facts are reviewed does the tool draft the case note. It opens a rich text editor alongside the full transcript, so a worker can read both at once, edit freely, and then send a client-facing summary separately.

Case note. Generated from the reviewed facts, editable in place, with the full transcript alongside for reference.

Where it is now

While the value of this tool seems clear, there are many challenges to overcome on the road to responsible implementation. Accuracy is key, especially considering the nuanced situations people actually show up with:

• several family members talking in the same room,

• someone switching languages mid-conversation,

• Accents,

• Correctly recording a specific names or $ amounts, or

• a client who says one thing early on and corrects it later.

All of this has to land in the record correctly, and getting it wrong is the risk the tool is supposed to reduce. Live speaker labeling, for instance, can drift in the first few seconds of a conversation while the model figures out who’s talking.

Evaluating a voice tool is harder than evaluating a chatbot. When most of the interaction is audio rather than text, there’s no clean transcript to check the output against by default, so it’s tougher to measure whether the tool heard things right.

Look forward to our team sharing more about this focus area, including things like:

• How we approach evaluating a tool like this

• Feedback and ideas from caseworkers

• How to implement tools like this responsibility and securely

• And more!

If you do benefits casework, or build tools for people who do, we’d like to hear how this lands. Reach us at labs@navapbc.com. See more about the caseworker tools we’re prototyping, piloting and scaling at caseworker.navapbc.com.

Nava Labs is a philanthropically-funded R&D team within Nava, focused on prototyping innovations to the social safety net that government agencies would benefit from but can’t fund directly. We build and test new approaches to delivering public services, evaluate what works, and advocate for scaling proven solutions.

While we already regularly share news, case studies, insights, and toolkits on our main website, this Substack will offer something different: a look behind the scenes. Content will include:

• Posts from our new Technologist-in-Residence team on their experimental work as they prototype and test emerging approaches to public service delivery

• Updates on our program evaluation work, including data deep dives, while analysis is still in progress

• Our latest thought processes and frameworks, even as we’re still mulling them over

• Other field observations from our work

You can subscribe and get all of our posts straight to your email inbox (or notifications on the Substack mobile app), or you can simply check out posts like a normal blog.

We’re excited to engage fellow practitioners in this space, building toward the goal of improving service delivery for the public. We’d love to exchange ideas with you in the comments on our posts.

Let’s learn together!