Intro
In our first post, we shared how models performed on a set of 500 SNAP (CalFresh) scenario questions. Here, I’ll look at the questions they got wrong most often and the experiments I tried to improve their answers.
In the original runs, GPT-6 Astra led the models we tested with an average accuracy of 92.2%. That still meant about 39 of 500 answers were wrong according to a key created by SNAP experts. Let’s dive in to the errors and how we might improve accuracy.
Our testing dataset focused on California questions, so you may see SNAP referred to as CalFresh (California’s name for the program.)
Error Categories
To begin, I wanted to see if any topics or types of questions caused the most errors. The dataset creators, Better Government Lab, tagged each question in the dataset with a SNAP quality control topic. This let us see where errors clustered in LLM responses. The categories with high errors were fairly consistent across models, though each model had its own highs and lows. I saw that four categories (self-employment, transitional benefits, social security numbers and citizenship and noncitizenship status) had the most errors.
Note: Some categories in the dataset only have a few questions, so one answer can cause a large swing in accuracy.
Why are LLMs getting questions wrong?
As LLMs improve, mistakes can be harder to untangle. In the frontier models we tested, we did not see clear examples of hallucinated policies or arithmetic errors. Instead, we saw subtle misreadings of wording and incorrect assumptions about households or income.
As an example, “for benefits,” “for the purpose of assistance,” and “application for assistance” can all lead to different correct responses to a question.
Other errors reflected outdated policy information (the models had no web access in our first tests) or reliance on general rules when more specific ones applied.
Error cause: not aware of policy updates
These were policies that had changed after the models had been trained. For example, non-citizen eligibility policies changed in the last year. Since we didn’t allow web-search in our initial testing, models had no way to see new information about the rules.
Hypothetical household: Tomas applies for CalFresh with his child. Tomas is an asylee.
How this could lead to AI mistakes: Previously, Tomas would’ve likely been eligible for SNAP. Under the latest policies, asylees are not eligible.
Error cause: relying on general policies when more specific rules applied.
The model didn’t surface or prioritize California policies that override general policies.
For example, SSN requirements for newborns or verification requirements for students.
Hypothetical situation: Dominic applies for CalFresh as a college student in an employment focused program.
How this could lead to AI mistakes: Most aspects of a SNAP application require verification. But California has a specific policy that “[Workers] must only require verification that a student meets an exemption when information provided by the household is questionable…” (Source). A model may say Dominic must provide documents when the policy doesn’t always require that.
Error cause: not applying the correct role for a household member
Some questions relied on understanding the difference between being reported on an application, having income or resources counted, belonging to a CalFresh household, and receiving benefits. Assuming a different scenario led the AI to answer questions incorrectly.
Hypothetical household: Elena would like to apply for CalFresh for her household. She lives with her child and her partner, Mateo. Mateo isn’t eligible for CalFresh.
How this could lead to mistakes:
“Is Mateo eligible to receive benefits? (No)”
“Should Mateo be included on an application? (Yes because workers need to include his income in the application)”
If the model misinterprets the intent of the question, it could answer incorrectly even with all the correct policy info.
Error cause: Nuances in income + benefit calculations
Determining how to count income and expenses for SNAP is a complex process. LLMs made errors in this category when they incorrectly assumed:
Whether an amount already had the earned income deduction applied to it
Whether an amount in a question was “gross” or “net” income
How to count one time (excluded) vs. recurring payments (included)
If an income amount was “reasonably anticipated” when calculating future income amounts
How to apply rules about self-employment expense deductions. Like whether a meal at a business conference is a deductible expense.
Experiments to improve results
Experiment #1: Add web search and increase reasoning level
This is the most straightforward change I could make. The experiment assumed that if models had more reasoning budget and access to newer resources, then they would improve accuracy, especially on questions with recent policy changes.
I tested two OpenAI models: GPT-6 Sol and GPT-6 Astra.
Changing reasoning effort and adding web search without specialized guidance produced modest overall results. Citizenship and noncitizenship questions showed a clearer improvement.
As expected, this category benefited most from access to current sources because its policies had recently changed.
On 34 citizenship questions, web search improved answer-key agreement for both Astra and Sol in these matched runs.
Experiment #2: Improve the system prompt
I reviewed questions GPT-6 Astra consistently got wrong in the dataset, wrote notes on the mistakes, and turned those notes into general guidance for later tests.
GPT-6 Astra with the revised prompt averaged 491.7/500 (98.3%) across three runs, up from 461/500 (92.2%) at baseline."
Baseline prompt
You are answering a SNAP scenario benchmark question.
Return only JSON: {"answer":"one multiple-choice letter","reasoning":"brief explanation","review_required":true|false}.
Use only facts in the scenario. Do not represent your response as a final benefits decision or legal advice.Types of instructions I added
Many of the instructions I added clarified terms or expected processes that had led to errors. A few examples are:
For questions determined to be about immigration or SSN requirements, the instructions told the model to use web search to find up to date policies since these topics have had recent changes.
For topics with California specific policies, I instructed the model to check current CDSS guidance
For income and deductions:
Gross income is before business expenses or household deductions; net income is after applicable deductions.
A benefit-increasing income decrease or expense claim generally needs verification before raising benefits; one lower payment or higher expense does not by itself establish a lasting reduction.
Treat an amount described as earnings as gross unless called net or after deductions.
Over-the-counter medicines and vitamins require prescription or approval by a qualified health professional
Use submitted official records to resolve a conflicting verbal estimate when those records already establish the amount and frequency.
For household composition:
Inclusion in the household assessment is distinct from an individual’s eligibility to receive benefits; an ineligible noncitizen’s income and resources can still matter.
Do not infer that co-residence makes unrelated people a food household.
And many more pieces of guidance and clarification for each category of error
Limitations
The changes I made to the prompt were designed specifically for this set of 500 questions.
This type of focus could lead to “overfitting” where the model will perform well on this set of questions, but do worse in the real world.
Further testing on new scenarios and different phrasings could show whether this guidance holds up.
While I am familiar with CalFresh and SNAP policies, I am not a policy expert, so some of the guidance I gave the tool would need to be verified before it would be used broadly.
What might this mean when developing AI tools?
An overall score can hide errors in the questions that matter most to people. Having each question assigned to a quality control category allowed us to have a better picture of how models perform in areas that matter to beneficiaries and staff. Like checking eligibility, identifying who belongs in a household, verifying an expense, or calculating income.
Based on this experience, here are a few suggestions for teams building AI tools for public benefits:
Review the trickiest errors with program experts. Some questions were straightforward to resolve, such as pointing the model to current citizenship rules. Others required understanding when staff request verification and when they can accept a client’s statement.
Clarify ambiguous phrasing in people’s inputs and the tool’s output. If a detail changes the answer, such as whether income is gross or net, the tool should ask before proceeding.
Make sure the model’s approach fits its audience. A public-facing tool should explain that people can apply for SNAP even if they are unsure whether they qualify. A caseworker-facing tool needs more precise guidance on process and verification to support staff decisions.
Be ready for a “whack-a-mole” improvement process. Guidance that helps with one topic can cause mistakes elsewhere. When I added broad guidance to verify evidence before using it, the model did worse on questions where verification was needed only if a claim was questionable.