SNAP helps millions of people get food. The program is hard to navigate: rules about who is eligible, who belongs on an application, and which deductions a household can get add up to a lot of ways to get something wrong.
Applied well in this space, AI tools can increase application accuracy, cut administrative burden, and get people food faster. Done poorly, they can discourage people from applying, create more work for staff who have to fix AI’s mistakes, and drive up payment errors, which are now tied to how much funding states get for SNAP.
All of these use cases hinge on how well AI models perform on fundamental SNAP questions. This post walks through how we evaluated frontier and open source Large Language Models (LLMs) on 500 questions about California’s SNAP program, known as CalFresh. Many of the questions would be applicable to nationwide SNAP questions. Future posts will detail how prompt, harness, and context engineering can continue to increase precision.
The dataset is available on request and we used open-source tools, so anyone can repeat this approach. The question scenarios were developed by Better Government Lab in partnership with SNAP policy experts. See more about the dataset in the appendix.
Sample question (illustrative)
Preamble
Age: Young adult (below 24)
Gender: Male
Employment status: Employed by other
Healthcare/Disability status: Not legally disabled
Citizenship status: U.S. citizenQuestion
The client is a 20-year-old enrolled full-time, year-round at a California community college. He lives with two roommates and buys and prepares his own food. He works an average of 22 hours per week at a grocery store. Does he meet the CalFresh student eligibility rules? [Select one correct answer.]A. No, full-time students are not eligible for CalFresh
B. No, he must live with his parents and apply with them because he is under 22
C. Yes, because he works an average of at least 20 hours per week
D. Yes, but only if he is approved for work-studyAnswer (reviewed by SNAP policy experts)
Answer: C
How the models performed
Last updated: Sept. 22, 2026
Accuracy
GPT-6 Astra scored highest, averaging 92.24% across three test runs.
That’s impressive! But this still means the LLM got around 39 out of 500 questions wrong. Depending on the question, one of those incorrect responses could have large impacts.
Costs
Costs per 500-question run varied considerably (Fable’s $5 compared to Gemini’s $0.49).
While costs were low for our runs of 500 questions, this costs would grow linearly as usage grows, unless questions are batched and cached. Costs would also be more expensive for back-and-forth conversations, as opposed to our test which constrained models’ responses to a single multiple-choice response.
Speed
Time to respond varied as well: Sonnet 5 was fastest, with a 1.8-second median response, compared to over 20 seconds for some local models.
Open-weight models took longer to respond, but speeds vary depending on the hardware running the model. The open models in this test were run on a Mac Studio M3 with ~819 GB/s unified memory bandwidth. Most of these local models could be run on a modest MacBook Pro or other consumer hardware.
Other observations
Open-weight models performed well, with Gemma 4 31B scoring highest at 85.8%. But they didn’t match the accuracy or speed of the frontier models.
Newer, larger models generally scored higher than their predecessors (GPT-6 outperformed GPT-5.6, for example), with one exception: Claude Opus did slightly better than Fable. A surprise was Gemini 3.8 Flash, which had a great balance of accuracy and low cost, but was slower to respond than other models.
AI made the most mistakes in 4 categories
The dataset creators, Better Government Lab, grouped their questions using the same error categories as SNAP quality-control reviewers. When we looked at how models did across those categories, we saw most struggled in the same four categories. Questions about citizenship status, self-employment, transitional benefits, and Social Security numbers had the most errors. In a future post, we’ll dive into these categories and questions to see what may be causing the higher error rates.
What’s next
Our next post will explore the high-error questions and categories to explore what types of questions cause LLMs issues, and what might be done to reduce those errors.
Multiple-choice questions are only a starting point. Building on our work with Form-Filling Assistants, we plan to measure the speed, accuracy, and cost of AI agents while completing government forms.
Appendix 1: The dataset
We used Better Government Lab’s SNAP Scenarios dataset to measure AI models. Better Government Lab is a Nava Labs partner and worked with us evaluating an assistive chatbot for caseworkers. They describe the dataset as:
…a detailed 500-question benchmarking dataset for measuring LLM or human-in-the-loop accuracy on realistic scenarios related to the U.S. Supplemental Nutrition Assistance Program (SNAP) in California...Questions were generated based on frequencies of types of errors from publicly available national quality control data, which pertains to the accuracy of SNAP benefit amounts.
Source: https://huggingface.co/datasets/cornellbgl/SNAP_scenarios
The questions are equally crucial to caseworkers and program staff who are under pressure to reduce their payment error rates in order to avoid cuts in already strained funding. Better Government Lab also worked with SNAP policy experts in California and Georgia to review and validate the set of questions to ensure they’re both accurate and relevant.
This round of testing was focused on California’s SNAP program. Many questions are consistent nationwide, but some rules vary by state. Results here shouldn’t be read as a measure of how models handle SNAP nationally.
Note: Any example questions and responses in this post are realistic, but not pulled directly from the test set, to avoid having the answers on the open internet.
Appendix 2: Full results
Appendix 3: Our testing setup
Tools: We used Inspect, which is described as “an open-source framework for large language model evaluations…developed by the UK AI Security Institute and Meridian Labs.”
This let us set up test conditions and run LLMs against the scenario dataset reliably and quickly.
Model setup: For all of the models we tested, we set reasoning to “low” or equivalent settings, and turned off web search and other tools.
Our aim was to provide a baseline measure of performance with settings similar to what may be used in chatbots or other tools for staff and the public.
We ran each frontier model 3 times to account for variability in the LLM’s responses.
Since local models had to run sequentially on someone’s machine, we only ran one test round for those.