When someone needs support for food, health insurance, veterans benefits, and other public programs, the experience is driven by documents. Someone uploads a proof of income, or medical expenses, or proof of enrollment in an educational program. Staff work with those documents to make sure they’re accurate, valid, and meet requirements.

Much of the work to improve the experience of these programs is about making that flow better: getting files uploaded correctly, making sure they’re readable, and letting people know as early as possible if they need to resubmit something, so they don’t wait weeks to hear back.

As part of my role on the Nava Labs program, I work on our Caseworker Empowerment Toolkit, which supports staff who connect people to benefits. In this toolkit, one of the questions we’re exploring is: how might document AI tools help staff classify documents, extract information, and figure out whether what someone’s shared will actually get them the benefits they’re looking for?

And as we develop this tool, I keep hitting a challenge of finding realistic documents to test with that don’t include people’s sensitive data. This is a challenge common to any project trying to improve benefit program experiences.

Why testing document workflows is hard

The main obstacle is creating or gathering documents that account for different scenarios applicants may be in. Many benefits require proof of income, which could come from salaried employment, self-employment, or the gig economy, to name a few.

To address this, teams either hand-create or find sample documents and hope they cover enough of the situations people are in. Another path is using real documents, but this requires securely handling people’s most sensitive data, so it’s not a good fit for early development and testing.

There are so many permutations of scenarios, document types, and input methods, and there’s a need to test as many as possible before anything is put in front of the public or staff. Synthetic documents add another testing method to the toolkit, one that doesn’t depend on real sensitive information.

Why now

Previously, ChatGPT and other image-generation models were unreliable. Many of us have seen examples of images where a person has six fingers, or text with typos and garbled characters. The quality was so low that the output was essentially useless for testing.

Gpt-image-2 was released this spring and now an entire document can now be generated. Not perfectly, but fairly consistently: full-page layouts, different document structures, text that refers to the same person throughout, and dates and amounts that line up well enough to support testing.

A quick comparison of models

Here’s the same prompt for a synthetic pay stub, run through the old model and the new one.

Test 1: Paystubs

gpt-image-1: Synthetic pay stub generated by gpt-image-1, showing garbled words like “Duringar” and “Gotalc” and numbers that don’t add up.

gpt-image-2: Synthetic pay stub generated by gpt-image-2, with a clean layout, itemized earnings and deductions, and year-to-date totals that reconcile (at least at first glance).

Test 2: SNAP award letters

The same jump shows up in a SNAP award letter. The gpt-image-1 version misspells “Effective” and “Caseworker” and signs off with “Sercrndy.” The gpt-image-2 version produces a full approval notice with a benefit summary, EBT card instructions, and fair hearing rights, all consistent with one made-up household.

The synthetic SNAP award letter from gpt-image-2 makes a much more internally consistent approval notice with a benefit summary and fair hearing rights.

Note: If anything, Image2 makes this document look too polished, since many real SNAP award letters are simple B&W text in a letter with an organization logo in the header

What this makes possible

This lets us start to more systematically test document experiences, rather than relying purely on ad-hoc examples from the internet or handmade documents from a few people on an internal team. If someone has income from self-employment or the gig economy, we can generate documentation from the different apps and services they use and see how our systems would handle it at a first pass. And if we hear about a scenario where experiences are poor, we can generate documents for that scenario specifically. So we get speed, we get depth of experimentation, and we can do all of that without using people’s real information.

This process also has potential applications in detecting, or training staff to detect, fake documents created with AI.

The flipside

The images are better, but they can still contain errors, and as the quality goes up, the errors become harder to notice. Fewer sixth fingers, more income amounts off by a few dollars. No synthetic document is a replacement for human-generated documents and human testing. It’s all a supplement and a way to expand the tools we have available to make getting and receiving benefits less onerous for the public and staff.

Further work

• Deeper evaluation: The images are great for coarse document classification and generally seem to follow instructions for household composition, dates, income amounts, and other details. But further human validation is needed before scaling up synthetic document creation as a testing method.

• Improving tools to support batch creation + sharing of documents

• Tailoring prompts: Continuing to improve layouts that don’t match real documents, increasing formats (phone photos vs. scans vs. handwritten notes) and quality (blurriness, stains, crumpling, etc.)

When they meet with clients, caseworkers have to make a choice over and over: pay attention to the person in front of them, or look away to type notes into a computer. Do the first, and they’re left trying to remember the conversation at the end of a long day. What the client said about their income, their rent, who lives in the household, and which documents they need to find. The alternative is spending most of the interaction with their focus on a screen instead of a person.

One client interaction can turn into a note in a case management system, required forms and other outputs. Often times different providers and funders want the same information reported in slightly different ways. Additionally, if the caseworker spoke with a client in another language, they’ll have to write an English record in their case management system, then write back to the client in their preferred language. All of this documentation burden adds stress to staff who support people who are often already in a crisis, leading to high rates of burnout and turnover.

We’ve been prototyping a tool for those interactions. With consent, the tool transcribes the conversation, pulls out the case facts, and drafts a case note for the worker to review and correct before anything gets saved. The tool does the first pass and the caseworker can correct and edit any outputs. This prototype is a way for us to explore the implications of voice AI on case note accuracy, administrative burden, privacy and consent.

What’s out there

“Ambient AI” has been maturing, with model providers like Assembly AI offering a Realtime Speech-to-Text API that can transcribe interactions and even assign speaker labels on the fly. There are many applications of models like these, especially in the healthcare space, where providers like Abridge offering documentation support for clinical providers.

Case workers we’ve talked to have been curious about the idea, with reservations about client privacy and the accuracy of the outputs. But so far, the only real application we’ve heard from case workers is them adding notes (with sensitive info manually removed) to ChatGPT to format them into the appropriate template.

Prototype walkthrough

To explore this concept, the prototype walks through five steps: set up the session, capture the conversation, review the facts, review the drafts, and finalize.

It opens by asking who’s in the meeting. The worker adds themselves and the client by name before any recording starts, and consent is confirmed here too.

Session setup. Add participants and confirm consent before the conversation begins.

During the conversation, the tool transcribes in real time. In the demo scenario, a SNAP renewal where the client speaks Spanish, the bilingual transcript shows each turn in both languages, color-coded by speaker. When the conversation ends, the transcript is ready to review before moving to fact extraction.

Live record. The bilingual transcript shows both languages side by side, speaker-labeled and timestamped.

After the conversation, the worker reviews extracted facts grouped into “Needs review” and “Accepted.” High-confidence details (address, household members, income, expenses) are pre-accepted but editable. Lower-confidence items surface at the top for explicit review.

Fact review. Lower-confidence items surface first. Every fact has Accept, Reject, and Source controls.

Every fact links back to the exact moment in the conversation it came from. The evidence dialog shows the bilingual transcript scroll alongside a usage panel: the source, confidence band, timestamp, transcript quote, and which note sections the fact was pulled into.

Evidence. The transcript quote, confidence, timestamp, and the note sections it feeds into, all in one place.

Only after the facts are reviewed does the tool draft the case note. It opens a rich text editor alongside the full transcript, so a worker can read both at once, edit freely, and then send a client-facing summary separately.

Case note. Generated from the reviewed facts, editable in place, with the full transcript alongside for reference.

Where it is now

While the value of this tool seems clear, there are many challenges to overcome on the road to responsible implementation. Accuracy is key, especially considering the nuanced situations people actually show up with:

• several family members talking in the same room,

• someone switching languages mid-conversation,

• Accents,

• Correctly recording a specific names or $ amounts, or

• a client who says one thing early on and corrects it later.

All of this has to land in the record correctly, and getting it wrong is the risk the tool is supposed to reduce. Live speaker labeling, for instance, can drift in the first few seconds of a conversation while the model figures out who’s talking.

Evaluating a voice tool is harder than evaluating a chatbot. When most of the interaction is audio rather than text, there’s no clean transcript to check the output against by default, so it’s tougher to measure whether the tool heard things right.

Look forward to our team sharing more about this focus area, including things like:

• How we approach evaluating a tool like this

• Feedback and ideas from caseworkers

• How to implement tools like this responsibility and securely

• And more!

If you do benefits casework, or build tools for people who do, we’d like to hear how this lands. Reach us at labs@navapbc.com. See more about the caseworker tools we’re prototyping, piloting and scaling at caseworker.navapbc.com.

Nava Labs is a philanthropically-funded R&D team within Nava, focused on prototyping innovations to the social safety net that government agencies would benefit from but can’t fund directly. We build and test new approaches to delivering public services, evaluate what works, and advocate for scaling proven solutions.

While we already regularly share news, case studies, insights, and toolkits on our main website, this Substack will offer something different: a look behind the scenes. Content will include:

• Posts from our new Technologist-in-Residence team on their experimental work as they prototype and test emerging approaches to public service delivery

• Updates on our program evaluation work, including data deep dives, while analysis is still in progress

• Our latest thought processes and frameworks, even as we’re still mulling them over

• Other field observations from our work

You can subscribe and get all of our posts straight to your email inbox (or notifications on the Substack mobile app), or you can simply check out posts like a normal blog.

We’re excited to engage fellow practitioners in this space, building toward the goal of improving service delivery for the public. We’d love to exchange ideas with you in the comments on our posts.

Let’s learn together!