How AI-generated test docs can improve safety net applications
Synthetic documents, real impact
When someone needs support for food, health insurance, veterans benefits, and other public programs, the experience is driven by documents. Someone uploads a proof of income, or medical expenses, or proof of enrollment in an educational program. Staff work with those documents to make sure they’re accurate, valid, and meet requirements.
Much of the work to improve the experience of these programs is about making that flow better: getting files uploaded correctly, making sure they’re readable, and letting people know as early as possible if they need to resubmit something, so they don’t wait weeks to hear back.
As part of my role on the Nava Labs program, I work on our Caseworker Empowerment Toolkit, which supports staff who connect people to benefits. In this toolkit, one of the questions we’re exploring is: how might document AI tools help staff classify documents, extract information, and figure out whether what someone’s shared will actually get them the benefits they’re looking for?
And as we develop this tool, I keep hitting a challenge of finding realistic documents to test with that don’t include people’s sensitive data. This is a challenge common to any project trying to improve benefit program experiences.
Why testing document workflows is hard
The main obstacle is creating or gathering documents that account for different scenarios applicants may be in. Many benefits require proof of income, which could come from salaried employment, self-employment, or the gig economy, to name a few.
To address this, teams either hand-create or find sample documents and hope they cover enough of the situations people are in. Another path is using real documents, but this requires securely handling people’s most sensitive data, so it’s not a good fit for early development and testing.
There are so many permutations of scenarios, document types, and input methods, and there’s a need to test as many as possible before anything is put in front of the public or staff. Synthetic documents add another testing method to the toolkit, one that doesn’t depend on real sensitive information.
Why now
Previously, ChatGPT and other image-generation models were unreliable. Many of us have seen examples of images where a person has six fingers, or text with typos and garbled characters. The quality was so low that the output was essentially useless for testing.
Gpt-image-2 was released this spring and now an entire document can now be generated. Not perfectly, but fairly consistently: full-page layouts, different document structures, text that refers to the same person throughout, and dates and amounts that line up well enough to support testing.
A quick comparison of models
Here’s the same prompt for a synthetic pay stub, run through the old model and the new one.
Test 1: Paystubs
gpt-image-1: Synthetic pay stub generated by gpt-image-1, showing garbled words like “Duringar” and “Gotalc” and numbers that don’t add up.
gpt-image-2: Synthetic pay stub generated by gpt-image-2, with a clean layout, itemized earnings and deductions, and year-to-date totals that reconcile (at least at first glance).
Test 2: SNAP award letters
The same jump shows up in a SNAP award letter. The gpt-image-1 version misspells “Effective” and “Caseworker” and signs off with “Sercrndy.” The gpt-image-2 version produces a full approval notice with a benefit summary, EBT card instructions, and fair hearing rights, all consistent with one made-up household.
The synthetic SNAP award letter from gpt-image-2 makes a much more internally consistent approval notice with a benefit summary and fair hearing rights.
Note: If anything, Image2 makes this document look too polished, since many real SNAP award letters are simple B&W text in a letter with an organization logo in the header
What this makes possible
This lets us start to more systematically test document experiences, rather than relying purely on ad-hoc examples from the internet or handmade documents from a few people on an internal team. If someone has income from self-employment or the gig economy, we can generate documentation from the different apps and services they use and see how our systems would handle it at a first pass. And if we hear about a scenario where experiences are poor, we can generate documents for that scenario specifically. So we get speed, we get depth of experimentation, and we can do all of that without using people’s real information.
This process also has potential applications in detecting, or training staff to detect, fake documents created with AI.
The flipside
The images are better, but they can still contain errors, and as the quality goes up, the errors become harder to notice. Fewer sixth fingers, more income amounts off by a few dollars. No synthetic document is a replacement for human-generated documents and human testing. It’s all a supplement and a way to expand the tools we have available to make getting and receiving benefits less onerous for the public and staff.
Further work
Deeper evaluation: The images are great for coarse document classification and generally seem to follow instructions for household composition, dates, income amounts, and other details. But further human validation is needed before scaling up synthetic document creation as a testing method.
Improving tools to support batch creation + sharing of documents
Tailoring prompts: Continuing to improve layouts that don’t match real documents, increasing formats (phone photos vs. scans vs. handwritten notes) and quality (blurriness, stains, crumpling, etc.)
