In our first post on this topic, we argued that AI governance in public benefits should run as a feedback loop. We think the best place to first run and test that loop should be within an agency itself.
Through this work, we’ve scanned AI governance in all 50 U.S. states plus D.C., read 35 AI policies at different levels and scored them each against 25 policy provisions, built case studies of different states with AI policies, and held conversations with state agency leaders doing the actual work to find out what failed in practice. We landed on the finding that AI guardrails in public benefits should start at the agency level. By agency level, we mean two administrative layers: statewide IT or CIO policy and the benefits agency’s own policy. In the policies we reviewed, both layers were more specific than legislation or executive orders, and neither has to wait for a new law to act.
Here are three reasons we came to that conclusion.
1. Shadow AI is already happening, and the agency is held responsible for it
In December 2023, a child protection worker in Victoria, Australia discreetly used ChatGPT to draft a report for the Children’s Court. The generated report raised some concerns, as it was found to downplay the severity of potential harm to the child and had impacts on decisions about the child’s care. This sparked further investigation by the state’s privacy regulator who found that nearly 900 department employees – almost 13 percent of the workforce – had used ChatGPT in the prior six months.
The final investigation report blamed the agency, finding that the Department of Families, Fairness and Housing (DFFH) failed to “take reasonable steps to ensure the accuracy of personal information and to protect personal information from unauthorised disclosure.”
While that case is about child protection in Australia, the same pattern is already showing up in the US. One report, a survey of 3,335 public servants across 10 countries including the US, found that in “low-enablement environments,” 64% of enthusiastic AI workers report using personal logins, and 70% report using AI for work tasks without their manager knowing. Additionally, journalists in Washington State documented this pattern when they obtained thousands of pages of ChatGPT conversations from local officials: constituent replies, policy documents, and grant applications – none carrying the state-required disclosure, and many from personal accounts.
While we haven’t seen a records request on a SNAP or Medicaid eligibility office, we’d expect it to find a similar outcome: a caseworker using a personal login for an AI tool to transcribe their lengthy interview, draft their case notes, or answer a tricky policy question.
When staff don’t have clear guidance and approved tools, AI use doesn’t stop. It moves to where it can’t be tracked, evaluated, or made safe. We understand caseworkers are under immense pressure from changing policies, staffing shortages, and increased administrative burden, and we know why they’re turning to AI. But for tools that touch benefit decisions, this kind of risk can’t be tolerated. The Australia case made it clear that agencies can still be held responsible for AI outcomes, even when they don’t act.
Giving staff an approved alternative is part of that responsibility, but a list of tools alone isn’t enough: if no one owns the approval process or staff have to wait indefinitely for an answer, the list functions like a ban and pushes experimentation back toward personal accounts.
2. Agency policies can survive legislative churn
Legislation takes multiple cycles and can end up weaker than it started. Colorado passed the country’s first comprehensive state AI law in May 2024. Its original requirements were set to begin in February 2026, then delayed until June while lawmakers negotiated a compromise. In May 2026, lawmakers replaced it with a narrower framework: some provisions took effect immediately, while the remaining requirements begin in January 2027. That is two years of legislative revision during which the original law never took effect, while AI capabilities continued to change rapidly.
New York shows what the agency layer can do when a statute is weakened. The legislature rewrote the LOADinG Act in 2025, before it took effect, and dropped its human-review requirement. The state’s own AI policy, updated in June 2026, still says “Automated final decision systems are not permitted,” and names benefits eligibility in scope. It’s statewide IT policy rather than law, making enforcement penalties slightly weaker. However, that guardrail survived where the statute didn’t.
The federal litigation task force created in 2025 targets state AI laws, while the broader federal strategy expressly preserves requirements governing a state’s own procurement and use of AI. Agency procurement terms and internal-use rules therefore remain a separate layer of protection.
3. Agency policies contain specific rules and review requirements
Agency policies already carry the most specific rules. We scored 35 policies on how many of 25 common rules each one actually requires (note: these are selected policies, not a representative sample). The rules we looked for included things like required pre-deployment testing, prohibited use cases, and impact evaluations.
Statewide IT policies contained on average 9.2 out of 25 provisions, and SNAP agency policies 7.0, compared with 5.8 for state statutes and 1.0 for executive orders. Prohibited uses, data classification, and AI inventories all show up at the agency level more often than in statute – and these are some of the provisions we think are the most useful for protecting beneficiaries.
The agency layer is also where policies actually get revised. Of the 150 policies we checked, only 42 percent require their own review with a real cadence, owner, or trigger. Another 43 percent say something like “may be reviewed periodically,” which functions the same as saying nothing at all. State agency and IT policies carry a real review requirement more often than any other layer of policy.
But a feedback loop has two halves, and the second one is missing almost everywhere. None of the 10 state agency policies we scored requires an outcome evaluation of the AI it governs, or a way to feed evaluation results back into the policy. That’s one recommendation we added into the model agency policy we’re building.
The agency is where a definition of AI meets real tools. Defining “artificial intelligence” and deciding which tools need to be governed by the policy has been one of the biggest challenges we’ve seen states struggle with.
Are we only governing generative AI?
Does that include traditional machine learning or automated systems the agency already uses?
Which use cases do we need to be tracking more closely?
How do we prevent overburdening our staff with an approval process?
These are real questions states are grappling with, and ones we hope to help states think through in the model policy guidance we’re writing.
A statutory definition is not enough to produce a complete inventory. Agency policy can pair the definition with clear ownership, reporting procedures, and a current inventory of the tools staff actually use.
New York shows why. Its state law had one of the broadest definitions of AI of any policy we reviewed, but its first public inventory still returned no use cases from SNAP or several other agencies with high-profile AI tools already in production. The state law included a requirement to inventory all AI tools that “directly impact the public.” While they attempt to define the terminology, the law lacks useful operational detail, which may be one reason agencies failed to report their tools.
That is why a benefits agency still needs its own policy when a statewide one exists: the statewide policy sets the groundwork, and the benefits agency applies it more specifically to its programs, and keeps the inventory current.
Limitations of agency-level policy
Agency policy is not a substitute for law, particularly when it comes to claimant rights. The statewide policies in our latest research were much stronger on how agencies manage AI than on explanation, appeal, or opt-out rights. Agencies can test operating rules now; legislatures still have a role in codifying protections that should not depend on which administration is in office.
Agency-level policy as a legislative testing ground
We think states should start governing at the agency level, now. Not instead of legislation, but ahead of it. An agency can write and revise its guardrails on something closer to the timeline the technology actually moves on. The ineffective guardrails can be removed, and what works well can become a candidate for state legislation.
In the two states where we scored both a statewide policy and the benefits agency’s policy, the statewide policy was the more detailed of the two. This suggests a benefits agency doesn’t have to start from scratch; it can build on what its state IT office already requires and add the rules specific to its programs.
What’s next
We plan to present more of our thinking at Nava’s Open Source Summit: AI Futures on October 14.
We’re also using this research to build a model agency AI policy that agencies can adapt now and revise as the evidence changes. We especially want to hear what happens after a policy is adopted:
Which rules hold up
Which create work without reducing risk
Which ones staff route around
What would trigger a revision to your policy
If your agency is drafting an AI policy, or has one that isn’t doing what it was meant to, comment below or email us at labs@navapbc.com.