Over the past couple of years, Nava Labs has been partnering with governments, nonprofits, and universities to ship AI-enabled tools to production environments.
In May 2026, we decided to open source our conglomeration of Caseworker Empowerment Tools.
With this work, Nava Labs maintains rigorous, multi-layered generative AI guardrails to ensure everything we ship meets the needs of caseworkers serving real people. We use strategies like automated schema enforcement (i.e. Zod), deterministic processing (simple function calls), observability with Open Telemetry standards, and persistent logging to ensure we can trace requests and responses from start to finish. Most critically: every tool allows for a “human in the loop” at various review gates to ensure safety and accuracy.
Our commitment to safety and technical integrity of rolling out these solutions is non-negotiable.
Why guardrails matter
Generative AI can fail convincingly. An answer can sound useful while creating or citing the wrong policy, exposing sensitive data, or taking an action the user never intended. We’ve all been there when we slightly push back against Claude or ChatGPT and receive a you are absolutely right message (see: Stochastic Parrot).
In public-benefit work, those failures could cause irreparable harm to caseworkers and the people they intend to serve. That is unacceptable.
Guardrails belong throughout the entire software development lifecycle.
How we embed AI guardrails across the software development lifecycle
Guardrails aren’t an afterthought or a quick prompt adjustment — they belong woven directly into the software development lifecycle. Here is how Nava Labs embeds safety, control, and observability from initial design to active maintenance:
1. Plan by setting hard boundaries
Before writing a single line of code, we define the system’s exact perimeter. Nava Labs does this by explicitly mapping what data the model can see. What state changes it can trigger. And which high-stakes decisions an automated tool is strictly forbidden from making.
2. Build by coding for least privilege
We don’t rely on the model to “behave” itself. We attempt to enforce this behavior in code. We do this by validating every input and output with strict type enforcement (like Zod schemas), sandboxing untrusted web content inside virtual machines, scoping retrieval strictly to vetted policy manuals and documents, and granting tools the absolute minimum access they need to operate.
3. Test by breaking things on purpose
Models fail in weird ways. So, we actively try to trip them up before caseworkers ever touch them. We can do this by running golden-set benchmarks, monitoring for model drift over time, and running aggressive security stress-tests. Specifically, targeting direct and indirect prompt injection attacks (including sneaky aria-tag injections hidden in scraped websites or documents).
4. Operate by watching the wires in real time
Shipping to production is just the start. We maintain total visibility by tracing every end-to-end request. Typically by using OpenTelemetry standards, tracking real-time failure modes, and keeping a fully tested, one-click rollback path ready if a model update acts up.
5. Review by keeping humans in the loop
At the end of the day, the machine is just a tool helping an individual. We review what is proposed by surfacing code-verified citations, confidence scores, and missing-field alerts directly to caseworkers, ensuring human navigators always have the context, time, and explicit authority to approve, correct, or shut down a workflow.
We released our Caseworker Empowerment Toolkit as open source on GitHub so that other teams can inspect, challenge, and improve these patterns with us. Feel free to dive in!
Questions to keep in mind when building with generative AI
These questions below are really for teams building with and implementing generative AI to think about throughout the entire delivery lifecycle.
What can the model see, and what can it change?
What happens when a retrieved document or webpage contains malicious instructions?
Can we trace every important claim or action to its source?
Which failures stop the workflow instead of triggering another model call?
What tests run when the model, prompt, data, or tools change?
Who can disable or roll back the system when something goes wrong?
Does the human reviewer have the evidence, time, and authority to disagree?
Etc.
Have you been doing any work on implementing LLM guardrails? We’d love to hear about it in the comments!