Guides · ← All guides · Reviewed November 2026 · 6 min read
Pre-Deployment Eval Safety Checklist
Run this checklist before any eval touches production: credentials, IAM scope, network binds, debug surface, and a re-scan habit.
The short answer: before an eval runs against anything production-connected: get live credentials out of the declared eval files, scope IAM to exact APIs with no wildcards and no admin policies, bind services to localhost, close 0.0.0.0/0 ingress and public buckets, turn debug off, and run a declared-file scan to prove all of it. Every item below maps to one of EvalEgress's 16 exposure classes, so the free scan can verify the checklist mechanically (reviewed November 2026).
Credentials — nothing live in declared files
- ☐ No private key blocks in any eval file (
private_key_block, CRITICAL) - ☐ No live cloud keys — AWS access key IDs or secret keys (
aws_access_key_id,aws_secret_access_key, HIGH) - ☐ No GitHub tokens or generic API keys / secret assignments (
github_token,generic_api_key,generic_secret_assignment, HIGH) - ☐ No JWT token material pasted into configs (
jwt_token, MEDIUM) - ☐ Eval credentials, where unavoidable, are eval-scoped and rotatable — and stored in a secrets store, not the repo
IAM — least privilege, no wildcards
- ☐ No AdministratorAccess or PowerUserAccess anywhere in the eval path (
admin_managed_policy, CRITICAL) - ☐ No policy principal of
*(wildcard_principal, CRITICAL) - ☐ No
"Action": "*"and no service-wide wildcards like"s3:*"(wildcard_iam_action,wildcard_iam_service, HIGH) - ☐ Every permission names the exact APIs the eval calls — read the policy the way the scanner does
Network — the eval box is not a public endpoint
- ☐ No 0.0.0.0/0 ingress on eval infrastructure (
open_ingress, HIGH) - ☐ Results storage is private — no public-read bucket ACLs (
public_bucket_acl, HIGH) - ☐ Harness and results viewers bind to localhost or the private network (
open_binding, MEDIUM)
Debug surface — off before it counts
- ☐ Debug mode disabled in the eval configuration (
debug_mode_enabled, MEDIUM) - ☐ Debug/admin route strings reviewed — not production-exposed (
debug_route, LOW)
Process — prove it, then keep proving it
- ☐ Declared files scanned with the free exposure scan; zero CRITICAL findings before the first production-connected run
- ☐ The scan report is kept with the eval run's records (what was declared, what was found, what was fixed)
- ☐ Re-scan is wired to configuration changes — exposure returns as copied files, not decisions
Background reading: what is eval-to-production exposure? and what is egress in AI evaluation? The scoring behind every item is documented on /docs. EvalEgress is one of the Ingenious AI platforms.
Frequently asked questions
When should this checklist run?
Before the eval's first production-connected run, and again whenever its configuration changes. The checklist is cheap; the findings it prevents are the CRITICAL classes — private keys, admin policies, wildcard principals.
What is the minimum viable pass?
Zero CRITICAL findings and no live credentials in declared files. HIGH items (wildcards, open ingress, public buckets) should be fixed or explicitly accepted in writing by whoever owns the eval — silent acceptance is how exposure becomes normal.
Who owns the checklist?
The team shipping the eval harness. The findings live in their files, and the fixes — scoping a policy, closing a bind, moving a secret — are changes only they can make.
Does passing the checklist guarantee the eval is safe?
No. The scan reads declared surfaces and finds exposure; it never proves exploitability and cannot see private infrastructure. Passing means the declared files no longer hand an eval agent the keys, the permissions and the network path — the three things an incident needs.