What Is Eval-to-Production Exposure?

The short answer: eval-to-production exposure is everything production an AI evaluation setup can reach through the files and configs it runs with — live cloud credentials, wildcard IAM permissions, public storage and open network paths declared in eval files. Evaluations run against real systems with real permissions, so whatever is declared in those files is reachable by the eval agent, and by anything that compromises it. EvalEgress scans declared files for the 16 exposure classes below and scores what it finds (reviewed November 2026). Run the free scan on your own files.

Why do evaluations touch production at all?

An eval that never touches reality measures nothing. Teams point evaluation harnesses at real APIs, real data shapes and the same SDK configuration the application uses, because fidelity is the point — a sandbox with fake credentials cannot tell you how the agent behaves with the real ones. The standard shortcut is copying production configs, environment files and IAM policies into the eval folder and trimming later. "Later" is where the exposure lives: the copied file still declares the live credential, the wildcard policy, the public bucket.

The boundary problem is asymmetry. Production systems are hardened, reviewed and monitored; eval folders are scratch space — committed to side repos, shared in tickets, pasted into tools. The eval agent itself is also an actor: it executes tools, reads files and follows instructions found in eval data. Anything the eval environment can reach, a steered or buggy agent can reach too.

What crosses the boundary? The 16 exposure classes

EvalEgress's scanner checks declared content against 16 finding classes in four families. Severity weights the exposure score: CRITICAL 25, HIGH 15, MEDIUM 8, LOW 3.

Family / class	Severity	What it means
private_key_block	CRITICAL	A private key header in declared content — revoke, purge, move to a secrets store.
admin_managed_policy	CRITICAL	AdministratorAccess / PowerUserAccess referenced — replace with least-privilege.
wildcard_principal	CRITICAL	Policy principal * — can expose the resource publicly.
aws_access_key_id, aws_secret_access_key, github_token, generic_api_key, generic_secret_assignment	HIGH	Secret-like patterns — presence, file, and line reported; values never stored or shown.
wildcard_iam_action, wildcard_iam_service	HIGH	"Action": "*" or "s3:*" — scope to exact APIs.
public_bucket_acl, open_ingress	HIGH	Public-read storage, 0.0.0.0/0 ingress — restrict to who needs it.
jwt_token, open_binding, debug_mode_enabled	MEDIUM	Token material, all-interface binds, debug on — tighten before exposure.
debug_route	LOW	Debug/admin route strings — confirm they're not production-exposed.

The full scoring method is on /docs. The network side of the boundary gets its own guide: what is egress in AI evaluation?

Exposure is not exploitability — and that is the point

EvalEgress finds exposure in declared surfaces. It never proves exploitability, and it cannot see private infrastructure — the scope note is printed on every report. That honesty is what makes the scan useful before deployment: a declared live credential is a fact about your files, not a speculation about an attacker. You do not need to prove someone would use the eval folder's admin policy; the finding is that the boundary you believed in is not in the files.

How do you check your own boundary?

Paste the declared files your eval runs with — configs, env files, IAM policies, infra definitions — into the free exposure scan. Fix CRITICAL findings first (private keys, admin policies, wildcard principals), then HIGH. Re-scan whenever the eval configuration changes; exposure creeps back in with every copied file. The step-by-step version is the pre-deployment eval safety checklist. EvalEgress is one of the Ingenious AI platforms.

Frequently asked questions

What is eval-to-production exposure?
Everything production that an AI evaluation setup can reach through the files and configs it runs with — live credentials, wildcard IAM permissions, public storage and open network paths declared in eval files. Whatever is declared there is reachable by the eval agent, and by anything that compromises it.

Does EvalEgress connect to my production systems?
No. You paste declared files or config text; the scanner reads only what you declare. No live systems are ever touched, and the report states that scope on its face.

Which findings matter most?
The CRITICAL classes: a private key block, an admin managed policy (AdministratorAccess or PowerUserAccess), and a wildcard principal. Severity weights the exposure score — CRITICAL 25, HIGH 15, MEDIUM 8, LOW 3 — so one CRITICAL outweighs a pile of LOWs.

Are the secret values in my files stored or shown?
Never. Secret-like findings report presence, file and line only; the values are redacted and are never stored or displayed.

How often should an eval setup be scanned?
Before its first production-connected run, and again on every configuration change. Exposure usually arrives as a copied file, not a decision — the re-scan is what catches the copy.