AI Safety Challenge reviewers: watch the one-minute walkthrough · inspect evidence and limitations · rerun the demo.
The concrete scenario is an agent requesting deletion after a configuration change has removed its approval protection. The project demonstrates two checks:
- Before merging: ACS compares instructions with a trusted baseline and can reject a recognized loss of protection, even when both scores remain 100.
- Before deletion: a separate example host checks operator-issued approval against the exact target and reviewed file state before allowing a one-file operation.
The released CLI performs static review; installing it does not intercept agent actions. The executor prototype needs a controlled host and stable workspace. The linked evidence separates scripted tests, caller-operated observations and unverified real-world outcomes.
Catch known regressions in coding-agent instructions before merging.
AgentConfigScore runs local, deterministic checks on AGENTS.md, CLAUDE.md,
Cursor, Copilot and Gemini instructions. It compares a change with its baseline,
reports rule-defined regressions, and uses reviewed baseline policies for CI
decisions. No model API or runtime dependencies are needed.
Setup demo · Workflow diagram · Score history demo · Visual gallery
Deletion approval prototype: replay a lost approval requirement, repair its coverage, and try a separate approval-gated executor on disposable files. Includes scripted cases and explicit host trust boundaries. The walkthrough now includes a one-minute video and a separately labeled assistant-operated smoke record.
The demo uses v0.24.0 on a local fixture. Adding a remote-shell instruction
lowers the score from 100 to 82, identifies AGENTS.md:5, and fails the
new-error gate. The gallery also includes an instruction move drawn from a
merged PR, where both scores stay 100.
Reproduce the reduced PR case.
A 100 means no active deterministic rule matched. It does not certify instruction usefulness, semantic correctness or safety. Read the score contract and limitations.
Read a public-source case study: a generated-file false positive, explicit directory checks, and a failing development replay that caught a new warning before merge. The new path behavior is included in v0.26.2, with retained before/after evidence. The feedback-to-delivery map connects a public maintainer request to implemented features and their limits.
AgentConfigScore reviews changes to persistent agent instructions. The safety demo pairs that static check with a separate, host-controlled approval gate for deleting one disposable file.
| Review topic | Approach and evidence |
|---|---|
| Project and problem | Instruction changes can remove an approval requirement or stop it covering a target while both scores remain 100. The configuration regression walkthrough shows the lost directive, a failing review and its repair. |
| Safety approach | Configured CI gates block recognized regressions using the reviewed baseline policy. Separately, the example host refuses delete_file without a matching, unexpired, single-use approval bound to the target and reviewed file state. The operator issues approval; an agent's approved: true cannot issue it. Executor and trust boundary. |
| Demo and supporting evidence | One-minute rendered video, 12 scripted host-operation cases and retained results, and rerun commands. Self-use checks separately retain concrete detections and high-score blind spots. Reports identify their tool versions and inputs; the video renders retained reports rather than recording a live agent session. |
| Limitations | Installing ACS does not intercept agent actions. The executor is a separate one-file prototype, requiring a stable workspace and a host that denies other execution channels; it is not an OS sandbox. Scripted cases and the separately labelled caller-operated smoke do not establish independent agent safety, general accuracy or sustained external use. Full scope and limitations. |
| Change | Review evidence |
|---|---|
| Add a recognized unsafe command, exposed credential or broken path | Rule ID, explanation and source location where available |
| Delete an instruction file or reverse an exact directive | A regression-only finding, even when both numeric scores stay the same |
| Lower the candidate's threshold or add its own exception | The normal comparison still uses baseline policy and suppressions; trusted caller overrides are explicit |
| Accept a known exception | A reason, expiry and visible audit record |
Existing repositories can adopt the gate without first reaching 100.
init enables new-error blocking and no score decrease; a passing comparison
means no regression under the selected rules and policy.
Run inside your repository with a supported instruction file:
python -m pip install --upgrade agent-config-score
agent-config-score --version
agent-config-score init
agent-config-score doctor
agent-config-score diffReview and commit the generated .agentconfigscore.json and
.github/workflows/agent-config-score.yml. init does not write your agent
instructions or overwrite conflicting setup files by default. acs and
python -m agent_config_score are alternative entrypoints.
diff includes uncommitted changes. It detects a conservative local baseline
without fetching; use agent-config-score diff origin/main when you want an
explicit ref. The user guide
covers setup, exits and troubleshooting.
The v0.24.0 demo shows setup and selected checks from doctor in a local fixture.
v0.27.0 Beta includes local lint, baseline-governed regression checks, auditable exceptions, reports, safeguard review and opt-in Cursor attachment/coverage analysis. The Cursor profile is a bounded static prediction with explicit unknowns. See the release and upgrade notes.
An instruction can stop attaching even when its words and score stay unchanged. For a declared API-only task:
| Property | Baseline | Candidate |
|---|---|---|
| Instruction | Never delete files without approval. | Same text |
| Cursor glob | src/** |
src/web/** |
| Declared task file | src/api/service.py |
Same path |
| Ordinary scan score | 100 | 100 |
| Predicted automatic attachment | Active | Inactive |
From a repository checkout, install the package and run the synthetic example:
python -m pip install --upgrade "agent-config-score==0.27.0"
agent-config-score scope compare \
examples/cursor-scope-impact/base examples/cursor-scope-impact/narrowed \
--target src/api/service.py --fail-on-lost-directives --fail-on-unknownExpected exit 1: review is requested for the lost recognized active directive. Reports identify metadata and directive source lines. Another active copy or a remaining multi-file trigger can retain availability; manual/model-selected loading and unsupported inputs remain unknown.
The comparison models the same declared file context on both sides. It does not observe an agent run or prove semantic equivalence. Comparisons are advisory unless flags opt into the review gate; these flags are caller-owned and do not extend the existing baseline score-policy schema.
Try the four controls,
or use scope diff with a local Git baseline.
To find affected files without declaring a task, v0.25.0+ also provides
scope coverage --base-ref origin/main --under src. It inventories regular
files on both sides and checks each as a separate context, keeping an API-only
loss visible when a frontend file still matches. The
coverage guide and installed replay
include a repair, rename, uncertainty and unchanged controls.
agent-config-score init --coverage-under src --dry-run
agent-config-score init --coverage-under srcReview and commit the generated files. The opt-in read-only workflow combines
the normal gate with strict per-file coverage, a readable job summary and
seven-day report artifacts. Existing setups can use --workflow-only and an
explicit reviewed --force. See the complete adoption path
and design and shipped boundary.
The generated policy enables regression checks:
{
"$schema": "https://raw.githubusercontent.com/LE0-Lin/AgentConfigScore/v0/schema/agentconfigscore.schema.json",
"version": 1,
"policy": {
"max_drop": 0,
"fail_on_new_errors": true
}
}Normal diff / compare and the reusable Action use the baseline's policy,
so candidate threshold edits cannot approve that same change. The candidate
config is still validated. Explicit CLI/Action overrides belong to the trusted
caller. Policy defaults, schema and overrides.
An accepted exception requires a stable rule ID, reason and expiry. Suppressed findings remain visible. Normal regression comparisons apply the baseline suppression set to both sides; a PR cannot authorize its own new exception. Suppression format and audit behavior.
Discovery includes root/nested AGENTS.md and AGENTS.override.md,
CLAUDE.md, GEMINI.md, legacy .cursorrules, .cursor/rules/*.md / *.mdc,
.github/copilot-instructions.md, .github/instructions/*.md, .claude/**/*.md,
.clinerules and .windsurfrules. .agentconfigscoreignore excludes discovery paths.
Format discovery does not imply complete language or native loading semantics.
The dedicated Cursor profile models root .cursor/rules/ separately and treats
plain .md as ignored there. See scope/override handling
and copy-ready templates.
- Maintained synthetic contracts and negative controls check defined behavior. Known semantic misses remain visible; these are not real-world accuracy estimates.
- The current pinned-repository smoke corpus has five completed scans and one expected source rejection. Pinned Reaparr is rejected for a dangling instruction link and receives no score. Its historical A 100 predates strict source validation and is not the current result.
- An independent comparison with direct AI review has not been completed. See the comparison protocol. Scanning public repositories does not establish adoption.
A high score can miss vague guidance, paraphrased contradictions or harmful text outside the rule grammar. Local analysis does not call a model API or execute instruction-file commands. Reports can contain repository details; review them before sharing. Limits and privacy-minimized feedback drafts.
| Task | Reference |
|---|---|
| Install, initialize, understand exits | User guide |
| Configure policy, CLI, Action inputs/outputs or SARIF | Detailed reference |
| Explain findings or set up optional PR comments | Maintainer workflow |
| Review missing deletion safeguards | Opt-in guardrail suggestions |
| Inspect installed rules | agent-config-score rules |
| Keep score history | History workflow |
| Browse all topics | Documentation index |
The chart shows three locally recorded fixture scores read by
agent-config-score history. Set up the history workflow.
The visual tour explains each demo. Choose 简体中文 or 繁體中文 for the localized overview, guide and limitations; advanced references remain English.
python -m pip install -e .
python -m unittest discover -s tests -v
agent-config-score . --fail-under 90Contributions and reproducible false-positive/regression cases are welcome. MIT licensed.


