Learnings hygiene

Suppressions are like dependencies: easy to add, easy to forget. A suppression that was correct six months ago can be silently wrong today, because the code that justified it has moved, been deleted, or has had its surrounding context change. This playbook is the workflow we use to keep the suppression set honest over time.

Why suppressions go stale

Three failure modes account for almost all stale suppressions:

1. The path moved. A marker says # sebastion: ignore X in lib/legacy/**, but lib/legacy/ was renamed to lib/v1/ in a refactor. The marker now suppresses nothing. The rule re-fires and somebody re-suppresses it on the new path, never noticing the old marker was now dead weight.
2. The justification disappeared. A chat-created Learning was added when the input was sanitised three calls upstream. That sanitisation was removed in a later PR. The Learning now silences a real finding.
3. The rule itself changed. Rare, but when we tune a rule to be more precise, a suppression that was needed under the old definition may no longer be needed under the new one. The suppression becomes harmless clutter rather than a security risk, but it still costs cognitive overhead.

The first two are the dangerous ones. Hygiene is mostly about catching them before they bite.

The quarterly review

Every three months, spend an hour on this:

Step 1. Dump the active Learnings

The dashboard at app.foundationmachines.ai/sebastion/learnings lists every active Learning for your installation, grouped by source (chat / auto_detect / manual) with count, created_at and last_seen_at.

The count and last_seen_at fields are your starting point:

• count = 0 for an auto_detect Learning means the marker exists in your AGENTS.md (or similar) but no finding has ever matched it. Either you got the glob wrong, or the rule has been permanently quiet on that path. Investigate.
• last_seen_at more than 90 days ago for any Learning is a signal that the underlying pattern may have moved. Worth a look.

Step 2. Walk the agent-file markers

Open every file Sebastion reads markers from:

• AGENTS.md
• CLAUDE.md
• .cursorrules
• .github/copilot-instructions.md

…plus any nested copies (e.g. bin/AGENTS.md, packages/legacy/AGENTS.md). For each # sebastion: ignore line:

1. Does the glob still match real files? git ls-files against the glob is the quickest check.
2. Is the rule still firing on those files? If count = 0 from step 1, the marker may have been written defensively for a rule you have since stopped triggering. Defensible to leave; defensible to remove for clarity.
3. Is the reason still true? This is the hardest question. The marker doesn't store the reason — only the rule and glob. The reason should live in a # comment line above the marker so the reviewer can re-evaluate it in a year.

Recommended format for new markers (and a refactor target for old ones):

# Our bin/ scripts intentionally use console.log for CLI output.
# Suppressing the rule there keeps the audit signal clean without
# disabling it project-wide. Reviewed: 2026-05.
# sebastion: ignore js.console-log in scripts/**

The "Reviewed: YYYY-MM" line gives you a quick visual cue at the next quarterly review of which markers have been examined recently.

Step 3. Walk the chat-created Learnings

In the dashboard, filter to source = chat. For each entry:

1. Open the original PR comment thread (linked from the audit trail) and re-read the reason the contributor gave.
2. Decide: is this still a one-off legitimate exception, or has it become a pattern across multiple files? If the same (rule, directory) shows up three or more times, promote it to an AGENTS.md marker and drop the individual chat Learnings (via support, until self-serve delete ships).

The dashboard caps at 500 active Learnings per installation. If you are at 80%+ of cap, we will already have alerted you — but the quarterly review is when you should be pruning, not when you hit the ceiling.

Step 4. Walk the disable_rules

Open .sebastionai.yml. For each entry in disable_rules:

1. Is this still org policy? Did the team that owned it move on? Did the framework change?
2. Has the rule become more precise? Sometimes a rule that was too noisy to keep on six months ago is now fine to re-enable. Try removing it on a branch and seeing what fires.

Operational rules that pay off

A few habits that compound:

• Every new marker gets a # comment line above it explaining why. Without this, the next reviewer has no context and is forced to either preserve the suppression defensively or risk removing it without understanding.
• Markers live close to the code they describe. A marker scoped to packages/legacy/** is better placed in packages/legacy/AGENTS.md than in the repo-root AGENTS.md — the scope of the marker matches the scope of the file, so renaming packages/legacy/ to something else takes the marker with it.
• Chat suppressions with no free-text reason are a smell. When you see one, follow up on the PR thread asking why. The reason is recorded in your event log only if it was given.
• Quarterly review is on the calendar. If it is not scheduled, it does not happen. We do ours on the first Monday of every quarter.

What to do when you find a problem

For each kind of issue, here is the action:

Related

• False positives and Learnings — full security model, lifecycle and audit trail of every Learning.
• Suppression strategy — how to pick the right mechanism in the first place.
• Rate limits — the per-install caps that make pruning matter.