Skip to content

fix(analyzer): retain fenced SKILL.md findings - #276

Open
rodboev wants to merge 1 commit into
NVIDIA:mainfrom
rodboev:pr/skill-md-fenced-example-filtering
Open

fix(analyzer): retain fenced SKILL.md findings#276
rodboev wants to merge 1 commit into
NVIDIA:mainfrom
rodboev:pr/skill-md-fenced-example-filtering

Conversation

@rodboev

@rodboev rodboev commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

The static runner now retains TM1 findings from fenced commands in canonical SKILL.md. It keeps existing code-example filtering for ordinary Markdown, including non-canonical *skill.md filenames, and leaves the unchanged analyzer-local fenced-example bypasses disclosed as residual scope.

Root cause

_scan_path() treated SKILL.md as non-executable Markdown, then dropped findings whose context contained triple backticks. The TM1 analyzer emitted the finding, but the shared per-path runner removed it before reporting.

Changes

  • Kept the existing loose _is_skill_md(path) helper at the three legacy SKILL.md doc-filter exemption sites, and added _is_canonical_skill_md(path) for the exact-basename exemption in _scan_path(), which serves both run_static_patterns() and run_static_patterns_with_ledger().
  • Skipped the generic code-example hard-drop and downweight branch only for files literally named SKILL.md.
  • Added runner coverage for fenced SKILL.md, fenced guide.md, non-canonical *skill.md paths that must stay filtered, and the loose-vs-canonical path-helper boundary.
  • Added a no-LLM CLI regression that writes the issue-shaped fixture to tmp_path / "SKILL.md" and asserts TM1 plus nonzero before/after filtering counts.

Scope

This implements only the fenced SKILL.md example-filtering slice of issue #268. It doesn't change meta_analyzer.py, scoring bands, report schema, analyzer registration, provider code, or other issue slices. The retained finding remains in the current LOW/SAFE aggregate band; this change surfaces and counts it without retuning scoring. Analyzer-local fenced-example bypasses still remain for TM4, EA2, E5, AR, and MP because those modules filter upstream of the runner.

Verification

  • pytest tests/nodes/analyzers/test_static_runner_filtering.py tests/unit/test_cli.py -v, 93 passed
  • pytest tests/unit/test_cli.py::test_cli_scan_no_llm_reports_fenced_skill_md_tool_misuse -q, 1 passed
  • uv run ruff check src/ tests/
  • uv run ruff format --check src/ tests/
  • git diff --check

Base reproduction: issues: [], findings_before_filtering: 0, and findings_after_filtering: 0.

Head reproduction: the CLI regression asserts TM1 in issues and both filtering counts greater than zero.

Upstream

Refs #268

rng1995
rng1995 previously approved these changes Jul 24, 2026

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Automated SkillSpector Review]

Approved. Canonical SKILL.md fenced instructions no longer create a runner-level detection blind spot, while ordinary Markdown and noncanonical suffix names retain the existing example filtering. Runner and CLI regressions cover the boundary without changing report schemas.

@rng1995

rng1995 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

@rodboev - Please address review comments and resolve merge conflicts.

Signed-off-by: Rod Boev <rod.boev@gmail.com>
@rodboev
rodboev force-pushed the pr/skill-md-fenced-example-filtering branch from f1010a7 to b95352d Compare August 12, 2026 15:28
@rodboev

rodboev commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, I've rebased onto current main and resolved the conflicts.

  1. I preserved the new _scan_path() architecture and migrated the exact-basename SKILL.md exemption into that shared per-path filter, so both run_static_patterns() and run_static_patterns_with_ledger() use the same behavior.
  2. Canonical SKILL.md fenced findings still survive, while ordinary Markdown and non-canonical *skill.md filenames retain the existing code-example filtering.
  3. The runner and no-LLM CLI regressions remain aligned with the rebased implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants