Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -624,9 +624,11 @@ jobs:
fi
- name: Score the corpus on real C#
# Precision is gated absolutely (every fix silent, zero false positives);
# recall is pinned at the measured floor (the frontend catches the
# subscription/region class; pool/dispose/handoff shapes are the itemized
# extraction backlog) and ratchets up as the extractor grows. Raise the
# floor whenever recall improves — a drop below it is a regression.
run: python scripts/benchmark.py --min-recall 3
# recall is pinned at the measured floor and ratchets up as the extractor
# (and the corpus fixtures) improve. Now 4/9 — the loaded-subscription case
# was understated (its reduction referenced an undeclared VM type -> OWN050,
# not a verdict); with a self-contained fixture our subscription detection
# catches it. pool/dispose/handoff shapes are the remaining backlog. Raise
# the floor whenever recall improves — a drop below it is a regression.
run: python scripts/benchmark.py --min-recall 4

16 changes: 16 additions & 0 deletions corpus/real-world/screentogif-loaded-subscription/after.cs
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,19 @@ private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs
private void OnHideError(object sender, EventArgs e) => StatusBand.Hide();
private void OnClose(object sender, EventArgs e) => DialogResult = true;
}

// Minimal in-file stand-ins so the reduction is self-contained (mirrors before.cs).
// With the events resolvable, the matching `-=` in Window_Closing releases each
// subscription, so the extractor stays silent — the fix is clean.
public sealed class VideoSourceViewModel
{
public event EventHandler ShowErrorRequested;
public event EventHandler HideErrorRequested;
public event EventHandler CloseRequested;
}

internal static class StatusBand
{
public static void Error(string message) { }
public static void Hide() { }
}
20 changes: 20 additions & 0 deletions corpus/real-world/screentogif-loaded-subscription/before.cs
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,23 @@ private void Window_Loaded(object sender, RoutedEventArgs e)
// Present in the real file, but it does NOT detach the handlers above.
private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e) { }
}

// Minimal in-file stand-ins so the reduction is self-contained. The real
// VideoSourceViewModel lives elsewhere in ScreenToGif, so in isolation the
// type-aware extractor cannot bind `_viewModel.ShowErrorRequested` to an event
// and honestly emits OWN050 (unresolved) instead of the OWN001 subscription leak
// — which the benchmark scored as a miss. With the type resolvable, the extractor
// flags the leak (warning-tier: an injected `DataContext` source it cannot prove
// outlives the window), exactly as it does on the full repo.
public sealed class VideoSourceViewModel
{
public event EventHandler ShowErrorRequested;
public event EventHandler HideErrorRequested;
public event EventHandler CloseRequested;
}

internal static class StatusBand
{
public static void Error(string message) { }
public static void Hide() { }
}
32 changes: 24 additions & 8 deletions docs/notes/corpus-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,31 @@ benchmark: 3/9 bugs caught in real C# · 9/9 fixes clean · 0 false positive(s)

That is the honest day-one number, and it is *sharp*: **specificity is perfect**
(every real fix is silent, zero false positives — the checker does not cry wolf on
correct code), and **recall is 3/9** — the three caught are exactly the
correct code), and **recall was 3/9** — the three caught are exactly the
subscription/region class the extractor is strongest at (`zombie-viewmodel` →
OWN001, two static-event escapes → OWN014). The six missed
(`arraypool-double-return`, `arraypool-use-after-return`, `ownership-handoff-consume`,
`screentogif-loaded-subscription`, `handler-use-after-dispose`,
`viewmodel-escapes-to-app`) are cases the `.own` reductions *all* catch but the C#
**frontend** does not yet extract — pool double-return/use-after-return,
ownership-handoff, and a few dispose/escape shapes. The benchmark just quantified
the frontend's recall debt and turned it into an itemized to-do list.
OWN001, two static-event escapes → OWN014).

### Ratchet → 4/9: a fixture was understating us

The first thing the number bought was a *diagnosis*. `screentogif-loaded-subscription`
is a **subscription** leak — our strongest class — yet it scored a miss. The cause
was not the extractor: the reduction's `before.cs` subscribed to
`_viewModel.ShowErrorRequested` but **never declared `VideoSourceViewModel`**, so
the type-aware extractor could not bind the `+=` to an event and honestly emitted
`OWN050` (unresolved) — a `note`, not a verdict. The fixture was understating our own
detection. Making the reduction self-contained (a minimal `VideoSourceViewModel` with
the three events, mirrored in `after.cs`) lets the extractor resolve it and flag the
leak (warning-tier — an injected `DataContext` source) exactly as it does on the full
ScreenToGif repo. **Recall is now 4/9** and the floor is raised to match. (Lesson: a
benchmark fixture that references an undeclared type silently degrades to `OWN050`;
self-contained fixtures, like the samples, measure honestly.)

The remaining five misses are genuine **frontend extraction gaps** — pool
double-return (`OWN003`) and use-after-return (`OWN002`), the interprocedural
ownership-handoff (`OWN001`+`OWN002`), a field/cross-method use-after-dispose, and a
region-escape shape — the `.own` reductions all catch them, the C# extractor does not
yet. That is the itemized recall backlog; each is a real capability the floor will
ratchet up to as it lands.

## Why catch/clean, not exact-code match

Expand Down
16 changes: 10 additions & 6 deletions docs/proposals/P-012-bug-corpus-mining.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,16 @@
*real* `before.cs`/`after.cs` (not just the `.own` reduction), measuring recall
(the bug is caught) and specificity (the fix is silent), gated in the
`corpus-benchmark` CI job. This is the measurement spine — the defensible number,
and the verifiable reward for any future learning loop. **First measurement: 3/9
caught · 9/9 fixes clean · 0 false positives** — perfect precision, and the C#
*frontend's* recall debt is now a tracked number (the `.own` reductions all fire;
the 6 missed are pool/dispose/handoff shapes the extractor does not yet lower —
the itemized extraction backlog). Still ahead: raising recall case-by-case, GitHub
mining at scale (stage 1) and the 50–100-repo prevalence scan (stage 2). See
and the verifiable reward for any future learning loop. First measurement **3/9
caught · 9/9 clean · 0 FP**, already ratcheted to **4/9**: the
`screentogif-loaded-subscription` miss was a *fixture* understating us (its
reduction referenced an undeclared VM type → `OWN050`, not a verdict); a
self-contained fixture lets our subscription detection catch it. Perfect precision
throughout. The remaining 5 misses are genuine frontend extraction gaps
(pool double-return/use-after-return, interprocedural handoff, a cross-method
use-after-dispose, a region-escape shape) — the tracked recall backlog the floor
ratchets up to. Still ahead: more case-by-case recall, GitHub mining at scale
(stage 1) and the 50–100-repo prevalence scan (stage 2). See
[docs/notes/corpus-benchmark.md](../notes/corpus-benchmark.md).
- **Depends on:** P-001 (C# → OwnIR extractor — the scanner that does stage 2);
the existing `corpus/` layout (`before.cs`, `after.cs`,
Expand Down
Loading