Skip to content

fix(framework): preserve remote eval row origins - #660

Merged
Abhijeet Prasad (AbhiPrasad) merged 2 commits into
mainfrom
codex/remote-eval-origin
Aug 5, 2026
Merged

fix(framework): preserve remote eval row origins#660
Abhijeet Prasad (AbhiPrasad) merged 2 commits into
mainfrom
codex/remote-eval-origin

Conversation

@cjgalione

@cjgalione Curtis Galione (cjgalione) commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • preserve valid inline row origins through dictionary-to-EvalCase conversion
  • prefer a complete active dataset-row identity over preserved source provenance
  • propagate the resolved origin to root spans, streamed progress events, and EvalResult
  • omit malformed origins and update the relevant public/internal types

Why

Remote playground requests include each selected row's origin, but Python's EvalCase.from_dict() dropped it because EvalCase did not declare the field. Streamed progress events consequently emitted a null origin, preventing the playground from associating results and scores with source rows and making runs appear stuck at 0%.

Validation

  • .venv/bin/pytest src/braintrust/test_framework.py -k 'origin' -q
  • .venv/bin/pytest src/braintrust/devserver/test_dataset.py -k 'returns_inline_data_unchanged' -q
  • .venv/bin/nox -s test_core
  • .venv/bin/nox -s test_types
  • .venv/bin/nox -s pylint
  • pre-commit hooks for the changed files

Tracking

resolves SDK-204

@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 4941422 into main Aug 5, 2026
82 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the codex/remote-eval-origin branch August 5, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants