ENG-10164 Add event-loop performance benchmark suite#6751
Open
Alek99 wants to merge 17 commits into
Open
Conversation
Contributor
Greptile SummaryThis PR adds a comprehensive performance benchmark suite for Reflex. The main changes are:
Confidence Score: 5/5The latest fixes look safe to merge.
Important Files Changed
Reviews (14): Last reviewed commit: "Fix more benchmark accuracy findings: re..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Merging this PR will degrade performance by 69.31%
|
Alek99
changed the title
…nchmarks Close the measurement gaps identified against peer frameworks (Phoenix LiveView, Blazor Server, Streamlit, Gradio, krausest): - wire-size suite recording serialized delta bytes per canonical interaction, plus received payload bytes in the load client - framework-overhead baseline against a bare Starlette + python-socketio echo server with byte-matched responses - latency-throughput curve with saturation-knee detection in event load - reconnect-storm scenario (simultaneous reconnect after all clients drop) - per-session backend memory with clients held in a subprocess - keyed-list browser ops (create/partial-update/select/swap) timed through paint on a new /rows page - compare_performance_results.py for baseline-vs-current regression gating with percentage thresholds and absolute floors - nightly release-scale workflow with a rolling cached baseline The production app harness is now a session fixture shared by the load, memory, and browser suites to avoid repeated production builds.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d88639608
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- guard the perf Redis helper: require an explicit db index in REFLEX_PERF_REDIS_URL and refuse to run against a non-empty database; make cleanup exception-safe and flush before close - measure the bundle budget against the shipped client bundle (.web/build/client) instead of all of .web; budget 500KB gzip (~2x a measured 238KB export), overridable via REFLEX_PERFORMANCE_JS_GZIP_BUDGET - bound all event-future waits in the PR-smoke event-loop suite with scale-aware timeouts; add timeout-minutes to the benchmarks CI job - benchmark the real header-decode path via reflex.app._decode_asgi_headers instead of a local copy; re.purge() per round so route-table construction is actually cold; drop asyncio.get_event_loop() from sync benchmarks - move processor construction/lifecycle out of measured regions in the event-loop benchmarks; assert shutdown drain actually drains; raise the drain budget so instrumentation slowdown cannot flip drain to cancel - replace the reconnect-storm settling sleep with token-cleanup polling, scale client timeouts with concurrency, and retry transient storm failures once before asserting - kill and report the session holder on wait timeout; add untraced warmup cycles before the memory leak gate; sweep pre-collected dev-backend children on teardown and retry lost port races - format each async stack frame once in diagnostics; integer Chrome trace tids with thread_name metadata; surface baseline-server thread crashes; use the public api_transformer seam; add tests/units/benchmarks/__init__.py - move the browser smoke into its own CI job and upload performance smoke artifacts
…enchmarks' into agent/event-loop-performance-benchmarks
FarhanAliRaza
changed the title
…asserts Address review findings that several benchmarks did not measure what they claim: - socket load client: prime + drain the hydrate/on_load deltas a fresh token emits so measured events are 1:1 request/response instead of pipelined two deep, fixing the event-load latency curve and framework-overhead comparison (which could otherwise report negative overhead). - browser hydration: gate on the websocket-driven is_hydrated marker instead of the prerendered #count text, so the timing spans real hydration; add the marker to the load app's index page. - route matching: seed a cached rxconfig so get_config() is a dict lookup as in production, instead of rebuilding Config (sys.path scan + env parse) on every match, which was 55-97% of the measured region. - process_event: purge the shared token per pedantic round so the cold path is actually exercised; CodSpeed's warmup otherwise left every round all-warm. - event-loop leak asserts: capture orphan task/future counts before stop() (which clears them), so the checks and metrics can catch a real leak; wait for the backend-exception task to drain in the failed-pipeline case. - wire size: encode with socket.io's compact separators so wire_bytes equals the real payload size instead of overstating it ~20%. - redis helper: close the manager before flushing (via a fresh client) so oplock lease write-backs land before the wipe and the next run's empty-db guard holds. - compiler lifecycle: keep zipping on the backend export so it measures real work rather than a no-op.
…th, ordering Second review pass: - state-manager cold get: run the measured construction inside an isolated RegistrationContext registering only PerformanceState, so it instantiates that state's subtree instead of every state in the collected session. Adding or removing an unrelated benchmark state no longer shifts this CodSpeed result. - event queue dispatch / future tree: run the measured dispatch inside the same context where start() attached the EventContext, so enqueue takes the production EventContext.get().fork() path instead of the LookupError fallback; build the event in per-round setup so only queueing and dispatch are measured. - wall-time ordering scenario: track handlers active per token and assert the peak stays 1. Completion-order alone can't catch a regression to concurrent same-token dispatch (equal sleeps still finish in order); the overlap counter can, and it's keyed per token so independent tokens still run concurrently. - compiler lifecycle: disable the live PyPI latest-version check so `init` timing is not contaminated by a network round-trip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Review fixes
BaseStateTokenhierarchy and persisted warm-state path in memory and Redis benchmarksreflex run --backend-onlywatcher and an observable live source-version endpointReview fixes (round 2)
REFLEX_PERF_REDIS_URLmust name an explicit db index and the database must be empty; cleanup is exception-safe.web/build/client, measured 238 KB gzip) instead of all of.web; 500 KB budget overridable viaREFLEX_PERFORMANCE_JS_GZIP_BUDGETtimeout-minutesreflex.app._decode_asgi_headers(extracted, behavior identical) instead of a local copy, andre.purge()per round so route-table construction is actually coldBenchmark model
tests/benchmarks/contains deterministic CPU work suitable for CodSpeed instrumentation.tests/performance/contains wall-time, service, process, memory, compiler, load, and browser scenarios; it is skipped unless--run-performanceis supplied.Shared schema-v1 reporting, environment metadata, percentiles, loop probes, pipeline traces, state/app fixtures, Redis isolation, Socket.IO clients, and diagnostics live under
tests/benchmarks/support/.CI scope
This draft adds pull-request smoke coverage to the existing performance workflow, plus artifact upload of the smoke reports. Per scope decision, it intentionally excludes all other former step-4 work: nightly or release workflows, automated profiler collection, runner validation, comparison tooling, and standalone external-load tooling.
Validation
uv run pytest tests/benchmarks --codspeed -q— 88 passeduv run pytest tests/units -q— 6645 passed, 78.04% coverageuv run pytest tests/performance -q— skipped by default; event-loop and memory smoke ran for real (2 passed), plus a real production export to measure the shipped bundleReview notes
The wall-time suites emit measurements without performance thresholds; regression gating should only be enabled after enough comparable baseline runs establish variance. Two narrow correctness gates are intentional: the shipped-bundle gzip budget (2x measured headroom, env-overridable) and the loop-probe detection bound (2 ms against a deliberately induced 10 ms block).