feat(stovepipe): add DLQ reconciliation for the process stage#347
Open
gsturges wants to merge 1 commit into
Open
feat(stovepipe): add DLQ reconciliation for the process stage#347gsturges wants to merge 1 commit into
gsturges wants to merge 1 commit into
Conversation
A request whose process-stage message exhausts retries would previously sit in a non-terminal state forever, indistinguishable from "not yet validated" to callers gating on greenness. Per the fail-closed policy in doc/rfc/stovepipe/workflow.md, such requests must be driven to a conservative not-green terminal state instead. - Add RequestStateFailed terminal state and IsRequestStateTerminal helper to the entity package. - Add stovepipe/controller/dlq: a consumer.Controller that drains the process_dlq topic and CAS-transitions the affected request to failed, releasing the queue's in_flight_count slot first when the request had already been admitted (processing), per the in_flight_count integrity rule in doc/rfc/stovepipe/steps/process.md. - Wire a second consumer in the stovepipe server using errs.AlwaysRetryableProcessor, register the process_dlq topic with DLQ disabled (no cascade) and raised retry attempts, and start/stop the two consumers in the safe order (DLQ up first, drained last). Reconciliation is idempotent: already-terminal requests are skipped, and optimistic locking lets a late legitimate pipeline transition win. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gsturges
requested review from
a team,
behinddwalls and
sbalabanov
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A request whose
process-stage message exhausts retries would previously sit in a non-terminal state forever — indistinguishable from "not yet validated" to callers gating deployments on greenness. Per the fail-closed policy in doc/rfc/stovepipe/workflow.md, such requests must be driven to a conservative not-green terminal state instead.RequestStateFailedterminal state andIsRequestStateTerminalhelper.stovepipe/controller/dlqpackage: aconsumer.Controllerthat drains theprocess_dlqtopic and CAS-transitions the affected request tofailed. If the request had already been admitted (processing), the queue'sin_flight_countslot is released first, per the integrity rule in doc/rfc/stovepipe/steps/process.md.errs.AlwaysRetryableProcessor(a reconciliation failure retries in place — the DLQ subscription is a final destination withDLQ.Enabled = false, so nothing can cascade to a_dlq_dlq);process_dlqregistered in the topic registry with raised retry attempts; DLQ consumer starts first and stops last.Reconciliation is idempotent: already-terminal requests are skipped, and optimistic locking lets a late legitimate pipeline transition win. The design mirrors SubmitQueue's proven
submitqueue/orchestrator/controller/dlqpattern, simplified for stovepipe's single payload shape.Only
processis a queue-consuming step today (ingestis synchronous RPC); future stages (build,buildsignal,record) can reuse the same pattern.Test plan
stovepipe/controller/dlq/dlq_test.go: accepted→failed, processing→failed with slot release, already-terminal no-ops, not-found no-op, CAS version-mismatch paths, malformed/empty payloadsgo build ./...andgo test ./stovepipe/... ./service/stovepipe/...passmake gazelle,make fmt,make lint-licenseclean🤖 Generated with Claude Code