Skip to content

doc(stovepipe): build and buildsignal pipeline steps#336

Open
roychying wants to merge 5 commits into
uber:mainfrom
roychying:chenghan.ying/stovepipe-build/buildsignal-doc
Open

doc(stovepipe): build and buildsignal pipeline steps#336
roychying wants to merge 5 commits into
uber:mainfrom
roychying:chenghan.ying/stovepipe-build/buildsignal-doc

Conversation

@roychying

@roychying roychying commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Why?

Stovepipe's pipeline had design docs for process but nothing for the two stages downstream of it - build (triggers the build-runner) and buildsignal (polls it to completion). Both stages needed a documented contract before implementation could start, in particular because stovepipe can't reuse SubmitQueue's BuildRunner interface as-is: SubmitQueue validates stacks of batches, stovepipe validates a single URI against a baseline, and that mismatch has to be resolved deliberately rather than discovered mid-implementation.

What?

  • doc/rfc/stovepipe/steps/build.md (new): documents the build stage end to end - the trigger-only algorithm, idempotency posture under at-least-once redelivery, error classification, edge cases (head == last-green, Phase 2 re-triggers), and the new Build entity/BuildStore/queue contract it needs. Defines the stovepipe BuildRunner contract (design sketch) with URI-based Trigger(headURI, baseURI, metadata) instead of SubmitQueue's batch-based trigger, explains why the contracts are kept separate (materialize-vs-checkout mismatch, baseline semantics batches can't express, URI ownership), and lays out - as open discussion, not settled decisions - alternative shapes for sharing the contract (single platform-level interface, metadata-smuggled scope, shared-backend-thin-contracts) and alternatives for how the build id could be keyed/deduped.
  • doc/rfc/stovepipe/steps/buildsignal.md (new): documents the buildsignal poll loop - reusing SubmitQueue's buildsignal shape (poll cadence, PublishAfter-not-Nack primitive, error classification) pipe diverges: it only forwards to record once terminal

Test Plan

N/A

Issue

@roychying roychying requested review from a team, behinddwalls and sbalabanov as code owners July 10, 2026 21:04
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


chenghan.ying seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@roychying roychying force-pushed the chenghan.ying/stovepipe-build/buildsignal-doc branch 2 times, most recently from 112337e to 1abb551 Compare July 10, 2026 22:47
@roychying roychying force-pushed the chenghan.ying/stovepipe-build/buildsignal-doc branch from 1abb551 to 4e4d789 Compare July 10, 2026 22:58
Comment thread doc/rfc/stovepipe/steps/build.md Outdated

It handles only the trigger: it does not poll for completion, record greenness, or decide incremental-vs-full — those are `buildsignal`'s, `record`'s, and `process`'s jobs respectively.

`build` is structurally the same controller as `submitqueue/orchestrator/controller/build/build.go`, and [doc/rfc/submitqueue/build-runner.md](../../submitqueue/build-runner.md) is the reference rationale for the trigger-then-poll shape this stage reuses. The `BuildRunner` contract's `Status`/`Cancel` half now carries over as literally shared code with SubmitQueue, via `platform/extension/buildrunner`; what does *not* carry over is `Trigger` — the two domains model different problems, so each keeps its own, following the [extension-contract.md](../../submitqueue/extension-contract.md) "identity in, resolve internally" principle.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use relative to root paths than relative to current

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to have it as buildrunner extension at the platform vs per technologu say buildkite, i don't think the idea of breaking the contract in places so defining somewhat similar thing in 2 places for status and cancel is fine..and we abstract away by the underlying impl like buildkite to so ? thoughts

@roychying roychying Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. paths fixed.
  2. yes and this is one of the alternatives I'd already recorded. The shared-interface route does give a compile-time guarantee that both domains' Status/Cancel stay identical (I believe they should be?), but it costs fragmenting one conceptual contract across two packages, plus migrating SubmitQueue's already-shipped code onto promoted platform/base types for a half only SubmitQueue uses today. Your approach is what the doc now adopts.

Comment thread doc/rfc/stovepipe/steps/build.md Outdated
Comment thread doc/rfc/stovepipe/steps/build.md Outdated
Comment thread doc/rfc/stovepipe/steps/build.md Outdated
Comment thread doc/rfc/stovepipe/steps/build.md Outdated
The shape isn't decided here because project semantics belong to `analyze`, not `build`: how a project maps to a buildable scope (a Bazel target pattern, a directory, a service name) is implementer-specific per [workflow.md](../workflow.md#project---greenness-at-a-finer-grain). The expectation is that this stays an opaque token — following the same "identity in, resolve internally" shape already used for `headURI`/`baseURI` (owned and interpreted by `SourceControl`) — that `build` reads off the `Request`/message and hands to the runner uninterpreted, rather than a structured type `build` would have to understand:

```go
Trigger(ctx context.Context, headURI, baseURI string, projectScope entity.ProjectScope, metadata base.BuildMetadata) (base.BuildID, error)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be added anyway in both SQ and Stovepipe... currently there no metadata as such from SQ to be passed but i can think of things which might be valuable to pass like conflict graph info or any other decisions which maybe relevant to build.

cc: @JamyDev

@roychying roychying Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we decided the concrete shape here? my preference is leaving it deferred until there's a real consumer in either domain..

Comment thread doc/rfc/stovepipe/steps/build.md Outdated
@roychying roychying changed the title (doc)stovepipe: build and buildsignal pipeline steps doc(stovepipe): build and buildsignal pipeline steps Jul 13, 2026
@roychying roychying force-pushed the chenghan.ying/stovepipe-build/buildsignal-doc branch from a437451 to 78a3df5 Compare July 13, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants