Repository navigation
Conversation
…nflictsignal Record what the two controllers do today when a message arrives for a request a batch has already claimed. Unit: landconflictsignal writes Validated over Batched and republishes to the batch topic on a landable result, and terminates the request on a not-landable one. validate re-runs the validator and either terminates the request or republishes the land-conflict check. E2E: with the request's build gated, a replayed SUCCEEDED result moves the request from Batched back to Validated while its batch is still speculating. A replayed FAILED result fails the request while its batch goes on to succeed, a cancel does not reach the batch, and a resubmission of the same URI is admitted and lands.
…past validation A redelivered validate message or land-conflict result could reach a request that a batch had already claimed. validate re-ran the validator and could terminate the request to Error; landconflictsignal wrote Validated over Batched, or terminated the request on a not-landable result. From Batched onward the batch owns the request's outcome, so both controllers now ack such deliveries without writing or publishing. landconflictsignal still lets Validated through: a retry lands there when the state write succeeded and the hand-off to batch did not.
|
|
bkgoksel
force-pushed
the
fix/stale-validation-redelivery
branch
2 times, most recently
from
November 18, 2026 05:06
2a1bb30 to
955270c
Compare
bkgoksel
marked this pull request as ready for review
November 18, 2026 05:14
bkgoksel
requested review from
a team,
behinddwalls and
sbalabanov
as code owners
November 18, 2026 05:14
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
validateandlandconflictsignalonly short-circuit for terminal or cancelling requests. Both topics are at-least-once, so a redelivery can arrive after dependency analysis has claimed the request for a batch. FromBatchedonward the batch owns the request's terminal outcome, but these two controllers still acted on it:Error. The batch kept running and merged the change. Conclude then left the request inError, a cancel no longer reached the batch, and a resubmission of the same URIs passed duplicate detection and landed as a second batch.ValidatedoverBatched. The republish to the batch topic normally deduplicates on message ID, so this one is mostly a state regression.The DLQ request reconciler already guards this case.
What?
validateacks without doing anything unless the request is stillStarted. No "validating" log, no validator call, no second land-conflict check.landconflictsignalacks without writing or publishing unless the request isStartedorValidated. The check sits before the outcome branch, so it covers both landable and not-landable results.Validatedstays allowed inlandconflictsignalbecause a retry lands there when the state write succeeded and the hand-off to batch failed. Skipping it would strand the request. This is the part I am least sure of: the cost is a version-only rewrite on a duplicate signal that arrives before the request is batched.redeliverLandConflictSignalhelper and two queues, one per new test.Notes for review:
Startedthe right and only pre-validation state forvalidateto accept? Nothing in the validate stage writes request state today.Test Plan
Unit:
./tool/bazel test //submitqueue/orchestrator/controller/validate/... //submitqueue/orchestrator/controller/landconflictsignal/...E2E, the two new tests (
TestStaleValidationMessages_LeaveABatchedRequestAlone,TestStaleNotLandableSignal_LeavesABatchedRequestAlone):Each holds a request in
Batchedbehind a build gate, replays the message, and asserts the request staysBatchedand then lands.Run locally with the fix:
make test(192 tests pass),make lint,make e2e-test(runway, stovepipe and submitqueue targets pass).Against the first commit alone, the e2e tests show the failure: after a replayed FAILED result, request 1 ends in
error, batch 1 reachessucceeded, and a resubmission of the same URI lands as batch 2.Issue
Closes #1115
Code, tests and commit messages were written with Claude Code.