Skip to content

fix(orchestrator): skip validate and landconflictsignal for requests past validation - #1114

Open
bkgoksel wants to merge 2 commits into
uber:mainfrom
imbik-ai:fix/stale-validation-redelivery
Open

bkgoksel wants to merge 2 commits into
uber:mainfrom
imbik-ai:fix/stale-validation-redelivery

Conversation

@bkgoksel

@bkgoksel bkgoksel commented Nov 18, 2026 •

Copy link
Copy Markdown

Why?

validate and landconflictsignal only short-circuit for terminal or cancelling requests. Both topics are at-least-once, so a redelivery can arrive after dependency analysis has claimed the request for a batch. From Batched onward the batch owns the request's terminal outcome, but these two controllers still acted on it:

  • A replayed not-landable result, or a replayed validate whose validator now rejects, terminated the request to Error. The batch kept running and merged the change. Conclude then left the request in Error, a cancel no longer reached the batch, and a resubmission of the same URIs passed duplicate detection and landed as a second batch.
  • A replayed landable result wrote Validated over Batched. The republish to the batch topic normally deduplicates on message ID, so this one is mostly a state regression.

The DLQ request reconciler already guards this case.

What?

  • validate acks without doing anything unless the request is still Started. No "validating" log, no validator call, no second land-conflict check.
  • landconflictsignal acks without writing or publishing unless the request is Started or Validated. The check sits before the outcome branch, so it covers both landable and not-landable results.
  • Validated stays allowed in landconflictsignal because a retry lands there when the state write succeeded and the hand-off to batch failed. Skipping it would strand the request. This is the part I am least sure of: the cost is a version-only rewrite on a duplicate signal that arrives before the request is batched.
  • The e2e suite gains a redeliverLandConflictSignal helper and two queues, one per new test.

Notes for review:

  • Is Started the right and only pre-validation state for validate to accept? Nothing in the validate stage writes request state today.
  • The first commit records the current behaviour with tests that pass against main; the second flips them. Squash if you prefer one commit.

Test Plan

Unit:

./tool/bazel test //submitqueue/orchestrator/controller/validate/... //submitqueue/orchestrator/controller/landconflictsignal/...

E2E, the two new tests (TestStaleValidationMessages_LeaveABatchedRequestAlone, TestStaleNotLandableSignal_LeavesABatchedRequestAlone):

./tool/bazel test //test/e2e/submitqueue:go_default_test --test_filter=TestE2EIntegration \
  --test_arg=-testify.m='TestStaleValidationMessages|TestStaleNotLandableSignal'

Each holds a request in Batched behind a build gate, replays the message, and asserts the request stays Batched and then lands.

Run locally with the fix: make test (192 tests pass), make lint, make e2e-test (runway, stovepipe and submitqueue targets pass).

Against the first commit alone, the e2e tests show the failure: after a replayed FAILED result, request 1 ends in error, batch 1 reaches succeeded, and a resubmission of the same URI lands as batch 2.

Issue

Closes #1115

Code, tests and commit messages were written with Claude Code.

…nflictsignal

Record what the two controllers do today when a message arrives for a
request a batch has already claimed.

Unit: landconflictsignal writes Validated over Batched and republishes to
the batch topic on a landable result, and terminates the request on a
not-landable one. validate re-runs the validator and either terminates
the request or republishes the land-conflict check.

E2E: with the request's build gated, a replayed SUCCEEDED result moves the
request from Batched back to Validated while its batch is still
speculating. A replayed FAILED result fails the request while its batch
goes on to succeed, a cancel does not reach the batch, and a resubmission
of the same URI is admitted and lands.
…past validation

A redelivered validate message or land-conflict result could reach a
request that a batch had already claimed. validate re-ran the validator
and could terminate the request to Error; landconflictsignal wrote
Validated over Batched, or terminated the request on a not-landable
result. From Batched onward the batch owns the request's outcome, so both
controllers now ack such deliveries without writing or publishing.

landconflictsignal still lets Validated through: a retry lands there when
the state write succeeded and the hand-off to batch did not.
@cla-assistant

cla-assistant Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cla-assistant

cla-assistant Bot commented Nov 18, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@bkgoksel
bkgoksel force-pushed the fix/stale-validation-redelivery branch 2 times, most recently from 2a1bb30 to 955270c Compare November 18, 2026 05:06
@bkgoksel
bkgoksel marked this pull request as ready for review November 18, 2026 05:14
@bkgoksel
bkgoksel requested review from a team, behinddwalls and sbalabanov as code owners November 18, 2026 05:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale validate-stage messages can fail a request whose batch is live

1 participant