Repository navigation
Description
Description
The validate and landconflictsignal controllers skip only when a request is in Error or halted. They do not check that the request is still in the validate stage. A redelivered or replayed message for a request that is already Batched is processed again, and a negative outcome terminates the request to Error while its batch keeps running.
A replayed SUCCEEDED signal rewinds Batched to Validated and republishes the request to the batch topic. In my e2e run the republish deduplicated and the batch still landed the request. #824 reports a 20-request run where a duplicate merge-conflict-check answer led the batch stage to create a second batch for the same request, which cost a full speculation and build cycle.
Steps to Reproduce
Observed in the e2e Compose stack at commit a76b075.
- Submit a request and let it reach Batched, with batch 1 speculating.
- Replay a not-landable landconflict signal for that request (the e2e test uses the message-replay helper).
- While batch 1 is still live, call cancel on the request and resubmit the same URI.
- Let both batches finish.
Repro tests are in imbik-ai@4bce6ff, one commit on top of a76b075. They assert the current behavior, so they pass without a fix:
- e2e:
TestStaleNotLandableSignal_FailsABatchedRequestandTestStaleValidationMessages_RewindABatchedRequestin test/e2e/submitqueue - unit:
TestProcess_SignalForAdvancedRequest(landconflictsignal) andTestController_Process_RedeliveryForAdvancedRequest(validate)
The same end state is reachable through validate: a redelivered validate message whose custom validator now rejects calls TerminateRequest on the Batched request. That path is reproduced in a unit test only.
Expected Behavior
A validate-stage message for a request that has already left the validate stage is acked and skipped. From Batched onward the batch owns the request's terminal outcome, as the comment on RequestStateBatched says and as the DLQ reconciler already enforces in dlq/request.go.
Actual Behavior
- Request 1 moves to
error. - Batch 1 continues to
succeededand Runway receives its merge request. Conclude logs "request already in different terminal state, skipping reconcile" (expectedlanded, actualerror). - Cancel returns success, but batch 1 carries on.
- A resubmission of the same URI is admitted as request 2, joins batch 2 and lands. Runway receives a merge request for each batch.
Environment
- Go version: 1.25.0 (the repo toolchain, GO_VERSION in MODULE.bazel)
- OS: Linux under WSL2, with Docker Desktop
- Bazel version: 8.4.1 (.bazelversion, run through ./tool/bazel)
Logs / Screenshots
Excerpt from the e2e run of TestStaleNotLandableSignal_FailsABatchedRequest:
STALE-FAILED after replayed signal: request 1 is "error"
STALE-FAILED request 1: status="error" last_error="replayed conflict" batch 1 is "speculating"
STALE-FAILED cancel while batch 1 is live: response= error=<nil>
STALE-FAILED resubmission of the same URI: response=sqid:"2" error=<nil>
conclude_controller request already in different terminal state, skipping reconcile {"batch_id": "1", "request_id": "1", "expected_state": "landed", "actual_state": "error"}
STALE-FAILED final: request 1 state="error" status="error" timeline=[accepted started validating validated batching batched speculating error cancelling speculated landing]; batch 1 is "succeeded"
STALE-FAILED final: resubmission 2 state="landed" status="landed" timeline=[... landing landed] batches=[2]
Additional Context
Trigger and limits:
- The e2e test injects the stale signal with the replay helper. I have not observed the negative-outcome case outside the test.
- Republished messages dedupe on message ID only while the original row exists. After garbage collection, a redelivered validate re-runs the conflict check, and a moved target branch can turn it negative.
- The stack uses the noop merger, so I can't say what a real merger does with the second merge request.
Duplicate deliveries on this path do happen. #824 reports a successfully acked message redelivered after about 12 seconds, with Runway answering the same merge-conflict-check twice. The harmful case here needs the second answer to differ from the first.
Proposed fix: ack and skip in validate unless the request is Started. Ack and skip in landconflictsignal unless it is Started or Validated (Validated stays allowed so a retry after a failed batch hand-off is not stranded). The fix is the next commit on the same branch, https://github.com/imbik-ai/submitqueue/tree/fix/stale-validation-redelivery, with the tests flipped. PR: #1114.
Related:
- A successfully acked message is redelivered long before its visibility timeout expires #824: a source of the duplicate deliveries this depends on.
- Batch-scoped DLQ reconciler fails requests that another live batch owns #826: the same end state (request reported
errorfor a change that lands), reached through the batch-scoped DLQ reconciler. It proposes mirroring the same guard in dlq/request.go. - Land and landsignal DLQs fail a
landingbatch that Runway goes on to merge #819: the same end state, reached through the land and landsignal DLQs.
Found with Rectifier (imbik.ai) and reproduced with unit and e2e tests.
Activity