Skip to content

Stale validate-stage messages can fail a request whose batch is live #1115

Open

Description

@bkgoksel

Description

The validate and landconflictsignal controllers skip only when a request is in Error or halted. They do not check that the request is still in the validate stage. A redelivered or replayed message for a request that is already Batched is processed again, and a negative outcome terminates the request to Error while its batch keeps running.

A replayed SUCCEEDED signal rewinds Batched to Validated and republishes the request to the batch topic. In my e2e run the republish deduplicated and the batch still landed the request. #824 reports a 20-request run where a duplicate merge-conflict-check answer led the batch stage to create a second batch for the same request, which cost a full speculation and build cycle.

Steps to Reproduce

Observed in the e2e Compose stack at commit a76b075.

  1. Submit a request and let it reach Batched, with batch 1 speculating.
  2. Replay a not-landable landconflict signal for that request (the e2e test uses the message-replay helper).
  3. While batch 1 is still live, call cancel on the request and resubmit the same URI.
  4. Let both batches finish.

Repro tests are in imbik-ai@4bce6ff, one commit on top of a76b075. They assert the current behavior, so they pass without a fix:

  • e2e: TestStaleNotLandableSignal_FailsABatchedRequest and TestStaleValidationMessages_RewindABatchedRequest in test/e2e/submitqueue
  • unit: TestProcess_SignalForAdvancedRequest (landconflictsignal) and TestController_Process_RedeliveryForAdvancedRequest (validate)

The same end state is reachable through validate: a redelivered validate message whose custom validator now rejects calls TerminateRequest on the Batched request. That path is reproduced in a unit test only.

Expected Behavior

A validate-stage message for a request that has already left the validate stage is acked and skipped. From Batched onward the batch owns the request's terminal outcome, as the comment on RequestStateBatched says and as the DLQ reconciler already enforces in dlq/request.go.

Actual Behavior

  • Request 1 moves to error.
  • Batch 1 continues to succeeded and Runway receives its merge request. Conclude logs "request already in different terminal state, skipping reconcile" (expected landed, actual error).
  • Cancel returns success, but batch 1 carries on.
  • A resubmission of the same URI is admitted as request 2, joins batch 2 and lands. Runway receives a merge request for each batch.

Environment

  • Go version: 1.25.0 (the repo toolchain, GO_VERSION in MODULE.bazel)
  • OS: Linux under WSL2, with Docker Desktop
  • Bazel version: 8.4.1 (.bazelversion, run through ./tool/bazel)

Logs / Screenshots

Excerpt from the e2e run of TestStaleNotLandableSignal_FailsABatchedRequest:

STALE-FAILED after replayed signal: request 1 is "error"
STALE-FAILED request 1: status="error" last_error="replayed conflict" batch 1 is "speculating"
STALE-FAILED cancel while batch 1 is live: response= error=<nil>
STALE-FAILED resubmission of the same URI: response=sqid:"2" error=<nil>
conclude_controller  request already in different terminal state, skipping reconcile {"batch_id": "1", "request_id": "1", "expected_state": "landed", "actual_state": "error"}
STALE-FAILED final: request 1 state="error" status="error" timeline=[accepted started validating validated batching batched speculating error cancelling speculated landing]; batch 1 is "succeeded"
STALE-FAILED final: resubmission 2 state="landed" status="landed" timeline=[... landing landed] batches=[2]

Additional Context

Trigger and limits:

  • The e2e test injects the stale signal with the replay helper. I have not observed the negative-outcome case outside the test.
  • Republished messages dedupe on message ID only while the original row exists. After garbage collection, a redelivered validate re-runs the conflict check, and a moved target branch can turn it negative.
  • The stack uses the noop merger, so I can't say what a real merger does with the second merge request.

Duplicate deliveries on this path do happen. #824 reports a successfully acked message redelivered after about 12 seconds, with Runway answering the same merge-conflict-check twice. The harmful case here needs the second answer to differ from the first.

Proposed fix: ack and skip in validate unless the request is Started. Ack and skip in landconflictsignal unless it is Started or Validated (Validated stays allowed so a retry after a failed batch hand-off is not stranded). The fix is the next commit on the same branch, https://github.com/imbik-ai/submitqueue/tree/fix/stale-validation-redelivery, with the tests flipped. PR: #1114.

Related:

Found with Rectifier (imbik.ai) and reproduced with unit and e2e tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions