From 417fb9b01bcda96d66a425f033e95cc828f8cb03 Mon Sep 17 00:00:00 2001 From: Chenghan Ying <125171961+roychying@users.noreply.github.com> Date: Mon, 16 Nov 2026 20:51:53 -0800 Subject: [PATCH] feat(submitqueue): Publish request hook events from the gateway to the orchestrator's hook stage --- doc/rfc/hook-framework.md | 3 ++- doc/rfc/submitqueue/workflow.md | 4 ++-- .../submitqueue/gateway/server/BUILD.bazel | 4 ++++ .../gateway/server/log_consumer.go | 3 +++ .../gateway/server/log_consumer_test.go | 22 +++++++++++++++++++ service/submitqueue/gateway/server/main.go | 4 ++-- submitqueue/gateway/README.md | 4 ++++ 7 files changed, 39 insertions(+), 5 deletions(-) diff --git a/doc/rfc/hook-framework.md b/doc/rfc/hook-framework.md index 186add8d2..3d655f2ae 100644 --- a/doc/rfc/hook-framework.md +++ b/doc/rfc/hook-framework.md @@ -4,7 +4,7 @@ Fire-and-forget side effects for pipeline lifecycle events: one shared event con ## Status -Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns. +Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage. The SubmitQueue gateway's request materializer publishes `request.received`, `request.landed`, and `request.failed` (`submitqueue/core/hookevent`) to it; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns. ## Problem @@ -52,6 +52,7 @@ Delivery promise: - Partition key = subject id; per-subject order only. - Replay finding the target state already written → republish (idempotent). Beyond it → superseded; the event may be lost. Named, accepted gap. - RPC-caused transitions are at-most-once; needing the guarantee means publishing from the first queue-driven stage. +- SubmitQueue request events follow the gateway's request summary, the status users see, rather than pipeline transitions, so a hook agrees with the UI even for a request whose start failed before the orchestrator saved it. Each status announces one event per change, so a hook retrying one change's failure never repeats another's. A finish is announced on every delivery of a log that restates it, which keeps the replay rule above: a failed publish fails the delivery and its redelivery republishes. `request.received` is published inside `Land`, which makes it RPC-caused and at-most-once; a later status standing in for a lost `accepted` log announces it instead. The events cross from the gateway to the orchestrator's hook stage, so their payload carries the change's URI, which never changes. - Opt-in per host via topic-key registration; a registered host never skips, so off and loss are distinguishable. ### Contract diff --git a/doc/rfc/submitqueue/workflow.md b/doc/rfc/submitqueue/workflow.md index a050b2357..2a40b16e8 100644 --- a/doc/rfc/submitqueue/workflow.md +++ b/doc/rfc/submitqueue/workflow.md @@ -6,7 +6,7 @@ Two cycles re-enter speculation. `speculate` dispatches funded paths to `build`; Terminal request states are not `conclude`'s alone. `cancel` completes a request that has not been enrolled in a batch. `landconflictsignal` fails a request Runway reports as conflicted. `conclude` maps a terminal batch onto its member requests. The DLQ reconcilers force a failed terminal state when a consumed stage cannot finish. -The same `Stages` table registers a `submitqueue-hook` stage, outside the order of the flow above. It consumes hook events and invokes the wired hooks, and it publishes nothing onward. No orchestrator controller publishes a hook event; `service/submitqueue/orchestrator/server` resolves every event to noop. The comment on that row in `pipeline.go` is the contract: any stage can publish there. +The same `Stages` table registers a `submitqueue-hook` stage, outside the order of the flow above. It consumes hook events and invokes the wired hooks, and it publishes nothing onward. The gateway's request materializer publishes request events there (see the [gateway README](../../../submitqueue/gateway/README.md)); no orchestrator controller does, and `service/submitqueue/orchestrator/server` resolves every event to noop. The comment on that row in `pipeline.go` is the contract: any stage can publish there. ## Diagram @@ -131,7 +131,7 @@ class node_storagecontract,node_queuestorage,node_requestlog,node_messagequeue t class node_submitter toneIndigo ``` -`hook` is drawn with the other stages because the topology registers it. Nothing in the orchestrator publishes to it, so the diagram has no edge into that node. +`hook` is drawn with the other stages because the topology registers it. Only the gateway's request materializer publishes to it, so the diagram has no edge into that node from an orchestrator stage. ## Per-controller summary diff --git a/service/submitqueue/gateway/server/BUILD.bazel b/service/submitqueue/gateway/server/BUILD.bazel index 5e1709930..cb2682eb6 100644 --- a/service/submitqueue/gateway/server/BUILD.bazel +++ b/service/submitqueue/gateway/server/BUILD.bazel @@ -19,6 +19,7 @@ go_library( importpath = "github.com/uber/submitqueue/service/submitqueue/gateway/server", visibility = ["//visibility:private"], deps = [ + "//api/base/hook:go_default_library", "//api/submitqueue/gateway/protopb:go_default_library", "//platform/consumer:go_default_library", "//platform/errs:go_default_library", @@ -93,6 +94,7 @@ go_test( ], embed = [":gateway_lib"], # keep deps = [ + "//api/base/hook:go_default_library", "//api/submitqueue/gateway/protopb:go_default_library", "//platform/base/failure:go_default_library", "//platform/base/messagequeue:go_default_library", @@ -103,12 +105,14 @@ go_test( "//platform/extension/messagequeue/mock:go_default_library", "//platform/lifecycle:go_default_library", "//submitqueue/core/messagequeue:go_default_library", + "//submitqueue/core/topickey:go_default_library", "//submitqueue/entity:go_default_library", "//submitqueue/extension/queueconfig/mock:go_default_library", "//submitqueue/gateway/controller:go_default_library", "//submitqueue/gateway/core/request:go_default_library", "//submitqueue/gateway/extension/storage:go_default_library", "//submitqueue/gateway/extension/storage/mock:go_default_library", + "//submitqueue/orchestrator:go_default_library", "@com_github_stretchr_testify//assert:go_default_library", "@com_github_stretchr_testify//require:go_default_library", "@com_github_uber_go_tally//:go_default_library", diff --git a/service/submitqueue/gateway/server/log_consumer.go b/service/submitqueue/gateway/server/log_consumer.go index b52e23a68..263af1a36 100644 --- a/service/submitqueue/gateway/server/log_consumer.go +++ b/service/submitqueue/gateway/server/log_consumer.go @@ -20,6 +20,7 @@ import ( "time" "github.com/uber-go/tally" + basehook "github.com/uber/submitqueue/api/base/hook" "github.com/uber/submitqueue/platform/consumer" "github.com/uber/submitqueue/platform/errs" genericerrs "github.com/uber/submitqueue/platform/errs/generic" @@ -47,6 +48,8 @@ func newGatewayTopicRegistry(queue extqueue.Queue, subscriberName string) (consu {Key: topickey.TopicKeyStart, Name: "start", Queue: queue}, {Key: topickey.TopicKeyCancel, Name: "cancel", Queue: queue}, {Key: topickey.TopicKeyLog, Name: "log", Queue: queue, Subscription: primary}, + // Publish-only: the orchestrator pipeline's hook stage consumes it under this name. + {Key: basehook.TopicKeyHook, Name: "submitqueue-hook", Queue: queue}, { Key: logDLQTopicKey, Name: "log" + logDLQSuffix, Queue: queue, Subscription: extqueue.DLQSubscriptionConfig(subscriberName, logDLQConsumerGroup), diff --git a/service/submitqueue/gateway/server/log_consumer_test.go b/service/submitqueue/gateway/server/log_consumer_test.go index ad52e957e..e54869839 100644 --- a/service/submitqueue/gateway/server/log_consumer_test.go +++ b/service/submitqueue/gateway/server/log_consumer_test.go @@ -22,6 +22,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/uber-go/tally" + basehook "github.com/uber/submitqueue/api/base/hook" "github.com/uber/submitqueue/platform/base/failure" entityqueue "github.com/uber/submitqueue/platform/base/messagequeue" "github.com/uber/submitqueue/platform/consumer" @@ -30,10 +31,12 @@ import ( queuemock "github.com/uber/submitqueue/platform/extension/messagequeue/mock" "github.com/uber/submitqueue/platform/lifecycle" sqmq "github.com/uber/submitqueue/submitqueue/core/messagequeue" + "github.com/uber/submitqueue/submitqueue/core/topickey" "github.com/uber/submitqueue/submitqueue/entity" requestcore "github.com/uber/submitqueue/submitqueue/gateway/core/request" gwstorage "github.com/uber/submitqueue/submitqueue/gateway/extension/storage" gwstoragemock "github.com/uber/submitqueue/submitqueue/gateway/extension/storage/mock" + "github.com/uber/submitqueue/submitqueue/orchestrator" "go.uber.org/mock/gomock" "go.uber.org/zap" "google.golang.org/grpc" @@ -112,6 +115,25 @@ func TestLogConsumers_RetainFailedRecovery(t *testing.T) { } } +func TestGatewayTopicRegistry_MatchesOrchestratorStages(t *testing.T) { + registry, err := newGatewayTopicRegistry(queuemock.NewMockQueue(gomock.NewController(t)), "worker") + require.NoError(t, err) + stageNames := map[consumer.TopicKey]string{} + for _, stage := range orchestrator.Stages { + stageNames[stage.Key] = stage.Name + } + + for _, key := range []consumer.TopicKey{topickey.TopicKeyStart, topickey.TopicKeyCancel, basehook.TopicKeyHook} { + t.Run(string(key), func(t *testing.T) { + want, ok := stageNames[key] + require.True(t, ok, "the orchestrator consumes every topic the gateway publishes to") + got, ok := registry.TopicName(key) + require.True(t, ok) + assert.Equal(t, want, got) + }) + } +} + type lifecycleTestConsumer struct { consumer.Consumer name string diff --git a/service/submitqueue/gateway/server/main.go b/service/submitqueue/gateway/server/main.go index eca086794..43ce33fd8 100644 --- a/service/submitqueue/gateway/server/main.go +++ b/service/submitqueue/gateway/server/main.go @@ -334,7 +334,7 @@ func run() error { // Land creates receipt projections, request-summary and List controllers // read materialized views, and request-history controllers read retained // logs. Normal log-topic persistence and terminal DLQ repair both use the - // materializer. + // materializer, which also publishes request hook events. store, err := mysqlstorage.NewStorage(appDB, scope.SubScope("storage")) if err != nil { return fmt.Errorf("failed to create storage: %w", err) @@ -343,7 +343,7 @@ func run() error { // Create controllers and wrap them for gRPC. Every store is queue-scoped and // resolves through the factory adapter; land/cancel/log share one materializer. storageFty := storageFactory{backend: store} - materializer := requestcore.NewMaterializer(storageFty, requestcore.WithMetricsScope(scope)) + materializer := requestcore.NewMaterializer(storageFty, requestcore.WithMetricsScope(scope), requestcore.WithHookPublishing(registry)) pingController := controller.NewPingController(logger, scope) landController := controller.NewLandController(logger.Sugar(), scope, cnt, storageFty, materializer, queueConfigs, queuePauses, registry) cancelController := controller.NewCancelController(logger.Sugar(), scope, storageFty, materializer, registry) diff --git a/submitqueue/gateway/README.md b/submitqueue/gateway/README.md index 24eefced5..259d7f9f9 100644 --- a/submitqueue/gateway/README.md +++ b/submitqueue/gateway/README.md @@ -25,6 +25,10 @@ The gateway owns the request log read model and is the only service that reads i The materializer appends every audit event, selects the current authoritative winner, and repairs the queue projection. The normal orchestrator pipeline does not read or write the request-log store directly. +## Request hook events + +The materializer publishes SubmitQueue's request hook events (`submitqueue/core/hookevent`) to the `submitqueue-hook` topic, which the orchestrator pipeline's hook stage consumes, one event per change of the request. The update that takes the summary out of `accepting` announces `request.received`; that happens inside `Land`, so a failed receipt publish is only counted on `request_hook.publish_errors`. Every delivery of a log that restates a `landed` or `error` summary announces `request.landed` or `request.failed`, so a failed publish is returned and the log's redelivery repairs it; a copy that arrives after the queue has forgotten the event id reaches hooks twice. Events therefore follow the summary users see, including a request whose start failed before the orchestrator saved it, and a stale or unfinished log publishes nothing. + ## Final log recovery The primary `gateway-log` consumer sends failed deliveries to `log_dlq` after a non-retryable error or exhaustion of its finite retry budget. The gateway's `gateway-log-dlq` consumer replays retained entries through the same log controller and materializer, with unlimited attempts, capped backoff, and no second-level DLQ. Valid log deliveries are acknowledged only after history, summary, URI mappings, and receipt projection succeed. Replays may append equivalent history rows; terminal-summary precedence and mapping repair are unchanged.