Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/rfc/hook-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Fire-and-forget side effects for pipeline lifecycle events: one shared event con

## Status

Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns.
Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage. The SubmitQueue gateway's request materializer publishes `request.received`, `request.landed`, and `request.failed` (`submitqueue/core/hookevent`) to it; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns.

## Problem

Expand Down Expand Up @@ -52,6 +52,7 @@ Delivery promise:
- Partition key = subject id; per-subject order only.
- Replay finding the target state already written → republish (idempotent). Beyond it → superseded; the event may be lost. Named, accepted gap.
- RPC-caused transitions are at-most-once; needing the guarantee means publishing from the first queue-driven stage.
- SubmitQueue request events follow the gateway's request summary, the status users see, rather than pipeline transitions, so a hook agrees with the UI even for a request whose start failed before the orchestrator saved it. Each status announces one event per change, so a hook retrying one change's failure never repeats another's. A finish is announced on every delivery of a log that restates it, which keeps the replay rule above: a failed publish fails the delivery and its redelivery republishes. `request.received` is published inside `Land`, which makes it RPC-caused and at-most-once; a later status standing in for a lost `accepted` log announces it instead. The events cross from the gateway to the orchestrator's hook stage, so their payload carries the change's URI, which never changes.
- Opt-in per host via topic-key registration; a registered host never skips, so off and loss are distinguishable.

### Contract
Expand Down
4 changes: 2 additions & 2 deletions doc/rfc/submitqueue/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Two cycles re-enter speculation. `speculate` dispatches funded paths to `build`;

Terminal request states are not `conclude`'s alone. `cancel` completes a request that has not been enrolled in a batch. `landconflictsignal` fails a request Runway reports as conflicted. `conclude` maps a terminal batch onto its member requests. The DLQ reconcilers force a failed terminal state when a consumed stage cannot finish.

The same `Stages` table registers a `submitqueue-hook` stage, outside the order of the flow above. It consumes hook events and invokes the wired hooks, and it publishes nothing onward. No orchestrator controller publishes a hook event; `service/submitqueue/orchestrator/server` resolves every event to noop. The comment on that row in `pipeline.go` is the contract: any stage can publish there.
The same `Stages` table registers a `submitqueue-hook` stage, outside the order of the flow above. It consumes hook events and invokes the wired hooks, and it publishes nothing onward. The gateway's request materializer publishes request events there (see the [gateway README](../../../submitqueue/gateway/README.md)); no orchestrator controller does, and `service/submitqueue/orchestrator/server` resolves every event to noop. The comment on that row in `pipeline.go` is the contract: any stage can publish there.

## Diagram

Expand Down Expand Up @@ -131,7 +131,7 @@ class node_storagecontract,node_queuestorage,node_requestlog,node_messagequeue t
class node_submitter toneIndigo
```

`hook` is drawn with the other stages because the topology registers it. Nothing in the orchestrator publishes to it, so the diagram has no edge into that node.
`hook` is drawn with the other stages because the topology registers it. Only the gateway's request materializer publishes to it, so the diagram has no edge into that node from an orchestrator stage.

## Per-controller summary

Expand Down
4 changes: 4 additions & 0 deletions service/submitqueue/gateway/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ go_library(
importpath = "github.com/uber/submitqueue/service/submitqueue/gateway/server",
visibility = ["//visibility:private"],
deps = [
"//api/base/hook:go_default_library",
"//api/submitqueue/gateway/protopb:go_default_library",
"//platform/consumer:go_default_library",
"//platform/errs:go_default_library",
Expand Down Expand Up @@ -93,6 +94,7 @@ go_test(
],
embed = [":gateway_lib"], # keep
deps = [
"//api/base/hook:go_default_library",
"//api/submitqueue/gateway/protopb:go_default_library",
"//platform/base/failure:go_default_library",
"//platform/base/messagequeue:go_default_library",
Expand All @@ -103,12 +105,14 @@ go_test(
"//platform/extension/messagequeue/mock:go_default_library",
"//platform/lifecycle:go_default_library",
"//submitqueue/core/messagequeue:go_default_library",
"//submitqueue/core/topickey:go_default_library",
"//submitqueue/entity:go_default_library",
"//submitqueue/extension/queueconfig/mock:go_default_library",
"//submitqueue/gateway/controller:go_default_library",
"//submitqueue/gateway/core/request:go_default_library",
"//submitqueue/gateway/extension/storage:go_default_library",
"//submitqueue/gateway/extension/storage/mock:go_default_library",
"//submitqueue/orchestrator:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
"@com_github_stretchr_testify//require:go_default_library",
"@com_github_uber_go_tally//:go_default_library",
Expand Down
3 changes: 3 additions & 0 deletions service/submitqueue/gateway/server/log_consumer.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
"time"

"github.com/uber-go/tally"
basehook "github.com/uber/submitqueue/api/base/hook"
"github.com/uber/submitqueue/platform/consumer"
"github.com/uber/submitqueue/platform/errs"
genericerrs "github.com/uber/submitqueue/platform/errs/generic"
Expand Down Expand Up @@ -47,6 +48,8 @@ func newGatewayTopicRegistry(queue extqueue.Queue, subscriberName string) (consu
{Key: topickey.TopicKeyStart, Name: "start", Queue: queue},
{Key: topickey.TopicKeyCancel, Name: "cancel", Queue: queue},
{Key: topickey.TopicKeyLog, Name: "log", Queue: queue, Subscription: primary},
// Publish-only: the orchestrator pipeline's hook stage consumes it under this name.
{Key: basehook.TopicKeyHook, Name: "submitqueue-hook", Queue: queue},
{
Key: logDLQTopicKey, Name: "log" + logDLQSuffix, Queue: queue,
Subscription: extqueue.DLQSubscriptionConfig(subscriberName, logDLQConsumerGroup),
Expand Down
22 changes: 22 additions & 0 deletions service/submitqueue/gateway/server/log_consumer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uber-go/tally"
basehook "github.com/uber/submitqueue/api/base/hook"
"github.com/uber/submitqueue/platform/base/failure"
entityqueue "github.com/uber/submitqueue/platform/base/messagequeue"
"github.com/uber/submitqueue/platform/consumer"
Expand All @@ -30,10 +31,12 @@ import (
queuemock "github.com/uber/submitqueue/platform/extension/messagequeue/mock"
"github.com/uber/submitqueue/platform/lifecycle"
sqmq "github.com/uber/submitqueue/submitqueue/core/messagequeue"
"github.com/uber/submitqueue/submitqueue/core/topickey"
"github.com/uber/submitqueue/submitqueue/entity"
requestcore "github.com/uber/submitqueue/submitqueue/gateway/core/request"
gwstorage "github.com/uber/submitqueue/submitqueue/gateway/extension/storage"
gwstoragemock "github.com/uber/submitqueue/submitqueue/gateway/extension/storage/mock"
"github.com/uber/submitqueue/submitqueue/orchestrator"
"go.uber.org/mock/gomock"
"go.uber.org/zap"
"google.golang.org/grpc"
Expand Down Expand Up @@ -112,6 +115,25 @@ func TestLogConsumers_RetainFailedRecovery(t *testing.T) {
}
}

func TestGatewayTopicRegistry_MatchesOrchestratorStages(t *testing.T) {
registry, err := newGatewayTopicRegistry(queuemock.NewMockQueue(gomock.NewController(t)), "worker")
require.NoError(t, err)
stageNames := map[consumer.TopicKey]string{}
for _, stage := range orchestrator.Stages {
stageNames[stage.Key] = stage.Name
}

for _, key := range []consumer.TopicKey{topickey.TopicKeyStart, topickey.TopicKeyCancel, basehook.TopicKeyHook} {
t.Run(string(key), func(t *testing.T) {
want, ok := stageNames[key]
require.True(t, ok, "the orchestrator consumes every topic the gateway publishes to")
got, ok := registry.TopicName(key)
require.True(t, ok)
assert.Equal(t, want, got)
})
}
}

type lifecycleTestConsumer struct {
consumer.Consumer
name string
Expand Down
4 changes: 2 additions & 2 deletions service/submitqueue/gateway/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ func run() error {
// Land creates receipt projections, request-summary and List controllers
// read materialized views, and request-history controllers read retained
// logs. Normal log-topic persistence and terminal DLQ repair both use the
// materializer.
// materializer, which also publishes request hook events.
store, err := mysqlstorage.NewStorage(appDB, scope.SubScope("storage"))
if err != nil {
return fmt.Errorf("failed to create storage: %w", err)
Expand All @@ -343,7 +343,7 @@ func run() error {
// Create controllers and wrap them for gRPC. Every store is queue-scoped and
// resolves through the factory adapter; land/cancel/log share one materializer.
storageFty := storageFactory{backend: store}
materializer := requestcore.NewMaterializer(storageFty, requestcore.WithMetricsScope(scope))
materializer := requestcore.NewMaterializer(storageFty, requestcore.WithMetricsScope(scope), requestcore.WithHookPublishing(registry))
pingController := controller.NewPingController(logger, scope)
landController := controller.NewLandController(logger.Sugar(), scope, cnt, storageFty, materializer, queueConfigs, queuePauses, registry)
cancelController := controller.NewCancelController(logger.Sugar(), scope, storageFty, materializer, registry)
Expand Down
4 changes: 4 additions & 0 deletions submitqueue/gateway/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ The gateway owns the request log read model and is the only service that reads i

The materializer appends every audit event, selects the current authoritative winner, and repairs the queue projection. The normal orchestrator pipeline does not read or write the request-log store directly.

## Request hook events

The materializer publishes SubmitQueue's request hook events (`submitqueue/core/hookevent`) to the `submitqueue-hook` topic, which the orchestrator pipeline's hook stage consumes, one event per change of the request. The update that takes the summary out of `accepting` announces `request.received`; that happens inside `Land`, so a failed receipt publish is only counted on `request_hook.publish_errors`. Every delivery of a log that restates a `landed` or `error` summary announces `request.landed` or `request.failed`, so a failed publish is returned and the log's redelivery repairs it; a copy that arrives after the queue has forgotten the event id reaches hooks twice. Events therefore follow the summary users see, including a request whose start failed before the orchestrator saved it, and a stale or unfinished log publishes nothing.

## Final log recovery

The primary `gateway-log` consumer sends failed deliveries to `log_dlq` after a non-retryable error or exhaustion of its finite retry budget. The gateway's `gateway-log-dlq` consumer replays retained entries through the same log controller and materializer, with unlimited attempts, capped backoff, and no second-level DLQ. Valid log deliveries are acknowledged only after history, summary, URI mappings, and receipt projection succeed. Replays may append equivalent history rows; terminal-summary precedence and mapping repair are unchanged.
Expand Down
Loading