Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/rfc/hook-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Fire-and-forget side effects for pipeline lifecycle events: one shared event con

## Status

Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns.
Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage. The SubmitQueue gateway's request materializer publishes `request.received`, `request.landed`, and `request.failed` (`submitqueue/core/hookevent`) to it; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns.

## Problem

Expand Down Expand Up @@ -52,6 +52,7 @@ Delivery promise:
- Partition key = subject id; per-subject order only.
- Replay finding the target state already written → republish (idempotent). Beyond it → superseded; the event may be lost. Named, accepted gap.
- RPC-caused transitions are at-most-once; needing the guarantee means publishing from the first queue-driven stage.
- SubmitQueue request events follow the gateway's request summary, the status users see, rather than pipeline transitions, so a hook agrees with the UI even for a request whose start failed before the orchestrator saved it. Each status announces one event per change, so a hook retrying one change's failure never repeats another's. A finish is announced on every delivery of a log that restates it, which keeps the replay rule above: a failed publish fails the delivery and its redelivery republishes. `request.received` is published inside `Land`, which makes it RPC-caused and at-most-once; a later status standing in for a lost `accepted` log announces it instead. The events cross from the gateway to the orchestrator's hook stage, so their payload carries the change's URI, which never changes.
- Opt-in per host via topic-key registration; a registered host never skips, so off and loss are distinguishable.

### Contract
Expand Down
4 changes: 2 additions & 2 deletions doc/rfc/submitqueue/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Two cycles re-enter speculation. `speculate` dispatches funded paths to `build`;

Terminal request states are not `conclude`'s alone. `cancel` completes a request that has not been enrolled in a batch. `landconflictsignal` fails a request Runway reports as conflicted. `conclude` maps a terminal batch onto its member requests. The DLQ reconcilers force a failed terminal state when a consumed stage cannot finish.

The same `Stages` table registers a `submitqueue-hook` stage, outside the order of the flow above. It consumes hook events and invokes the wired hooks, and it publishes nothing onward. No orchestrator controller publishes a hook event; `service/submitqueue/orchestrator/server` resolves every event to noop. The comment on that row in `pipeline.go` is the contract: any stage can publish there.
The same `Stages` table registers a `submitqueue-hook` stage, outside the order of the flow above. It consumes hook events and invokes the wired hooks, and it publishes nothing onward. The gateway's request materializer publishes request events there (see the [gateway README](../../../submitqueue/gateway/README.md)); no orchestrator controller does, and `service/submitqueue/orchestrator/server` resolves every event to noop. The comment on that row in `pipeline.go` is the contract: any stage can publish there.

## Diagram

Expand Down Expand Up @@ -131,7 +131,7 @@ class node_storagecontract,node_queuestorage,node_requestlog,node_messagequeue t
class node_submitter toneIndigo
```

`hook` is drawn with the other stages because the topology registers it. Nothing in the orchestrator publishes to it, so the diagram has no edge into that node.
`hook` is drawn with the other stages because the topology registers it. Only the gateway's request materializer publishes to it, so the diagram has no edge into that node from an orchestrator stage.

## Per-controller summary

Expand Down
4 changes: 4 additions & 0 deletions service/submitqueue/gateway/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ go_library(
importpath = "github.com/uber/submitqueue/service/submitqueue/gateway/server",
visibility = ["//visibility:private"],
deps = [
"//api/base/hook:go_default_library",
"//api/submitqueue/gateway/protopb:go_default_library",
"//platform/consumer:go_default_library",
"//platform/errs:go_default_library",
Expand Down Expand Up @@ -93,6 +94,7 @@ go_test(
],
embed = [":gateway_lib"], # keep
deps = [
"//api/base/hook:go_default_library",
"//api/submitqueue/gateway/protopb:go_default_library",
"//platform/base/failure:go_default_library",
"//platform/base/messagequeue:go_default_library",
Expand All @@ -103,12 +105,14 @@ go_test(
"//platform/extension/messagequeue/mock:go_default_library",
"//platform/lifecycle:go_default_library",
"//submitqueue/core/messagequeue:go_default_library",
"//submitqueue/core/topickey:go_default_library",
"//submitqueue/entity:go_default_library",
"//submitqueue/extension/queueconfig/mock:go_default_library",
"//submitqueue/gateway/controller:go_default_library",
"//submitqueue/gateway/core/request:go_default_library",
"//submitqueue/gateway/extension/storage:go_default_library",
"//submitqueue/gateway/extension/storage/mock:go_default_library",
"//submitqueue/orchestrator:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
"@com_github_stretchr_testify//require:go_default_library",
"@com_github_uber_go_tally//:go_default_library",
Expand Down
3 changes: 3 additions & 0 deletions service/submitqueue/gateway/server/log_consumer.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
"time"

"github.com/uber-go/tally"
basehook "github.com/uber/submitqueue/api/base/hook"
"github.com/uber/submitqueue/platform/consumer"
"github.com/uber/submitqueue/platform/errs"
genericerrs "github.com/uber/submitqueue/platform/errs/generic"
Expand Down Expand Up @@ -47,6 +48,8 @@ func newGatewayTopicRegistry(queue extqueue.Queue, subscriberName string) (consu
{Key: topickey.TopicKeyStart, Name: "start", Queue: queue},
{Key: topickey.TopicKeyCancel, Name: "cancel", Queue: queue},
{Key: topickey.TopicKeyLog, Name: "log", Queue: queue, Subscription: primary},
// Publish-only: the orchestrator pipeline's hook stage consumes it under this name.
{Key: basehook.TopicKeyHook, Name: "submitqueue-hook", Queue: queue},
{
Key: logDLQTopicKey, Name: "log" + logDLQSuffix, Queue: queue,
Subscription: extqueue.DLQSubscriptionConfig(subscriberName, logDLQConsumerGroup),
Expand Down
22 changes: 22 additions & 0 deletions service/submitqueue/gateway/server/log_consumer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uber-go/tally"
basehook "github.com/uber/submitqueue/api/base/hook"
"github.com/uber/submitqueue/platform/base/failure"
entityqueue "github.com/uber/submitqueue/platform/base/messagequeue"
"github.com/uber/submitqueue/platform/consumer"
Expand All @@ -30,10 +31,12 @@ import (
queuemock "github.com/uber/submitqueue/platform/extension/messagequeue/mock"
"github.com/uber/submitqueue/platform/lifecycle"
sqmq "github.com/uber/submitqueue/submitqueue/core/messagequeue"
"github.com/uber/submitqueue/submitqueue/core/topickey"
"github.com/uber/submitqueue/submitqueue/entity"
requestcore "github.com/uber/submitqueue/submitqueue/gateway/core/request"
gwstorage "github.com/uber/submitqueue/submitqueue/gateway/extension/storage"
gwstoragemock "github.com/uber/submitqueue/submitqueue/gateway/extension/storage/mock"
"github.com/uber/submitqueue/submitqueue/orchestrator"
"go.uber.org/mock/gomock"
"go.uber.org/zap"
"google.golang.org/grpc"
Expand Down Expand Up @@ -112,6 +115,25 @@ func TestLogConsumers_RetainFailedRecovery(t *testing.T) {
}
}

func TestGatewayTopicRegistry_MatchesOrchestratorStages(t *testing.T) {
registry, err := newGatewayTopicRegistry(queuemock.NewMockQueue(gomock.NewController(t)), "worker")
require.NoError(t, err)
stageNames := map[consumer.TopicKey]string{}
for _, stage := range orchestrator.Stages {
stageNames[stage.Key] = stage.Name
}

for _, key := range []consumer.TopicKey{topickey.TopicKeyStart, topickey.TopicKeyCancel, basehook.TopicKeyHook} {
t.Run(string(key), func(t *testing.T) {
want, ok := stageNames[key]
require.True(t, ok, "the orchestrator consumes every topic the gateway publishes to")
got, ok := registry.TopicName(key)
require.True(t, ok)
assert.Equal(t, want, got)
})
}
}

type lifecycleTestConsumer struct {
consumer.Consumer
name string
Expand Down
4 changes: 2 additions & 2 deletions service/submitqueue/gateway/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ func run() error {
// Land creates receipt projections, request-summary and List controllers
// read materialized views, and request-history controllers read retained
// logs. Normal log-topic persistence and terminal DLQ repair both use the
// materializer.
// materializer, which also publishes request hook events.
store, err := mysqlstorage.NewStorage(appDB, scope.SubScope("storage"))
if err != nil {
return fmt.Errorf("failed to create storage: %w", err)
Expand All @@ -343,7 +343,7 @@ func run() error {
// Create controllers and wrap them for gRPC. Every store is queue-scoped and
// resolves through the factory adapter; land/cancel/log share one materializer.
storageFty := storageFactory{backend: store}
materializer := requestcore.NewMaterializer(storageFty, requestcore.WithMetricsScope(scope))
materializer := requestcore.NewMaterializer(storageFty, requestcore.WithMetricsScope(scope), requestcore.WithHookPublishing(registry))
pingController := controller.NewPingController(logger, scope)
landController := controller.NewLandController(logger.Sugar(), scope, cnt, storageFty, materializer, queueConfigs, queuePauses, registry)
cancelController := controller.NewCancelController(logger.Sugar(), scope, storageFty, materializer, registry)
Expand Down
25 changes: 25 additions & 0 deletions submitqueue/core/hookevent/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
name = "go_default_library",
srcs = ["hookevent.go"],
importpath = "github.com/uber/submitqueue/submitqueue/core/hookevent",
visibility = ["//visibility:public"],
deps = [
"//api/base/hook:go_default_library",
"//submitqueue/entity:go_default_library",
"@org_golang_google_protobuf//types/known/structpb:go_default_library",
],
)

go_test(
name = "go_default_test",
srcs = ["hookevent_test.go"],
embed = [":go_default_library"],
deps = [
"//api/base/hook:go_default_library",
"//submitqueue/entity:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
"@com_github_stretchr_testify//require:go_default_library",
],
)
126 changes: 126 additions & 0 deletions submitqueue/core/hookevent/hookevent.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
// Copyright (c) 2026 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package hookevent holds SubmitQueue's hook events: the source it reports, the
// types it publishes, and a constructor per type. Centralizing them keeps the
// wire contract in one place, so a publisher names an event rather than
// assembling one.
//
// The events follow a request's public status, the one its summary shows: one
// when the request is received, and one when it lands or fails. Cancellation
// has no event yet because no hook needs one. Deriving them from the summary
// rather than from pipeline transitions keeps every hook in step with what a
// user sees, including a request whose start failed before the pipeline saved
// it.
//
// Each status announces one event per change of the request, so a hook that
// retries one change's failed side effect never repeats the others'. Payloads
// carry the queue, the request id, and that change's URI. The gateway
// publishes these events and the orchestrator's hook stage consumes them, and a
// hook there cannot read the gateway's store, so the payload carries what a
// hook needs to reach the change. A request's URIs never change, so they cannot
// go stale on redelivery. See doc/rfc/hook-framework.md.
//
// Known gaps, accepted for now:
// - request.received is published inside Land, an RPC. If Land records
// accepted but the publish fails, no later status announces the receipt.
// - A finished status is announced again whenever its log is redelivered or
// resent, so a failed publish repairs itself; a copy that arrives after the
// queue forgot the id reaches hooks twice.
// - Order holds within a request's partition, but a receipt waiting to retry
// does not hold back the finish, so a hook can see the finish first.
// - The queue drops a repeated event id only until the first copy is consumed
// and collected, seconds later.
package hookevent

import (
"crypto/sha256"
"fmt"

basehook "github.com/uber/submitqueue/api/base/hook"
"github.com/uber/submitqueue/submitqueue/entity"
"google.golang.org/protobuf/types/known/structpb"
)

// Source is reported on every hook event SubmitQueue publishes. A consumer
// serving several domains matches on it to recognize SubmitQueue's events.
const Source = "submitqueue"

// Type is the event type a consumer filters on.
type Type string

const (
// TypeRequestReceived announces that the request named in the payload has
// left the internal accepting status and is visible to users. It carries no
// verdict: the request has not been validated yet.
TypeRequestReceived Type = "request.received"
// TypeRequestLanded announces that the request named in the payload landed.
TypeRequestLanded Type = "request.landed"
// TypeRequestFailed announces that the request named in the payload ended in
// error. The request's summary carries the reason.
TypeRequestFailed Type = "request.failed"
)

// finishedTypeByStatus maps each finished status that has an event to its type.
var finishedTypeByStatus = map[entity.RequestStatus]Type{
entity.RequestStatusLanded: TypeRequestLanded,
entity.RequestStatusError: TypeRequestFailed,
}

// NewRequestReceived builds the events announcing that summary's request was
// received, one per change. A request is received once, so the ids carry version
// 0 and every republish for the same request mints the same ids.
func NewRequestReceived(summary entity.RequestSummary) []*basehook.HookEvent {
return newRequestEvents(TypeRequestReceived, summary, 0, summary.ReceivedAtMs)
}

// NewRequestFinished builds the events announcing summary's finished status, one
// per change, and none for a status that has no event. The ids carry
// summary.RequestVersion, so a status the summary restates mints the same ids and
// a later finished status the summary moves to mints new ones.
func NewRequestFinished(summary entity.RequestSummary) []*basehook.HookEvent {
eventType, ok := finishedTypeByStatus[summary.Status]
if !ok {
return nil
}
return newRequestEvents(eventType, summary, summary.RequestVersion, summary.StatusTimestampMs)
}

func newRequestEvents(eventType Type, summary entity.RequestSummary, version int32, timestampMs int64) []*basehook.HookEvent {
events := make([]*basehook.HookEvent, 0, len(summary.ChangeURIs))
for _, uri := range summary.ChangeURIs {
events = append(events, &basehook.HookEvent{
Id: basehook.NewEventID(Source, string(eventType), changeSubject(summary, uri), version),
Source: Source,
Type: string(eventType),
TimestampMs: timestampMs,
Version: version,
Payload: &structpb.Struct{Fields: map[string]*structpb.Value{
"queue": structpb.NewStringValue(summary.Queue),
"request_id": structpb.NewStringValue(summary.RequestID),
"change_uri": structpb.NewStringValue(uri),
}},
})
}
return events
}

// changeSubject identifies one change of a request across queues. Request IDs
// are only unique within a queue, and queue names and URIs contain the id
// separator, so every part is length-prefixed and hashed into one separator-free
// component.
func changeSubject(summary entity.RequestSummary, uri string) string {
identity := fmt.Sprintf("%d:%s%d:%s%d:%s", len(summary.Queue), summary.Queue, len(summary.RequestID), summary.RequestID, len(uri), uri)
return fmt.Sprintf("%x", sha256.Sum256([]byte(identity)))
}
Loading
Loading