Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/rfc/stovepipe/steps/buildanalysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ Status-polling DLQ recovery does not publish a build-completion notice. The anal

## Analysis errors and dead letters

Analysis errors are returned to the normal consumer whether the Build is running or terminal; errors do not use the two-minute polling hold. The reference service uses backend error classification and the extended retry configuration: a maximum of 10 attempts, with retry delays of 5, 10, 20, 40, then 60 seconds, capped at 60 seconds. A successful hold resets failure accounting. Non-retryable errors and exhausted retries send the message to the analysis DLQ, so final analysis is attempted rather than guaranteed to succeed.
Analysis errors are returned to the normal consumer whether the Build is running or terminal; errors do not use the configured polling hold. The reference service uses backend error classification and the extended retry configuration: a maximum of 10 attempts, with retry delays of 5, 10, 20, 40, then 60 seconds, capped at 60 seconds. A successful hold resets failure accounting. Non-retryable errors and exhausted retries send the message to the analysis DLQ, so final analysis is attempted rather than guaranteed to succeed.

The analysis DLQ uses the existing DLQ consumer, with its always-retryable processor and unlimited-retry subscription. The recovery controller records incomplete analysis in Request history without rerunning detection or changing the validation outcome. Only state reads and history persistence retry; unusable messages are logged and acknowledged. See [DLQ recovery](../../../../stovepipe/controller/buildanalysis/README.md#dlq-recovery) for the history identity and error-handling rules.
6 changes: 3 additions & 3 deletions doc/rfc/stovepipe/steps/buildsignal.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,10 @@ Two-tier cadence, matching SubmitQueue:

| Status | Delay | Rationale |
|---|---|---|
| `accepted` | `PollDelayAcceptedMs` (default 5000ms) | Queued by the runner, not started — poll infrequently. |
| `running` | `PollDelayRunningMs` (default 2000ms) | Actively executing — poll more often. |
| `accepted` | `BuildSignal.AcceptedPollDelayMs` (default 5000ms) | Queued by the runner, not started — poll infrequently. |
| `running` | `BuildSignal.RunningPollDelayMs` (default 2000ms) | Actively executing — poll more often. |

Package-level `var`s (not `const`s) so tests can shorten them; the server always uses the defaults. Future: move these behind a `queueconfig`-style extension so operators can tune cadence per queue without a code change (the same TODO SubmitQueue's buildsignal carries).
Each delivery resolves these settings through the queue configuration store. Hosts apply a service baseline, deployment-wide settings, and per-queue overrides before supplying immutable configurations. The reference server embeds its baseline YAML and layers an optional configuration file over it. See the [configuration guide](../../../../service/stovepipe/server/README.md).

## Error classification

Expand Down
2 changes: 2 additions & 0 deletions service/stovepipe/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -104,11 +104,13 @@ go_test(
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildissue/noop:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
"//stovepipe/extension/queueconfig/mock:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"//stovepipe/extension/storage/mock:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
"@com_github_stretchr_testify//require:go_default_library",
"@com_github_uber_go_tally//:go_default_library",
"@org_golang_google_protobuf//proto:go_default_library",
"@org_uber_go_mock//gomock:go_default_library",
"@org_uber_go_zap//zaptest:go_default_library",
],
Expand Down
2 changes: 0 additions & 2 deletions service/stovepipe/server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,3 @@ Deployment-wide stage settings from the optional file override the service basel
Every queue override must name a queue in `MQ_TENANTS`; queues without overrides inherit the deployment configuration. Queue registration is separate from stage settings. The static store only serves registered queues.

The baseline retains a concurrency limit of one, a five-second admission wait, five-second accepted-build polling, two-second running-build polling, two-minute build analysis polling, and thirty-second issue refresh and recovery waits. These values are defined only in the baseline YAML.

Process-stage admission consumes this configuration. Other stage controllers retain their existing intervals.
3 changes: 3 additions & 0 deletions service/stovepipe/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,7 @@ func registerPrimaryControllers(
logger,
scope,
store,
configs,
materializer,
brf,
registry,
Expand All @@ -474,6 +475,7 @@ func registerPrimaryControllers(
logger,
scope,
store,
configs,
materializer,
detectorFactory,
registry,
Expand All @@ -489,6 +491,7 @@ func registerPrimaryControllers(
logger,
scope,
store,
configs,
materializer,
registry,
stovepipemq.TopicKeyIssueTrack,
Expand Down
58 changes: 58 additions & 0 deletions service/stovepipe/server/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,12 @@ import (
"github.com/uber/submitqueue/stovepipe/entity"
buildissuenoop "github.com/uber/submitqueue/stovepipe/extension/buildissue/noop"
"github.com/uber/submitqueue/stovepipe/extension/queueconfig"
queueconfigmock "github.com/uber/submitqueue/stovepipe/extension/queueconfig/mock"
"github.com/uber/submitqueue/stovepipe/extension/storage"
storagemock "github.com/uber/submitqueue/stovepipe/extension/storage/mock"
"go.uber.org/mock/gomock"
"go.uber.org/zap/zaptest"
"google.golang.org/protobuf/proto"
)

// recordingConsumer captures what the host registers instead of subscribing.
Expand Down Expand Up @@ -393,3 +395,59 @@ func TestBuildAnalysisRegistration(t *testing.T) {
assert.True(t, cfg.DLQ.Enabled)
assert.Equal(t, "_dlq", cfg.DLQ.TopicSuffix)
}

func TestStageControllersUseHostConfiguration(t *testing.T) {
const queue = "configured-queue"

for _, tt := range []struct {
name string
topic consumer.TopicKey
notification proto.Message
partition string
}{
{
"build signal",
stovepipemq.TopicKeyBuildSignal,
&stovepipemq.BuildSignal{Id: "build", QueueName: queue},
"build",
},
{
"build analysis",
stovepipemq.TopicKeyBuildAnalysis,
&stovepipemq.BuildAnalysis{Id: "build", QueueName: queue},
"build",
},
{
"issue tracking",
stovepipemq.TopicKeyIssueTrack,
&stovepipemq.IssueTrack{RequestId: "1", QueueName: queue},
queue,
},
} {
t.Run(tt.name, func(t *testing.T) {
ctrl := gomock.NewController(t)
configs := queueconfigmock.NewMockStore(ctrl)
configs.EXPECT().Get(gomock.Any(), queue).Return(entity.QueueConfig{}, assert.AnError)

_, controllers, _ := registeredControllersWithConfig(t, configs)

payload, err := stovepipemq.Marshal(tt.notification)
require.NoError(t, err)

message := entityqueue.NewMessage("notification", payload, tt.partition, nil)
message.Tenant = queue

delivery := consumermock.NewMockDelivery(ctrl)
delivery.EXPECT().Message().Return(message)

for _, controller := range controllers {
if controller.TopicKey() == tt.topic {
require.ErrorIs(t, controller.Process(context.Background(), delivery), assert.AnError)
return
}
}

require.Fail(t, "stage controller was not registered")
})
}
}
1 change: 1 addition & 0 deletions stovepipe/controller/build/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ go_test(
"//stovepipe/extension/buildissue/mock:go_default_library",
"//stovepipe/extension/buildrunner:go_default_library",
"//stovepipe/extension/buildrunner/mock:go_default_library",
"//stovepipe/extension/queueconfig/mock:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"//stovepipe/extension/storage/mock:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
Expand Down
37 changes: 30 additions & 7 deletions stovepipe/controller/build/build_analysis_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import (
"github.com/uber/submitqueue/stovepipe/extension/buildissue"
buildissuemock "github.com/uber/submitqueue/stovepipe/extension/buildissue/mock"
"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
queueconfigmock "github.com/uber/submitqueue/stovepipe/extension/queueconfig/mock"
"github.com/uber/submitqueue/stovepipe/extension/storage"
storagemock "github.com/uber/submitqueue/stovepipe/extension/storage/mock"
"go.uber.org/mock/gomock"
Expand Down Expand Up @@ -142,11 +143,33 @@ func TestBuildAnalysisPipeline(t *testing.T) {
detector.EXPECT().Detect(gomock.Any(), entity.BuildID{ID: testBuildID}).Return([]entity.BuildIssue{a, b}, nil),
)

analysisController := buildanalysis.NewController(zap.NewNop().Sugar(), tally.NoopScope,
staticStorageFactory{store: m.store}, requestlog.NewMaterializer(tally.NoopScope), detectorFactory, buildController.registry, stovepipemq.TopicKeyBuildAnalysis, "analysis")
statusController := buildsignal.NewController(zap.NewNop().Sugar(), tally.NoopScope,
staticStorageFactory{store: m.store}, m.materializer, m.runnerFactory, buildController.registry,
stovepipemq.TopicKeyBuildSignal, "signal")
configs := queueconfigmock.NewMockStore(ctrl)
cfg := entity.QueueConfig{Name: testQueue}
cfg.Stages.BuildAnalysis.PollDelayMs = 137
configs.EXPECT().Get(gomock.Any(), testQueue).Return(cfg, nil).AnyTimes()

analysisController := buildanalysis.NewController(
zap.NewNop().Sugar(),
tally.NoopScope,
staticStorageFactory{store: m.store},
configs,
requestlog.NewMaterializer(tally.NoopScope),
detectorFactory,
buildController.registry,
stovepipemq.TopicKeyBuildAnalysis,
"analysis",
)
statusController := buildsignal.NewController(
zap.NewNop().Sugar(),
tally.NoopScope,
staticStorageFactory{store: m.store},
configs,
m.materializer,
m.runnerFactory,
buildController.registry,
stovepipemq.TopicKeyBuildSignal,
"signal",
)

delivered := func(msg entityqueue.Message) *consumermock.MockDelivery {
d := consumermock.NewMockDelivery(ctrl)
Expand All @@ -159,13 +182,13 @@ func TestBuildAnalysisPipeline(t *testing.T) {
require.Len(t, published["buildsignal"], 1)

firstPoll := delivered(published["buildanalysis"][0])
firstPoll.EXPECT().Hold(buildanalysis.PollDelayMs)
firstPoll.EXPECT().Hold(cfg.Stages.BuildAnalysis.PollDelayMs)
require.NoError(t, analysisController.Process(ctx, firstPoll))
assert.True(t, persisted[a])
assert.False(t, persisted[b])

secondPoll := delivered(published["buildanalysis"][0])
secondPoll.EXPECT().Hold(buildanalysis.PollDelayMs)
secondPoll.EXPECT().Hold(cfg.Stages.BuildAnalysis.PollDelayMs)
require.NoError(t, analysisController.Process(ctx, secondPoll))
assert.True(t, persisted[b])

Expand Down
2 changes: 2 additions & 0 deletions stovepipe/controller/buildanalysis/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ go_library(
"//stovepipe/core/requestlog:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildissue:go_default_library",
"//stovepipe/extension/queueconfig:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"@com_github_uber_go_tally//:go_default_library",
"@org_uber_go_zap//:go_default_library",
Expand Down Expand Up @@ -46,6 +47,7 @@ go_test(
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildissue:go_default_library",
"//stovepipe/extension/buildissue/mock:go_default_library",
"//stovepipe/extension/queueconfig/mock:go_default_library",
"//stovepipe/extension/storage:go_default_library",
"//stovepipe/extension/storage/mock:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
Expand Down
4 changes: 2 additions & 2 deletions stovepipe/controller/buildanalysis/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ All analysis deliveries for a build share its build-ID partition and consumer gr

## Polling and completion

Every delivery reloads the Build and its Request before detection. After a successful pass, a nonterminal Build holds the delivery for two minutes; a terminal Build completes after persistence and completion history once its Request outcome is known, or the Request has ended without one. A pass that begins with a nonterminal Build always holds, even if the Build finishes during detection, so another delivery analyzes it after observing terminal status. More than one delivery may analyze the same terminal Build; build-issue persistence remains idempotent.
Every delivery reloads the Build and its Request before detection. After a successful pass, a nonterminal Build holds the delivery for the configured polling interval (two minutes by default); a terminal Build completes after persistence and completion history once its Request outcome is known, or the Request has ended without one. A pass that begins with a nonterminal Build always holds, even if the Build finishes during detection, so another delivery analyzes it after observing terminal status. More than one delivery may analyze the same terminal Build; build-issue persistence remains idempotent.

If the Request is terminal but the loaded Build is nonterminal, analysis reloads the Build before deciding to stop. The Build may have finished between the initial Build read and the Request read; reloading avoids skipping its final analysis. If it is still nonterminal, analysis retains abandonment and stops without detection or a hold. This can occur when the status-polling DLQ handler fails the Request without learning the actual build result, leaving the Build at its last observed status.

Status polling and updates remain the responsibility of buildsignal. Analysis never changes Request state or releases build slots.

## Errors

Detection, persistence, configuration, state-loading, and message-validation errors are returned to the consumer for its normal retry or DLQ policy, whether the Build is running or terminal. Analysis errors do not use the two-minute polling hold. The consumer classifies errors and applies its configured retry limits and backoff; a final analysis can still fail and dead-letter.
Detection, persistence, configuration, state-loading, and message-validation errors are returned to the consumer for its normal retry or DLQ policy, whether the Build is running or terminal. Analysis errors do not use the configured polling hold. The consumer classifies errors and applies its configured retry limits and backoff; a final analysis can still fail and dead-letter.

## DLQ recovery

Expand Down
15 changes: 10 additions & 5 deletions stovepipe/controller/buildanalysis/buildanalysis.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,15 @@ import (
"github.com/uber/submitqueue/stovepipe/core/requestlog"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/buildissue"
"github.com/uber/submitqueue/stovepipe/extension/queueconfig"
"github.com/uber/submitqueue/stovepipe/extension/storage"
"go.uber.org/zap"
)

// PollDelayMs is the delay in milliseconds between successful polling passes.
// Two minutes limits provider traffic during long-running builds.
const PollDelayMs int64 = 120000

// Controller accumulates build issues while builds run and after completion.
// Terminal analysis must persist all build issues and its completion event before acknowledging.
type Controller struct {
queueConfigs queueconfig.Store
logger *zap.SugaredLogger
metricsScope tally.Scope
stores storage.Factory
Expand All @@ -57,13 +55,15 @@ func NewController(
logger *zap.SugaredLogger,
scope tally.Scope,
stores storage.Factory,
queueConfigs queueconfig.Store,
materializer requestlog.Materializer,
detectorFactory buildissue.DetectorFactory,
registry consumer.TopicRegistry,
topicKey consumer.TopicKey,
consumerGroup string,
) *Controller {
return &Controller{
queueConfigs: queueConfigs,
logger: logger.Named("buildanalysis_controller"),
metricsScope: scope.SubScope("buildanalysis_controller"),
stores: stores,
Expand Down Expand Up @@ -98,6 +98,11 @@ func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) (r
return fmt.Errorf("build analysis must be partitioned by build %q", analysis.GetId())
}

cfg, err := c.queueConfigs.Get(ctx, analysis.GetQueueName())
if err != nil {
return fmt.Errorf("load queue configuration: %w", err)
}

store, err := c.stores.For(storage.Config{QueueName: analysis.GetQueueName()})
if err != nil {
return fmt.Errorf("failed to resolve storage for queue %q: %w", analysis.GetQueueName(), err)
Expand Down Expand Up @@ -207,7 +212,7 @@ func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) (r
}

if hold {
delivery.Hold(PollDelayMs)
delivery.Hold(cfg.Stages.BuildAnalysis.PollDelayMs)
}

c.logger.Debugw("build analysis pass complete",
Expand Down
Loading
Loading