uber · behinddwalls · Jul 24, 2026
@@ -1,64 +1,17 @@
-# Scorer
+# scorer
 
-Vendor-agnostic interface for computing success probability scores for code changes.
+A `Scorer` returns the probability that a batch's build succeeds — a number between 0.0 and 1.0. It is handed the batch identity and resolves the batch's changes itself through an injected `changeset.Resolver`, so callers pass an `entity.Batch` and nothing more.
 
-## Interface
+Callers may score every batch a queue is waiting on, so implementations should be cheap. A speculation run scores each batch at most once, but it does not carry results across runs; anything expensive belongs behind the implementation's own cache.
 
-### Scorer
-
-Computes a success probability for a given change.
-
-```go
-type Scorer interface {
-    Score(ctx context.Context, change entity.Change) (float64, error)
-}
-```
-
-- **change**: A `entity.Change` identifying the code change to score.
-- **Score**: Returns a probability between 0.0 and 1.0 indicating the likelihood of a successful land. Returns an error if scoring fails.
+Like the other extensions, a `Scorer` is selected **per queue** by the wiring layer through the `Config` (queue name) and `Factory` interface.
 
 ## Implementations
 
-### Heuristic
-
-Scores a change by extracting a numeric value via a `ValueFunc` and matching it against ordered buckets. Each bucket maps a `[Min, Max]` range to a probability.
-
-```go
-s := heuristic.New(
-    []heuristic.Bucket{
-        {Min: 0, Max: 5, Score: 0.95},
-        {Min: 6, Max: 20, Score: 0.75},
-        {Min: 21, Max: 100, Score: 0.5},
-    },
-    func(ctx context.Context, change entity.Change) (int, error) {
-        // resolve the change into a numeric metric
-        return filesChanged, nil
-    },
-)
-
-score, err := s.Score(ctx, change)
-```
-
-### Composite
-
-Combines multiple named scorers into a single score using a reduce function. The reduce function receives a `map[string]float64` mapping scorer names to their scores, enabling domain-aware aggregation.
-
-Built-in reduce functions: `Min`, `Max`, `Avg`.
-
-```go
-s := composite.New(
-    map[string]scorer.Scorer{
-        "files": fileScorer,
-        "deps":  depScorer,
-    },
-    composite.Min,
-)
+**`heuristic`** scores a batch by extracting one number from its changes and matching that against ordered buckets, each mapping a `[Min, Max]` range to a probability. The extraction is a caller-supplied `ValueFunc` over the resolved `entity.BatchChanges`, so the same bucketing works for files touched, lines changed, or any other metric.
 
-score, err := s.Score(ctx, change)
-```
+**`composite`** runs several named scorers and reduces their scores to one. The reduce function receives the scores keyed by scorer name, so it can weigh sources differently rather than treating them as interchangeable; `Min`, `Max`, and `Avg` are provided.
 
-## Implementing a Backend
+## Adding a backend
 
-1. Create `extension/scorer/{backend}/` directory
-2. Implement the `Scorer` interface
-3. Accept `entity.Change` and resolve it into whatever data the implementation needs
+Create a package under `scorer/<backend>/` whose `New(...)` returns a `scorer.Scorer`, injecting whatever it needs at construction — a `changeset.Resolver` to reach the batch's changes, a metrics scope, any client. Do not add a `Config` or `Factory` implementation here; per-queue routing and the factory adapter live in the wiring layer.
@@ -22,11 +22,17 @@ import (
 	"github.com/uber/submitqueue/submitqueue/entity"
 )
 
-// Scorer computes a success probability score for a batch based on its changes.
+// Scorer computes the probability that a batch's build succeeds, based on its
+// changes.
 type Scorer interface {
-	// Score returns a probability between 0.0 and 1.0 indicating the likelihood
-	// of a successful land for the given batch. It is handed the batch identity
-	// and resolves the batch's changes itself through an injected changeset.Resolver.
+	// Score returns a probability between 0.0 and 1.0 that the given batch's
+	// build succeeds. It is handed the batch identity and resolves the batch's
+	// changes itself through an injected changeset.Resolver.
+	//
+	// Callers may score every batch a queue is waiting on, so implementations
+	// should be cheap: a speculation run scores each batch at most once, but it
+	// does not carry results over to the next run, so anything expensive to
+	// compute belongs behind the implementation's own cache.
 	Score(ctx context.Context, batch entity.Batch) (float64, error)
 }
 

@@ -0,0 +1,9 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["generator.go"],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator",
+    visibility = ["//visibility:public"],
+    deps = ["//submitqueue/entity:go_default_library"],
+)
@@ -0,0 +1,9 @@
+# generator
+
+The `generator` package is a piece the `standard` `Speculator` is built from: a `Generator` produces the queue's candidate paths as one stream, best first, across all heads. It is **not** controller-facing — the speculate controller only knows the `Speculator` contract, and a different `Speculator` need not split its work this way. So there is no `Config` or `Factory` here; a `Generator` is chosen when the `standard` `Speculator` is constructed.
+
+`Open` starts the stream over the queue's live batches and returns a `PathIterator`. The caller pulls one candidate at a time and the generator does only the work that answer needs. A cancelled or expired context ends the stream with its error.
+
+Candidates never repeat and never contradict a known fact. Beyond that, the order is the `Generator`'s own: it yields candidates in whatever ranking it implements, and each carries the score it ranked by — higher first, on a scale the generator defines. Consumers take the iterator in the order given and do not interpret the score. Scores mean something only within the run and are never stored.
+
+The generator offers every path in the space, including paths whose builds already ran. Suppressing finished paths is the `Allocator`'s job, since that is the piece reconciling candidates against the stored path sets.
@@ -0,0 +1,30 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = [
+        "bestfirst.go",
+        "head.go",
+        "iterator.go",
+    ],
+    importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/generator/bestfirst",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["bestfirst_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/scorer:go_default_library",
+        "//submitqueue/extension/speculation/generator:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+    ],
+)
@@ -0,0 +1,47 @@
+# bestfirst
+
+A head waiting on unfinished dependencies has one path per combination of outcomes — 2ⁿ of them — while callers want the best few. `bestfirst` hands them out in score order without building the rest.
+
+Dependencies that already resolved carry a forced assumption: succeeded means the head builds on top of it, failed or cancelled means it builds without. They stay in the path but drop out of the search. Each unresolved one is a two-way assumption whose probability — that the dependency's build succeeds — comes from an injected `scorer.Scorer`, remembered per dependency so one shared by several heads is scored once. A dependency that is not a live batch cannot be scored at all, and gets a high default, since nearly every batch a queue accepts does build successfully. A path's score is the probability that all its assumptions hold.
+
+**The best path is not "assume everything succeeds."** Each dependency takes its *preferred* assumption — the one with the higher probability — so a dependency that will probably fail is assumed to fail. Every other path takes the unpreferred assumption on some of them, and each of those costs a known factor.
+
+**Scores are logarithms.** Scores use summed log probabilities to preserve ordering without underflow: a plain product across a wide head rounds to zero, ties, and loses the order entirely. A score is at most 0, and an assumption that cannot happen scores negative infinity. The package doc in `bestfirst.go` shows the failure it avoids.
+
+**Laziness.** One heap holds every path built but not yet handed out, across all heads. Each path handed out is replaced by the one or two that come after it within the same head, so the heap always holds each head's current best and the top of it is the best path in the queue.
+
+The full 2ⁿ path space is never generated, and no path is ever compared against more than the handful in the heap. Work is proportional to the number of heads plus the number of candidates pulled: `Open` scores each unresolved dependency once and sorts each head's alternatives, then every later path costs one build. What is *not* sorted is the path space itself — the alternatives within a head are, once, at `Open`.
+
+`bestfirst` discards nothing: pull long enough and every path for every head comes out. It does no conflict relaxation (`ignored` assumptions) — that is the `Speculator`'s call, not the generator's.
+
+## Worked example
+
+Say a head waits on three dependencies with scores d0 0.9, d1 0.8, d2 0.6. All three are better than even, so its best path assumes all three succeed, with probability 0.9 × 0.8 × 0.6 = 0.432. The tables below use probabilities because they read more easily; the code compares their logarithms. Sorted cheapest-first the alternatives are a0=d2, a1=d1, a2=d0.
+
+Every other path is the best path with some alternatives taken, written as the set of taken alternatives: `{a0}` takes only the cheapest, `{a0,a1}` the two cheapest, and so on. `expand` in `iterator.go` grows this space one step at a time — each path handed out queues the one or two that come just after it:
+
+```
+{}      -> {0}
+{0}     -> {0,1}, {1}
+{1}     -> {1,2}, {2}
+{0,1}   -> {0,1,2}, {0,2}
+{2}, {0,2}, {1,2}, {0,1,2} have nothing left to take above position 2
+```
+
+Pulling all eight paths then goes:
+
+```
+handed out         still waiting afterwards
+{}         .432    {a0}·.288
+{a0}       .288    {a1}·.108  {a0,a1}·.072
+{a1}       .108    {a0,a1}·.072  {a2}·.048  {a1,a2}·.012
+{a0,a1}    .072    {a2}·.048  {a0,a2}·.032  {a1,a2}·.012  {a0,a1,a2}·.008
+{a2}       .048    {a0,a2}·.032  {a1,a2}·.012  {a0,a1,a2}·.008
+{a0,a2}    .032    {a1,a2}·.012  {a0,a1,a2}·.008
+{a1,a2}    .012    {a0,a1,a2}·.008
+{a0,a1,a2} .008    —
+```
+
+The scores come out in order without the eight paths ever being enumerated or sorted together, and the waiting list never holds more than four of them. Stop after two pulls and only four paths were ever built: the head's best, plus the successors queued by those two.
+
+The proof that this walk reaches every path exactly once, in score order, is on `expand` in `iterator.go`. The behavior is pinned by tests in `bestfirst_test.go`.
@@ -0,0 +1,178 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package bestfirst hands out speculation paths for a queue, best ones first.
+//
+// A batch we want to build (a "head") often depends on other batches that have
+// not finished yet. Each unresolved dependency could go either way, so there is
+// more than one sensible thing to build: one path per combination of outcomes.
+//
+// Three terms carry the package:
+//
+//   - An assumption is what a path expects of one dependency — that it succeeds
+//     or that it does not. Once the dependency resolves the assumption is
+//     forced rather than chosen. Of the two, the one with the higher
+//     probability is preferred and the other unpreferred.
+//   - An alternative is the option to take the unpreferred assumption for a
+//     dependency still unresolved. The head's best path takes none of them;
+//     every other path takes some subset.
+//   - A candidate is one complete set of assumptions — a path — waiting in the
+//     heap with its score.
+//
+// # Scores are logarithms
+//
+// Scores use summed log probabilities to preserve ordering without underflow.
+// Multiplying the probabilities instead loses the ordering outright: two heads
+// waiting on 1200 and 1300 coin-flip dependencies both round to 0.0 (the
+// smallest positive float64 is about 2^-1074), so they tie, the heap falls
+// back to comparing path IDs, and it can emit the 1300-dependency path first —
+// one that is 2^100 times less probable.
+//
+// A score is therefore at most 0, more negative meaning less probable, and
+// negative infinity for an assumption that cannot happen. Scores combine by
+// addition, never multiplication. Nothing exponentiates; the score is a
+// ranking key, never a number anyone reports.
+//
+// # Laziness
+//
+// A head with n unresolved dependencies has 2^n paths, and callers want the
+// best handful. The full space is never enumerated: Open builds one path per
+// head, and each pull builds at most the two that come after the path it hands
+// out. Pulling k paths builds at most 1+2k of them, whatever n is.
+package bestfirst
+
+import (
+	"context"
+	"fmt"
+	"math"
+
+	"github.com/uber/submitqueue/submitqueue/entity"
+	"github.com/uber/submitqueue/submitqueue/extension/scorer"
+	"github.com/uber/submitqueue/submitqueue/extension/speculation/generator"
+)
+
+const (
+	// scoreCutoff is where the preferred assumption flips: at or above it a
+	// dependency is assumed to succeed, below it assumed to fail.
+	//
+	// Not a tuning knob. The preferred side is whichever one this favors, so
+	// moving it would leave that side below probability 0.5 and relativeScore
+	// above 0, which breaks the best-first ordering.
+	scoreCutoff = 0.5
+
+	// defaultScore is used for a dependency that is not among the live batches
+	// and so cannot be scored. Nearly every batch a queue accepts does build
+	// successfully, so a high default is closer to the truth than an even split
+	// and keeps an unscoreable dependency from dragging down every path that
+	// assumes it succeeds.
+	defaultScore = 0.95
+)
+
+// gen builds best-first iterators. scorer gives each dependency's probability
+// of building successfully.
+type gen struct {
+	scorer scorer.Scorer
+}
+
+// New returns a best-first generator.Generator. scorer scores each unresolved
+// dependency.
+func New(sc scorer.Scorer) generator.Generator {
+	return gen{scorer: sc}
+}
+
+// Open works out each head's alternatives and queues that head's best path,
+// ready to hand out. All the scoring happens here, which is why Next never
+// calls the scorer.
+//
+// A cancelled or expired ctx aborts with its error and no iterator.
+func (g gen) Open(ctx context.Context, batches []entity.Batch) (generator.PathIterator, error) {
+	batchByID := make(map[string]entity.Batch, len(batches))
+	for _, b := range batches {
+		batchByID[b.ID] = b
+	}
+	scores := newScoreCache(g.scorer, batchByID)
+
+	it := &iterator{}
+	for _, batch := range batches {
+		// Scoring a queue's worth of heads can take a while, so give up as soon
+		// as the caller stops waiting rather than at the end.
+		if err := ctx.Err(); err != nil {
+			return nil, err
+		}
+		// Only Speculating heads get paths. Batches in any other state (created,
+		// merging, cancelling, finished) still tell us how their dependencies
+		// turned out, but we never propose work on them.
+		if batch.State != entity.BatchStateSpeculating {
+			continue
+		}
+		h, err := newHead(ctx, batch, batchByID, scores)
+		if err != nil {
+			return nil, err
+		}
+		// Queue the head's best path — no alternatives taken, every dependency
+		// on its preferred assumption. It is the only path built up front; the
+		// rest grow out of it as the caller pulls.
+		it.push(h, []int{})
+	}
+	return it, nil
+}
+
+// scoreCache answers "what is this dependency's probability of building
+// successfully", scoring each one at most once per run. Several heads usually
+// wait on the same dependency, and there is no reason to score it once per
+// head.
+//
+// The cache lives for one Open and no longer. A later run scores the same batch
+// again, deliberately: a score can move as the queue does. scorer.Scorer's
+// contract says the same and asks implementations to be cheap, so anything
+// expensive to compute belongs behind the implementation's own cache, where it
+// can decide what is safe to reuse across runs.
+type scoreCache struct {
+	scorer  scorer.Scorer
+	batches map[string]entity.Batch
+	scores  map[string]float64
+}
+
+func newScoreCache(sc scorer.Scorer, batches map[string]entity.Batch) *scoreCache {
+	return &scoreCache{scorer: sc, batches: batches, scores: make(map[string]float64)}
+}
+
+// of returns the given dependency's score: the probability its build succeeds.
+//
+// A score outside [0, 1] is rejected rather than used. Everything downstream
+// treats it as a probability — its log is the ranking key, its complement is
+// the other side's probability — so a bad value would not fail loudly. It
+// would produce a positive log, an ordering that is no longer best-first, or a
+// NaN that makes every comparison false.
+func (c *scoreCache) of(ctx context.Context, batchID string) (float64, error) {
+	if score, ok := c.scores[batchID]; ok {
+		return score, nil
+	}
+	batch, ok := c.batches[batchID]
+	if !ok {
+		// Not a live batch, so there is nothing to score. Remember the default
+		// too, so we don't look it up again.
+		c.scores[batchID] = defaultScore
+		return defaultScore, nil
+	}
+	score, err := c.scorer.Score(ctx, batch)
+	if err != nil {
+		return 0, err
+	}
+	if math.IsNaN(score) || score < 0 || score > 1 {
+		return 0, fmt.Errorf("scorer returned %v for batch %q: want a probability in [0, 1]", score, batchID)
+	}
+	c.scores[batchID] = score
+	return score, nil
+}