Skip to content

fix(bazel): classify bazelisk download network failures as retryable - #236

Merged
xytan0056 merged 3 commits into
mainfrom
yushan/bazel-query-timeout-error
Jul 28, 2026
Merged

fix(bazel): classify bazelisk download network failures as retryable#236
xytan0056 merged 3 commits into
mainfrom
yushan/bazel-query-timeout-error

Conversation

@yushan8

@yushan8 yushan8 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Intent:

  • A bazel client creation failure previously returned a plain error that
    fell through to the ErrorInfra default in tangoerrors.GetErrorCode --
    correct by accident, not by design. A transient network failure while
    downloading bazelisk is retryable and shouldn't be lumped in with
    permanent infra failures (e.g. a bad BazelCommand path).

Changes:

  • Add bazel.ErrDownloadBazeliskNetwork, wrapped when downloading bazelisk
    fails with a net.Error.
  • Add orchestrator.classifyBazelClientError, classifying
    ErrDownloadBazeliskNetwork as tangoerrors.ErrorInfraRetryable and
    everything else as tangoerrors.ErrorInfra; wired into nativeOrchestrator.

Test Plan

unit test

Revert Plan

Revert this PR; classification falls back to the existing ErrorInfra default.

Issues

@yushan8 yushan8 changed the title fix(bazel): classify bazel query timeout as an infra error fix(bazel): classify bazel client/query failures as infra errors Jul 24, 2026
Comment thread graphrunner/native.go Outdated
g.emitter.DurationHistogram(_opCompute, "bazel_query_duration", _phaseDurationBuckets).RecordDuration(time.Since(bazelStart))
if err != nil {
return targethasher.EmptyResult(), err
return targethasher.EmptyResult(), classifyBazelQueryError(err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks simple enough to inline ?

return targethasher.Empty, tangoerrors.NewInfra(error.Wrap(err, "baze query fail..."))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. We have different errors based on the exit code so this needs to be wrapped in a function to classify them.

@yushan8
yushan8 requested a review from xytan0056 July 27, 2026 16:38
@yushan8 yushan8 changed the title fix(bazel): classify bazel client/query failures as infra errors fix(bazel): classify bazel client/query failures by error type Jul 27, 2026
@yushan8
yushan8 force-pushed the yushan/bazel-query-timeout-error branch from 0c6e0ed to c9b8d63 Compare July 27, 2026 16:47
@yushan8
yushan8 marked this pull request as ready for review July 27, 2026 16:55
@yushan8
yushan8 requested review from a team as code owners July 27, 2026 16:55
Comment thread core/bazel/query.go Outdated
streamErr := g.Wait()
if waitErr != nil {
if isGraphFailure(waitErr) {
return queryResults, b.wrapQueryFailure("bazel query failed", fmt.Errorf("%w: %w", ErrQueryGraphFailure, waitErr), &stderrBuf)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justinwon777
CLI imports this pacakge , does this error surface to bk jobs (end up a specific error code) and get classified accordingly? or should we let uber's bk classifier take over everything?

Summary:
Intent:
- A bazel client creation failure previously returned a plain error that
  fell through to the ErrorInfra default in tangoerrors.GetErrorCode --
  correct by accident, not by design. A transient network failure while
  downloading bazelisk is retryable and shouldn't be lumped in with
  permanent infra failures (e.g. a bad BazelCommand path).

Changes:
- Add bazel.ErrDownloadBazeliskNetwork, wrapped when downloading bazelisk
  fails with a net.Error.
- Add orchestrator.classifyBazelClientError, classifying
  ErrDownloadBazeliskNetwork as tangoerrors.ErrorInfraRetryable and
  everything else as tangoerrors.ErrorInfra; wired into nativeOrchestrator.

## Test Plan
unit test

## Revert Plan
Revert this PR; classification falls back to the existing ErrorInfra default.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@yushan8
yushan8 force-pushed the yushan/bazel-query-timeout-error branch from 0f24f7c to 5d5ed61 Compare July 27, 2026 20:30
@yushan8
yushan8 requested a review from xytan0056 July 27, 2026 20:30
@yushan8 yushan8 changed the title fix(bazel): classify bazel client/query failures by error type fix(bazel): classify bazelisk download network failures as retryable Jul 27, 2026
Comment thread core/bazel/bazel.go Outdated
Review feedback: the sentinel fires on any net.Error, not something
specific to bazelisk downloads, and the old name doubled up with the
"download bazelisk" prefix already added at the call site.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@xytan0056
xytan0056 merged commit d0e9bb3 into main Jul 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants