Skip to content

Add instance_pools resource type to bundles (direct engine only)#5934

Open
radakam wants to merge 3 commits into
mainfrom
cli-instance-pool-creation
Open

Add instance_pools resource type to bundles (direct engine only)#5934
radakam wants to merge 3 commits into
mainfrom
cli-instance-pool-creation

Conversation

@radakam

@radakam radakam commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Changes

Add support for a new instance_pools resource type in DABs. This includes:

  • New resources.InstancePool config type (wrapping compute.CreateInstancePool) with permissions support.
  • Direct-engine resource implementation (ResourceInstancePool) covering create/read/update/delete, state remapping, and field behaviors (ignored remote changes, recreate-on-change, backend defaults).
  • Wiring into resource enumerations: supported resources, bundle permissions, presets (prefix + tags), target mode, bind support, and workspace URL patterns.
  • Regenerated JSON schema, validation (enum/required fields), reference schema, and annotations.
  • Testserver handlers for the /api/2.0/instance-pools/* endpoints.

Instance pools are only supported in direct deployment mode.

Why

Users want to manage instance pools—pre-provisioned, idle cloud instances that reduce cluster start and auto-scaling times—declaratively alongside their other bundle resources, instead of provisioning them out-of-band.

Closes #5641.

Tests

  • New acceptance test (acceptance/bundle/resources/instance_pools) exercising validate, summary, deploy, update, and destroy, asserting the exact create/edit/delete API requests.
  • Added instance_pools coverage to invariant acceptance suites (no-drift, migrate, continue), with the resource excluded from Terraform-mode migration since it's direct-only.
  • Unit tests for permissions, presets/target-mode prefixing, bind support, and state-load round-tripping (create/modify/delete).

Adds support for defining Databricks instance pools as a bundle resource.
Instance pools are implemented as a direct-engine-only resource: they are
registered in the direct engine's SupportedResources but not wired to
Terraform, so ValidateDirectOnlyResources rejects them under the terraform
engine.

Includes the resource definition and ConfigResource implementation, the
direct-engine adapter (create/read/edit/delete + RemapState), presets and
bundle-permission wiring, workspace URL generation, testserver CRUD handlers,
schema annotations, and unit + acceptance test coverage.
@radakam
radakam temporarily deployed to test-trigger-is July 15, 2026 14:18 — with GitHub Actions Inactive
@radakam
radakam temporarily deployed to test-trigger-is July 15, 2026 14:18 — with GitHub Actions Inactive
radakam added 2 commits July 16, 2026 07:54
…lidation

Fixes CI failures from adding the instance_pools resource: update the
workspace-open supported-type lists, skip instance_pools (direct-only) in
the Terraform lifecycle test, extend the statemgmt StateToBundle tests for
full resource coverage, and regenerate the validation enum/required fields.
Adds an instance_pools invariant test config (with permissions) so
TestInvariantConfigsCoverage passes, wires it into the invariant matrix,
and excludes it from the migrate variant since instance_pools is
direct-only and the migrate seed deploy uses Terraform. Also refreshes the
experimental/open acceptance golden output to include instance_pools.
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: e17841b

Run: 29482990818

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 227 1104 4:40
💚​ aws windows 4 4 229 1102 6:49
💚​ aws-ucws linux 4 4 316 1020 5:32
💚​ aws-ucws windows 4 4 318 1018 7:22
💚​ azure linux 4 4 227 1103 4:30
💚​ azure windows 4 4 229 1101 7:59
💚​ azure-ucws linux 4 4 318 1017 5:49
💚​ azure-ucws windows 4 4 320 1015 7:48
💚​ gcp linux 4 4 226 1105 4:30
💚​ gcp windows 4 4 228 1103 6:42
8 interesting tests: 4 RECOVERED, 4 SKIP
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 10 slowest tests (at least 2 minutes):
duration env testname
7:35 azure windows TestAccept
6:24 azure-ucws windows TestAccept
6:20 aws windows TestAccept
6:17 gcp windows TestAccept
6:14 aws-ucws windows TestAccept
3:01 azure linux TestAccept
2:57 gcp linux TestAccept
2:54 aws linux TestAccept
2:50 azure-ucws linux TestAccept
2:49 aws-ucws linux TestAccept

@radakam
radakam marked this pull request as ready for review July 16, 2026 09:03
@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

11 files changed
Suggested: @pietern
Also eligible: @janniklasrose, @denik, @andrewnester, @shreyas-goenka, @anton-107, @lennartkats-db

/bundle/ - needs approval

22 files changed
Suggested: @pietern
Also eligible: @janniklasrose, @denik, @andrewnester, @shreyas-goenka, @anton-107, @lennartkats-db

General files (require maintainer)

7 files changed
Based on git history:

  • @pietern -- recent work in bundle/direct/dresources/, libs/testserver/, bundle/internal/validation/generated/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

}

func (r *ResourceInstancePool) DoCreate(ctx context.Context, config *compute.CreateInstancePool) (string, *compute.GetInstancePool, error) {
resp, err := r.client.InstancePools.Create(ctx, *config)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe instance pools have ACTIVE / STOPPED state? Shall we integrate lifecycle: started option here to like we do for clusters and wait for ACTIVE state if we need to?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The states are there, but the analogy doesn't quite fit I think. Unlike clusters, InstancePools doesn't have a start/stop operation or a waiter, Create returns immediately, with no PENDING → ACTIVE transition, so there isn't really anything for a lifecycle.started flag to trigger or wait for. Or is there a use case I'm missing?

@radakam
radakam requested a review from andrewnester July 17, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Add a way to create instance pools

3 participants