Pack eng/docker-tools content in ImageBuilder#2159
Draft
lbussell wants to merge 7 commits into
Draft
Conversation
GenerateArtifactsCommand built its Cottle DocumentConfiguration inline. Move that configuration into a CottleDocumentConfiguration.Create() helper in a new Templating namespace so the same template delimiters and whitespace handling can be reused by other commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add CreateFile, DirectoryExists, DeleteDirectory, and GetCurrentDirectory to IFileSystem (and the FileSystem pass-through implementation) so callers can perform directory-aware, streaming filesystem work behind the existing abstraction. Update the InMemoryFileSystem test double to implement the new members, tracking created/deleted directories and committing CreateFile streams on dispose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the Microsoft.DotNet.DockerTools.Infrastructure library, which embeds a copy of the entire eng/docker-tools directory (pipeline templates, PowerShell scripts, and docs) under src/Infrastructure/Content/ as assembly resources. InfrastructureContent exposes the embedded files by relative path via a trim/AOT-safe manifest-resource index. The copy lives under src/ so it is reachable from the container build context, and register the project in the solution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the 'update' command, which writes ImageBuilder's embedded eng/docker-tools infrastructure to disk as a full mirror, rendering this build's ImageBuilder tag into docker-images.yml via Cottle. The tag is baked into assembly metadata through the IMAGEBUILDER_TAG build arg/env var that the Dockerfiles set and that manifest.json supplies as the pipeline UniqueId; it falls back to "latest" for local builds. Reference the Infrastructure project, register the command, and document the bundling in AGENTS.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds the plumbing for ImageBuilder to ship a bundled copy of the repository’s eng/docker-tools Azure Pipelines infrastructure (templates/scripts/docs) and introduces an update command to write that bundled content back to disk in consuming repos, enabling code + pipeline changes to move together.
Changes:
- Add
Microsoft.DotNet.DockerTools.Infrastructureproject that embedseng/docker-toolscontent as assembly resources and is referenced by ImageBuilder. - Introduce
updatecommand + tag-stamping support so ImageBuilder can emiteng/docker-toolscontent (including rendering the ImageBuilder tag intodocker-images.yml). - Add the initial in-box infrastructure content under
src/Infrastructure/Content/(pipeline templates, scripts, docs) and wire up Docker build args to stamp the tag.
Reviewed changes
Copilot reviewed 101 out of 101 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/manifest.json | Pass IMAGEBUILDER_TAG build arg into ImageBuilder container builds. |
| src/Infrastructure/README.md | Documents why infra is duplicated and how Content/ vs eng/docker-tools/ relate. |
| src/Infrastructure/Microsoft.DotNet.DockerTools.Infrastructure.csproj | New project embedding infra files as resources (AOT/trim analyzers enabled). |
| src/Infrastructure/InfrastructureContent.cs | Resource index + stream access for embedded infra content. |
| src/Infrastructure/Content/templates/variables/sdl-pool.yml | Bundled pipeline variable template. |
| src/Infrastructure/Content/templates/variables/dotnet/secrets.yml | Bundled pipeline secrets variable template. |
| src/Infrastructure/Content/templates/variables/dotnet/secrets-unofficial.yml | Bundled unofficial secrets variable template. |
| src/Infrastructure/Content/templates/variables/dotnet/common.yml | Bundled .NET common variable composition. |
| src/Infrastructure/Content/templates/variables/dotnet/build-test-publish.yml | Bundled .NET build/test/publish variable set. |
| src/Infrastructure/Content/templates/variables/docker-images.yml | Bundled image-name variables with templated ImageBuilder tag placeholder. |
| src/Infrastructure/Content/templates/variables/dnceng-signing.yml | Bundled MicroBuild signing variables. |
| src/Infrastructure/Content/templates/variables/dnceng-project-names.yml | Bundled Azure DevOps project-name variables. |
| src/Infrastructure/Content/templates/variables/dnceng-build-pools.yml | Bundled build pool/image variables. |
| src/Infrastructure/Content/templates/variables/common.yml | Bundled common variables used across templates. |
| src/Infrastructure/Content/templates/variables/common-paths.yml | Bundled common path variables. |
| src/Infrastructure/Content/templates/task-prefix-decorator.yml | Template to prefix task display names when extending 1ES templates. |
| src/Infrastructure/Content/templates/steps/wait-for-mcr-image-ingestion.yml | Step template to wait for MCR image ingestion. |
| src/Infrastructure/Content/templates/steps/wait-for-mcr-doc-ingestion.yml | Step template to wait for MCR doc ingestion. |
| src/Infrastructure/Content/templates/steps/validate-branch.yml | Step template enforcing official-branch publishing rules. |
| src/Infrastructure/Content/templates/steps/test-images-windows-client.yml | Windows image test step template. |
| src/Infrastructure/Content/templates/steps/test-images-linux-client.yml | Linux image test step template (containerized test runner). |
| src/Infrastructure/Content/templates/steps/set-image-info-path-var.yml | Step template to compute versions-repo image-info paths. |
| src/Infrastructure/Content/templates/steps/set-dry-run.yml | Step template to compute dry-run behavior for non-prod scenarios. |
| src/Infrastructure/Content/templates/steps/run-pwsh-with-auth.yml | Authenticated AzureCLI wrapper step template. |
| src/Infrastructure/Content/templates/steps/run-imagebuilder.yml | Standard ImageBuilder invocation step template (authed/non-authed). |
| src/Infrastructure/Content/templates/steps/retain-build.yml | Step template to retain builds permanently. |
| src/Infrastructure/Content/templates/steps/reference-service-connections.yml | Step template to reference service connections for OIDC token issuance. |
| src/Infrastructure/Content/templates/steps/publish-readmes.yml | Step template to publish MCR docs/readmes and wait for ingestion. |
| src/Infrastructure/Content/templates/steps/publish-artifact.yml | Step template to publish pipeline artifacts (internal vs public). |
| src/Infrastructure/Content/templates/steps/parse-test-arg-arrays.yml | Step template to parse test arg lists into PowerShell array literals. |
| src/Infrastructure/Content/templates/steps/init-testrunner.yml | Step template to prep the Linux test-runner image. |
| src/Infrastructure/Content/templates/steps/init-signing-linux.yml | Step template to install/configure signing tooling for Linux jobs. |
| src/Infrastructure/Content/templates/steps/init-imagebuilder.yml | Step template to set up ImageBuilder (pull/extract/bootstrap + appsettings). |
| src/Infrastructure/Content/templates/steps/init-common.yml | Common initialization template (checkout, path vars, cleanup, ImageBuilder setup). |
| src/Infrastructure/Content/templates/steps/generate-appsettings.yml | Step template generating appsettings.json publish/build configuration. |
| src/Infrastructure/Content/templates/steps/download-build-artifact.yml | Step template to download pipeline artifacts (current/specific run). |
| src/Infrastructure/Content/templates/steps/copy-base-images.yml | Step template to copy base images into an ACR mirror (supports dry-run). |
| src/Infrastructure/Content/templates/steps/cleanup-docker-windows.yml | Step template to clean up Docker resources on Windows agents. |
| src/Infrastructure/Content/templates/steps/cleanup-docker-linux.yml | Step template to clean up Docker resources on Linux agents. |
| src/Infrastructure/Content/templates/steps/clean-acr-images.yml | Step template for ACR cleanup operations via ImageBuilder. |
| src/Infrastructure/Content/templates/steps/annotate-eol-digests.yml | Step template to generate/publish EOL annotations and wait for ingestion. |
| src/Infrastructure/Content/templates/stages/publish.yml | Publish stage wrapper template. |
| src/Infrastructure/Content/templates/stages/dotnet/publish.yml | .NET-specific publish stage wrapper. |
| src/Infrastructure/Content/templates/stages/dotnet/publish-config-prod.yml | .NET production publish config injection template. |
| src/Infrastructure/Content/templates/stages/dotnet/publish-config-nonprod.yml | .NET non-prod publish config injection template. |
| src/Infrastructure/Content/templates/stages/dotnet/build-test-publish-repo.yml | .NET build/test/publish stage composition wrapper. |
| src/Infrastructure/Content/templates/stages/dotnet/build-and-test.yml | .NET-specific build-and-test stage wrapper. |
| src/Infrastructure/Content/templates/stages/build-and-test.yml | Core build/test/sign/publish pipeline stage orchestration. |
| src/Infrastructure/Content/templates/jobs/test-images-windows-client.yml | Job template for Windows image testing. |
| src/Infrastructure/Content/templates/jobs/test-images-linux-client.yml | Job template for Linux image testing. |
| src/Infrastructure/Content/templates/jobs/sign-images.yml | Job template for signing + signature verification. |
| src/Infrastructure/Content/templates/jobs/publish.yml | Job template to publish images, readmes, image-info, and notifications. |
| src/Infrastructure/Content/templates/jobs/post-build.yml | Job template to merge image-info and create manifest lists post-build. |
| src/Infrastructure/Content/templates/jobs/generate-matrix.yml | Job template for generating build/test matrices. |
| src/Infrastructure/Content/templates/jobs/copy-base-images.yml | Job template for copy-base-images execution. |
| src/Infrastructure/Content/templates/jobs/copy-base-images-staging.yml | Staging wrapper for copy-base-images job template. |
| src/Infrastructure/Content/templates/jobs/cg-build-projects.yml | CG-oriented job template to build projects outside Docker. |
| src/Infrastructure/Content/templates/jobs/build-images.yml | Job template to build images (and SBOM generation/publish where applicable). |
| src/Infrastructure/Content/templates/1es.yml | 1ES wrapper template using the task-prefix decorator. |
| src/Infrastructure/Content/templates/1es-unofficial.yml | 1ES unofficial wrapper template with SDL/CG settings. |
| src/Infrastructure/Content/templates/1es-official.yml | 1ES official wrapper template with SDL/CG settings. |
| src/Infrastructure/Content/skill-helpers/Show-PullRequestComments.ps1 | Helper script to summarize PR comments and inline review comments. |
| src/Infrastructure/Content/skill-helpers/Show-PullRequestBuilds.ps1 | Helper script to summarize PR checks and expand AzDO build timelines. |
| src/Infrastructure/Content/skill-helpers/Show-BuildTimeline.ps1 | Helper script to print AzDO build timeline tree. |
| src/Infrastructure/Content/skill-helpers/Get-RecentBuilds.ps1 | Helper script to list recent builds under an AzDO folder. |
| src/Infrastructure/Content/skill-helpers/Get-FailingPipelines.ps1 | Helper script to list pipelines with failing latest runs. |
| src/Infrastructure/Content/skill-helpers/Get-BuildLog.ps1 | Helper script to retrieve a specific AzDO build log. |
| src/Infrastructure/Content/skill-helpers/AzureDevOps.ps1 | Helper library for authenticated AzDO REST API calls. |
| src/Infrastructure/Content/Retain-Build.ps1 | Script to set AzDO builds to keep-forever. |
| src/Infrastructure/Content/readme.md | “Do not edit” readme for the bundled infra directory. |
| src/Infrastructure/Content/Pull-Image.ps1 | Script helper to docker pull with retry. |
| src/Infrastructure/Content/Invoke-WithRetry.ps1 | Script helper to retry arbitrary commands. |
| src/Infrastructure/Content/Invoke-ImageBuilder.ps1 | Script helper to run ImageBuilder (containerized on Linux, native on Windows). |
| src/Infrastructure/Content/Invoke-CleanupDocker.ps1 | Script helper to prune docker resources while preserving key images. |
| src/Infrastructure/Content/Install-DotNetSdk.ps1 | Script helper to install a specific .NET SDK channel. |
| src/Infrastructure/Content/Get-ImageNameVars.ps1 | Script helper to parse docker image-name variables from YAML. |
| src/Infrastructure/Content/Get-ImageBuilder.ps1 | Script helper to ensure ImageBuilder image is pulled locally. |
| src/Infrastructure/Content/Get-BaseImageStatus.ps1 | Script helper to query base-image status via ImageBuilder. |
| src/Infrastructure/Content/Dockerfile.WithRepo | Dockerfile layering repo content into an existing image. |
| src/Infrastructure/Content/Dockerfile.syft | Dockerfile used to produce SBOMs via syft. |
| src/Infrastructure/Content/CHANGELOG.md | Bundled docker-tools changelog. |
| src/Infrastructure/Content/build.ps1 | Script helper to invoke ImageBuilder build with filters. |
| src/ImageBuilder/Templating/CottleDocumentConfiguration.cs | Centralized Cottle template configuration factory. |
| src/ImageBuilder/PathHelper.cs | Adds SafeCombine helper for traversal-resistant path joining. |
| src/ImageBuilder/Microsoft.DotNet.ImageBuilder.csproj | Stamps tag as assembly metadata; references new Infrastructure project. |
| src/ImageBuilder/ImageBuilder.cs | Registers tag provider and new update command in DI. |
| src/ImageBuilder/IImageBuilderTagProvider.cs | New abstraction to provide the current ImageBuilder tag. |
| src/ImageBuilder/IFileSystem.cs | Extends filesystem abstraction for directory ops + stream creation + CWD access. |
| src/ImageBuilder/FileSystem.cs | Implements the expanded IFileSystem contract. |
| src/ImageBuilder/Commands/UpdateOptions.cs | CLI options for the update command (--init). |
| src/ImageBuilder/Commands/UpdateCommand.cs | Implements update to mirror bundled infra content to eng/docker-tools. |
| src/ImageBuilder/Commands/GenerateArtifactsCommand.cs | Uses shared Cottle configuration (dedup). |
| src/ImageBuilder/AssemblyImageBuilderTagProvider.cs | Reads ImageBuilder tag from assembly metadata. |
| src/ImageBuilder.Tests/UpdateCommandTests.cs | Unit tests covering update mirroring, templating, dry-run, and guardrails. |
| src/ImageBuilder.Tests/PathHelperTests.cs | Unit tests for SafeCombine. |
| src/ImageBuilder.Tests/Helpers/InMemoryFileSystem.cs | Test FS updated to support new IFileSystem API surface. |
| src/Dockerfile.windows | Plumbs IMAGEBUILDER_TAG env var + restores new Infrastructure project. |
| src/Dockerfile.linux | Plumbs IMAGEBUILDER_TAG env var + restores new Infrastructure project. |
| Microsoft.DotNet.DockerTools.slnx | Adds Infrastructure project to the solution. |
| AGENTS.md | Documents the new bundled-infra workflow and where to edit templates. |
Comment on lines
+76
to
+78
| - template: /eng/docker-tools/templates/steps/validate-branch.yml@self | ||
| parameters: | ||
| internalProjectName: ${{ parameters.internalProjectName }} |
| parameters: | ||
| targetPath: $(imageInfoHostDir) | ||
| artifactName: image-info | ||
| piplineDefinitionId: ${{ parameters.sourceBuildPipelineDefinitionId }} |
Comment on lines
+137
to
+140
| - template: /eng/docker-tools/templates/steps/publish-readmes.yml@self | ||
| parameters: | ||
| dryRunArg: $(dryRunArg) | ||
| condition: and(succeeded(), eq(variables['publishReadme'], 'true')) |
Comment on lines
+1
to
+3
| parameters: | ||
| dryRunArg: "" | ||
| condition: true |
Member
I like the tagging approach. For #2132, I mostly think that is lower priority. One option to consider is a solution that uses Renovate to handle the invocation of the |
Member
Author
Yes, ultimately, I would like to move away from dotnet-docker bot driving the updates. However, you just discovered the problem I faced when choosing this implementation direction. The problem with this solution is that ImageBuilder, at least in its current form, needs to emit a template that references itself. If Renovate updated the I think the only way around that might be to store the ImageBuilder version/tag/digest in an unmanaged file. That way, the version only flows in one direction, from the host repo to the ImageBuilder tooling, and not the other way around. One other option is to have some update script that tells ImageBuilder at runtime what digest it's running from. It would resolve the latest digest and pass it in as an env var when running the |
Member
What about passing the digest/tag in as a parameter to the |
Member
Author
Yep, that's one option that I wrote at the end of my comment. |
The update command now accepts the ImageBuilder image reference as an
optional argument instead of resolving a tag baked into the build via
assembly metadata. When omitted, it falls back to the published 'latest'
reference.
Removes the embedded-tag mechanism (IImageBuilderTagProvider, the
ImageBuilderTag assembly metadata, the IMAGEBUILDER_TAG Docker build
arg/env, and the manifest buildArgs) and renames the docker-images.yml
template placeholder to {{IMAGEBUILDER_REF}}, which now holds the full
image reference.
Adds an example Update-ImageBuilder.ps1 (bundled in ImageBuilder and
mirrored to eng/docker-tools) that resolves the multi-platform image
index digest of the latest image via 'docker buildx imagetools inspect'
and passes it to the update command.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of #2130. It includes the infrastructure for packaging the templates/infra inside ImageBuilder. Each commit is individually reviewable.
The new
updatecommand outputs the templates onto disk, assuming some basic pre-conditions are met. The one tricky part is how ImageBuilder knows about/resolves its own tag.Not included yet: Updates to the ImageBuilder update pipelines. That is blocked on #2158.