Skip to content

feat(source-files): generative source dependencies#271

Open
Tonisal-byte wants to merge 7 commits into
microsoft:mainfrom
Tonisal-byte:asalinas/source-generation-dependencies
Open

feat(source-files): generative source dependencies#271
Tonisal-byte wants to merge 7 commits into
microsoft:mainfrom
Tonisal-byte:asalinas/source-generation-dependencies

Conversation

@Tonisal-byte

Copy link
Copy Markdown
Contributor

Adds support for custom source generation inputs.

Custom source-files entries can now declare origin.inputs, a list of already-fetched source filenames that azldev stages next to the generation script inside the mock chroot. This lets scripts consume upstream tarballs or earlier source files directly by relative filename, without knowing azldev’s internal staging paths.

Also updates source preparation ordering so component sources are fetched before custom source generation runs, making upstream source tarballs available as inputs. Custom script stdout/stderr remain quiet by default, but are streamed with --verbose and included in failure diagnostics.

[[components.telegraf.source-files]]
filename = "telegraf-1.38.2-vendor-stripped.tar.gz"
hash-type = "SHA512"
hash = "abc123..."
origin.type = "custom"
origin.script = "gen-telegraf-vendor-stripped.sh"
origin.inputs = ["telegraf-1.38.2.tar.gz"]
origin.mock-packages = ["tar", "gzip"]

Inside gen-telegraf-vendor-stripped.sh, the input is available beside the script:

#!/usr/bin/env bash
set -euo pipefail

tar -xf ./telegraf-1.38.2.tar.gz
# write final generated contents into /azldev-gen/output/

Copilot AI review requested due to automatic review settings July 8, 2026 23:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for declaring “already-fetched” source-file inputs for origin.type = "custom" generation scripts, and adjusts source preparation ordering so upstream sources are available before custom generation runs.

Changes:

  • Introduces origin.inputs for components.*.source-files and validates it in config parsing (schema + docs + tests updated).
  • Updates source preparation flow to fetch component sources before running custom generation, and prevents lookaside extraction from overwriting configured source-files names.
  • Enhances custom generation execution to optionally stream script output with --verbose and include stdout/stderr in failure diagnostics.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
schemas/azldev.schema.json Adds JSON schema for origin.inputs.
scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap Updates schema snapshot to include origin.inputs.
scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap Updates schema snapshot to include origin.inputs.
internal/providers/sourceproviders/sourcemanager.go Passes verbosity into custom provider; prevents lookaside extraction from overwriting configured source-files.
internal/providers/sourceproviders/customsourceprovider.go Stages declared inputs next to script; refactors runner setup; improves script output handling/diagnostics.
internal/providers/sourceproviders/customsourceprovider_internal_test.go Adds unit tests for input staging and output formatting.
internal/projectconfig/configfile.go Validates origin.inputs (only allowed for custom, and entries must be valid filenames).
internal/projectconfig/configfile_test.go Adds tests covering origin.inputs validation.
internal/projectconfig/component.go Adds Origin.Inputs and fingerprints it when non-empty.
internal/app/azldev/core/sources/sourceprep.go Reorders source fetching so component sources come before source-files fetching.
internal/app/azldev/core/sources/sourceprep_test.go Updates expectations for the new fetch ordering.
internal/app/azldev/cmds/component/history_internal_test.go Extends fingerprintable-field coverage to include Origin.Inputs.
internal/app/azldev/cmds/component/history_customizations.go Surfaces source-files.inputs in customization history.
docs/user/reference/config/components.md Documents origin.inputs and clarifies upstream tarballs are already available as inputs.

Comment thread internal/providers/sourceproviders/customsourceprovider.go Outdated
Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/providers/sourceproviders/customsourceprovider.go Outdated
Comment thread internal/app/azldev/core/sources/sourceprep.go
Copilot AI review requested due to automatic review settings July 9, 2026 17:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comment thread internal/projectconfig/configfile.go Outdated
Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/providers/sourceproviders/customsourceprovider.go
Copilot AI review requested due to automatic review settings July 9, 2026 17:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/providers/sourceproviders/customsourceprovider.go
Copilot AI review requested due to automatic review settings July 9, 2026 17:54
@Tonisal-byte Tonisal-byte force-pushed the asalinas/source-generation-dependencies branch from 4330c9f to 96dcb51 Compare July 9, 2026 17:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread docs/user/reference/config/components.md
component.GetName(), err)
}

// Fetch custom and downloaded source files.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is an ordering issue here, FetchComponent can grab a dist-git file first, and if it happens to have the same name as something in source-files, we skip further checks. We should probably just not allow things like combining replace-upstream with this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the latest iteration I make sure source preparation never downloads any file already defined in source-files. Of course we continue to support replace-upstream to ensure the dev understands they are replacing a source already present in the upstream

Comment thread internal/projectconfig/configfile.go
Copilot AI review requested due to automatic review settings July 10, 2026 18:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/providers/sourceproviders/sourcemanager_internal_test.go
Copilot AI review requested due to automatic review settings July 10, 2026 21:10
@Tonisal-byte Tonisal-byte force-pushed the asalinas/source-generation-dependencies branch from b5068fa to 4377f80 Compare July 10, 2026 21:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/providers/sourceproviders/customsourceprovider.go Outdated
Copilot AI review requested due to automatic review settings July 10, 2026 21:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comment thread internal/providers/sourceproviders/customsourceprovider.go
Comment thread internal/app/azldev/core/sources/sourceprep.go
Copilot AI review requested due to automatic review settings July 10, 2026 21:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comment thread internal/providers/sourceproviders/sourcemanager.go
Comment thread internal/providers/sourceproviders/customsourceprovider.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants