Skip to content
Draft
42 changes: 40 additions & 2 deletions docs/user/reference/config/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,8 +319,21 @@ The `origin` field specifies how to obtain the source file.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Type | `type` | string | **Yes** | Origin type. Currently only `"download"` is supported. |
| URI | `uri` | string | No | URI to download the file from (required when type is `"download"`) |
| Type | `type` | string | **Yes** | Origin type: `"download"` or `"overlay"` (see below) |
| URI | `uri` | string | No | URI to download the file from (required when type is `"download"`; must be omitted when type is `"overlay"`) |

#### `"download"` origin

Downloads the file from the given `uri`. Tries the lookaside cache first using `hash`/`hash-type`, then falls back to the URI.

#### `"overlay"` origin

Used when archive overlays (see [Archive overlays](overlays.md#archive-overlays)) modify an existing upstream source archive. No download occurs; the archive is already present on disk as a spec source. The `hash` and `hash-type` fields record the **expected hash after all overlays have been applied** to that archive.

- **`hash` and `hash-type` are required** for `"overlay"` entries — the hash cannot be computed at render time because sources are not downloaded.
- **`replace-upstream = true` is required** — the archive already exists in the upstream `sources` file, and this entry replaces its hash with the post-overlay value.
- During `prep-sources` (full run), azldev verifies that the hash it computed after repacking the archive matches the stated `hash`. A mismatch means the config is stale and must be updated.
- During `render --check-only`, the stated hash is injected directly without downloading or repacking, allowing the check to pass deterministically.

### Example

Expand All @@ -338,6 +351,31 @@ hash-type = "SHA512"
origin = { type = "download", uri = "https://example.com/repo/pkgs/shim/shimaa64.efi/sha512/.../shimaa64.efi" }
```

### Recording the post-overlay hash for archive overlays

When you apply archive overlays (e.g. removing vendored files from a tarball) using `file-remove` or `file-search-replace` with an archive-scoped path, the repacked archive has a different hash than the original. Use a `source-files` entry with `origin.type = "overlay"` to record the expected post-overlay hash:

```toml
[[components.apache-commons-compress.source-files]]
filename = "commons-compress-1.27.1-src.tar.gz"
hash = "c7a2cef26959e687ad19b96b5ba8393d7514095e13bf0f29bd41e6b3c3cb2260d8ff23283ff3d5fd137b2522b843e7f0f50ab46bcf0f66df5383674f35f223ab"
hash-type = "SHA512"
origin = { type = "overlay" }
replace-upstream = true
replace-reason = "Upstream source tarball contains test fixtures flagged as malware by the AZL RPM signing pipeline. These files are not needed at runtime and are removed to allow SRPM publication."
```

**Workflow:**

1. Add the archive overlay(s) in the component's `[[overlays]]` array.
2. Run `prep-sources` once — this repacks the archive and prints the computed hash in the error message.
3. Paste the computed `hash` and `hash-type` into the `source-files` entry above.
4. Run `prep-sources` again to confirm the hash matches, then commit.
Comment thread
Tonisal-byte marked this conversation as resolved.

After that, `render --check-only` will pass deterministically without downloading or repacking the archive.

`replace-upstream = true` and `replace-reason` are required because the archive is already in the upstream `sources` file. The entry replaces its hash with the post-overlay value, regardless of how many overlays target that archive.

### Replacing an upstream `sources` entry

By default, declaring a `source-files` entry whose `filename` matches one already
Expand Down
63 changes: 61 additions & 2 deletions docs/user/reference/config/overlays.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,20 +47,51 @@ successfully makes a replacement to at least one matching file.
| `file-remove` | Removes a file | `file` | Glob pattern for files to remove |
| `file-rename` | Renames a file within the same directory | `file`, `replacement` | Name of file to rename |

> **Tip:** `file-remove` and `file-search-replace` can also operate inside a source archive by
> prefixing the `file` path with the archive name — see [Archive Overlays](#archive-overlays).

### Archive Overlays

A `file-remove` or `file-search-replace` overlay can modify files **inside** a source archive
instead of loose files in the sources tree. This is detected from the `file` path: when its first
segment is a source archive (e.g. `pkg-1.0.tar.gz`) followed by an inner path, the overlay is
scoped to that archive and the remainder is matched against its contents. The archive is extracted,
the matching files are modified with the same machinery as loose-file overlays, and the archive is
repacked with its original compression format.

```
file = "pkg-1.0.tar.gz/vendor/**" # inside the archive (glob = vendor/**)
file = "vendor/**" # loose files in the sources tree
file = "old.tar.gz" # removes the archive file itself (bare name, no inner path)
```

> **Note:** Archive overlays are batched per archive — all overlays targeting the same archive
> share a single extract/modify/repack cycle — and the `sources` file is rehashed afterward to
> reflect the repacked archive. They are processed independently of spec and loose-file overlays.

> **Extraction root:** The inner path is interpreted relative to the archive's extraction root: if the archive unpacks to a single top-level directory (the conventional `%{name}-%{version}` layout) that directory is the root; otherwise the archive root is used.

> **Supported entry types:** Only regular files, directories, and symlinks are supported inside an archive overlay's target. If the archive contains an entry that cannot be repacked safely (a hardlink, device node, FIFO, etc.), the overlay fails with an error rather than silently dropping the entry from the repacked archive.

| Type | Description | Required Fields |
|------|-------------|-----------------|
| `file-remove` (archive-scoped path) | Removes file(s) matching a glob pattern from inside an archive | `file` (e.g. `pkg-1.0.tar.gz/vendor/**`) |
| `file-search-replace` (archive-scoped path) | Regex-based search and replace on file(s) inside an archive | `file`, `regex` |

## Field Reference

| Field | TOML Key | Description | Used By |
|-------|----------|-------------|---------|
| Type | `type` | **Required.** The overlay type to apply | All overlays |
| Description | `description` | Human-readable explanation documenting the need for the change; helps identify overlays in error messages | All (optional) |
| Tag | `tag` | The spec tag name (e.g., `BuildRequires`, `Requires`, `Version`) | `spec-add-tag`, `spec-insert-tag`, `spec-set-tag`, `spec-update-tag`, `spec-remove-tag` |
| Value | `value` | The tag value to set, or value to match for removal | `spec-add-tag`, `spec-insert-tag`, `spec-set-tag`, `spec-update-tag`, `spec-remove-tag` (optional for matching) |
| Value | `value` | The tag value to set, or value to match for removal. | `spec-add-tag`, `spec-insert-tag`, `spec-set-tag`, `spec-update-tag`, `spec-remove-tag` (optional for matching) |
| Section | `section` | The spec section to target (e.g., `%build`, `%install`, `%files`, `%description`) | `spec-prepend-lines`, `spec-append-lines`, `spec-search-replace` (optional), `spec-remove-section` |
| Package | `package` | The sub-package name for multi-package specs; omit to target the main package | All spec overlays (optional, except `spec-remove-subpackage` which **requires** it) |
| Regex | `regex` | Regular expression pattern to match | `spec-search-replace`, `file-search-replace` |
| Replacement | `replacement` | Literal replacement text; capture group references like `$1` are **not** expanded. Omit or leave empty to delete matched text. | `spec-search-replace`, `file-search-replace`, `file-rename` |
| Lines | `lines` | Array of text lines to insert | `spec-prepend-lines`, `spec-append-lines`, `file-prepend-lines` |
| File | `file` | The name of the non-spec file to modify or add | `file-prepend-lines`, `file-search-replace`, `file-add`, `file-remove`, `file-rename`, `patch-add` (optional), `patch-remove` |
| File | `file` | The name of the non-spec file to modify or add, or a glob pattern. An archive-scoped path (e.g. `pkg-1.0.tar.gz/vendor/**`) targets files inside that source archive. | `file-prepend-lines`, `file-search-replace`, `file-add`, `file-remove`, `file-rename`, `patch-add` (optional), `patch-remove` |
| Source | `source` | Path to source file for `file-add` and `patch-add`; relative paths are relative to the config file that defines the overlay (the overlay file if loaded via [`overlay-files`](#per-file-overlay-format), otherwise the component config) | `file-add`, `patch-add` |
| Metadata | `metadata` | Documentation table describing intent and provenance — see [Overlay Metadata](#overlay-metadata). Not allowed inside an overlay file loaded via `overlay-files` (the file-level `[metadata]` block applies to every overlay in the file). | All (optional) |

Expand Down Expand Up @@ -417,6 +448,34 @@ description = "Remove CVE patches that are now upstream"
> `PatchN` tags. Macro-based tag numbering (e.g., `Patch%{n}`) is not expanded and may
> conflict with auto-assigned numbers.

### Removing a File from an Archive

Prefix the `file` path with the archive name to delete files matching a glob pattern from inside a
source archive. The archive is extracted, matching files are removed, and the archive is repacked.

```toml
[[components.mypackage.overlays]]
type = "file-remove"
file = "mypackage-1.0.tar.gz/vendor/**"
description = "Remove all bundled vendor files"
```

> **Tip:** Without the archive-name prefix, the same `file-remove` overlay removes a loose file
> from the sources tree instead.

### Search and Replace Inside an Archive

Prefix the `file` path with the archive name to rewrite content inside an archive:

```toml
[[components.mypackage.overlays]]
type = "file-search-replace"
file = "mypackage-1.0.tar.xz/configure.ac"
regex = "AC_CHECK_LIB\\(old_lib"
replacement = "AC_CHECK_LIB(new_lib"
description = "Update library reference in configure script"
```

### Removing a Section

The `spec-remove-section` overlay removes an entire section from the spec, including its
Expand Down
25 changes: 18 additions & 7 deletions internal/app/azldev/cmds/component/preparesources.go
Original file line number Diff line number Diff line change
Expand Up @@ -138,13 +138,7 @@ func PrepareComponentSources(env *azldev.Env, options *PrepareSourcesOptions) er
)
}

if options.AllowNoHashes {
preparerOpts = append(preparerOpts, sources.WithAllowNoHashes())
}

if options.SkipSources {
preparerOpts = append(preparerOpts, sources.WithSkipLookaside())
}
preparerOpts = appendPrepareSourcesOptions(preparerOpts, options)

preparer, err := sources.NewPreparer(sourceManager, env.FS(), env, env, preparerOpts...)
if err != nil {
Expand Down Expand Up @@ -194,3 +188,20 @@ func CheckOutputDir(env *azldev.Env, options *PrepareSourcesOptions) error {
"use --force to delete and recreate it",
options.OutputDir)
}

// appendPrepareSourcesOptions appends conditional preparer options that control
// hashing and lookaside behavior.
func appendPrepareSourcesOptions(
opts []sources.PreparerOption,
options *PrepareSourcesOptions,
) []sources.PreparerOption {
if options.AllowNoHashes {
opts = append(opts, sources.WithAllowNoHashes())
}

if options.SkipSources {
opts = append(opts, sources.WithSkipLookaside())
}

return opts
}
Loading
Loading