Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion documentation/adrs/extension-points.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ not provide any help or support for those cases.
- more predictable behavior
- reduced cost of maintaining backward compatibility
- easier to find an actual extension points
- impossible to mock classes that arent marked as `final` in tests (which is a good thing, users shouldnt mock Flow classes in their test suites)
- impossible to mock classes that aren't marked as `final` in tests (which is a good thing, users shouldn't mock Flow classes in their test suites)

## Alternatives Considered (optional)
---
Expand Down
30 changes: 15 additions & 15 deletions documentation/adrs/schema-immutability.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Date: 2026-07-27
## Context
---

`Schema` was mutable `add()`, `remove()`, `rename()`, `merge()` and every other mutator rewrote
`Schema` was mutable - `add()`, `remove()`, `rename()`, `merge()` and every other mutator rewrote
`$this->definitions` and returned `$this`. `Definition::addMetadata()` and `Definition::setMetadata()` did the same
with `$this->metadata`.

Expand All @@ -18,16 +18,16 @@ undeclared row keys, so a column that should materialize in rows must be present
storing the caller's `Schema` therefore wrote every internal extension into the caller's object, producing three
observable defects (#2536 regression):

1. **Caller schema pollution** a `Schema` the user holds for other purposes gains non-nullable columns it never
1. **Caller schema pollution** - a `Schema` the user holds for other purposes gains non-nullable columns it never
declared.
2. **Extractor-lifetime pollution** columns added during one `extract()` run persist into subsequent runs.
3. **Cross-stream pollution** partition columns of one stream leak into the next, and
2. **Extractor-lifetime pollution** - columns added during one `extract()` run persist into subsequent runs.
3. **Cross-stream pollution** - partition columns of one stream leak into the next, and
`Hydrator::cast(fillMissing: true)` injects `null` into a non-nullable definition.

The first fix cloned: `withSchema()` stored `clone $schema`, and extractors cloned again per run and per stream.
That fix was incomplete. `clone` is shallow and `Schema`'s only state is `array<string, Definition>`, so a cloned
`Schema` **shares its `Definition` instances**. `Schema::addMetadata()` / `setMetadata()` reached into a shared
`Definition` and mutated it in place, so metadata writes aliased through every copy including the caller's.
`Definition` and mutated it in place, so metadata writes aliased through every copy - including the caller's.

The same mutable contract left latent aliasing traps elsewhere: `Rows::schema()` seeded its merge loop with row 0's
*memoized* `Schema` and corrupted it, `FloeStreamWriter` retained a caller-owned `Schema` for the lifetime of a
Expand All @@ -39,23 +39,23 @@ write session, and `merge()`'s fast paths returned `$this` or the argument.
**`Schema` and its whole state chain are immutable. Every mutator returns a new instance; nothing is ever written
in place.**

- `Schema` a `final readonly class`. All 19 mutators return `new self(...)`; `setDefinitions()` is the
- `Schema` - a `final readonly class`. All 19 mutators return `new self(...)`; `setDefinitions()` is the
constructor's validation helper and is called from the constructor only.
- `Definition` (19 implementations) each a `final readonly class`. `addMetadata()` and `setMetadata()` return a
- `Definition` (19 implementations) - each a `final readonly class`. `addMetadata()` and `setMetadata()` return a
per-class `new self(...)`, matching the idiom `makeNullable()` and `rename()` already used.
- `Metadata` already a `final readonly class`.
- `Metadata` - already a `final readonly class`.

Immutability is declared at the class level, not per property: a `readonly class` cannot gain a writable property
later, so the guarantee survives future edits instead of depending on whoever adds property number 20 remembering
the rule. It is compiler-enforced, not convention. Extractors and the DSL hold caller-provided `Schema` instances
directly: there is nothing to clone because there is nothing to mutate. Sharing an instance `merge()`'s fast
paths, a retained base `Definition` in the hydrator, `Rows`' memoized schema is safe by construction.
directly: there is nothing to clone because there is nothing to mutate. Sharing an instance - `merge()`'s fast
paths, a retained base `Definition` in the hydrator, `Rows`' memoized schema - is safe by construction.

DSL `from_*()` functions stay pure delegation.

### Out of scope

`EntryReference` remains mutable `as()`, `asc()` and `desc()` write `$alias` / `$sort` on `$this`. It is shared
`EntryReference` remains mutable - `as()`, `asc()` and `desc()` write `$alias` / `$sort` on `$this`. It is shared
with the entire expression DSL, so making it immutable is a separate project and is not attempted here.

### Breaking change
Expand All @@ -73,7 +73,7 @@ $schema = $schema->add(str_schema('x')); // correct

**Advantages:**

- **The bug class is gone**, not patched aliasing is impossible because there is no writable state to alias.
- **The bug class is gone**, not patched - aliasing is impossible because there is no writable state to alias.
- **Compiler-enforced**: a `readonly` violation is a fatal error, not a convention a new extractor can forget.
- **Sharing becomes free**: no defensive clones in extractors, `PhpRowHydrator`, or the native hydrator.
- Fixes the `Rows::schema()` and `FloeStreamWriter` aliasing traps without touching either.
Expand All @@ -92,8 +92,8 @@ $schema = $schema->add(str_schema('x')); // correct
Every `withSchema()` stores `clone $schema`; extractors clone again per run and per stream.

**Rejected because:** the clone is shallow, so `Definition` instances stay shared and metadata mutations alias
through every copy anyway. It is also convention rather than a compiler-enforced rule a new extractor can forget
to clone and it leaves unobservable dead clones in extractors that never extend the schema.
through every copy anyway. It is also convention rather than a compiler-enforced rule - a new extractor can forget
to clone - and it leaves unobservable dead clones in extractors that never extend the schema.

### 2. Deep `Schema::__clone()` + `Definition::__clone()`

Expand All @@ -105,7 +105,7 @@ also makes every clone more expensive without removing the need to remember to c

### 3. Materialize auto-added columns post-hydration via `Row::add()` (Floe style)

**Rejected because:** the extended schema *is* the hydrator's instruction set `Hydrator::cast()` drops undeclared
**Rejected because:** the extended schema *is* the hydrator's instruction set - `Hydrator::cast()` drops undeclared
row keys, and the `findDefinition()` guard lets a user-declared partition column keep its user-defined type.
Post-hydration adds would bypass both.

Expand Down
2 changes: 1 addition & 1 deletion documentation/components/adapters/avro.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[DOC_LINK:/documentation/introduction.md]

- [➡️ Installation](/documentation/installation/packages/etl-adapter-avro.md)
- [Installation](/documentation/installation/packages/etl-adapter-avro.md)

Avro integration was temporarily abandoned due to the lack of availability of good libraries for PHP.
If you are interested in this integration, please let us know by creating an issue in the repository.
Expand Down
6 changes: 3 additions & 3 deletions documentation/components/adapters/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,16 +97,16 @@ data_frame()
->run();
```

This walks `?per_page=100&page=1,2,3` until the `items` path of a response comes back empty.
This walks `?per_page=100&page=1,2,3...` until the `items` path of a response comes back empty.

### Supported strategies

```php
// page number - ?per_page=100&page=1,2,3 default stop: empty page at records_path
// page number - ?per_page=100&page=1,2,3... default stop: empty page at records_path
http_pagination_page_number(http_request_option_query('page'),
page_size: 100, size_option: http_request_option_query('per_page'), records_path: 'items');

// offset / limit - ?limit=100&offset=0,100,200 default stop: total_path reached, else empty page
// offset / limit - ?limit=100&offset=0,100,200... default stop: total_path reached, else empty page
http_pagination_offset(http_request_option_query('offset'), http_request_option_query('limit'),
limit: 100, total_path: 'meta.total');

Expand Down
10 changes: 5 additions & 5 deletions documentation/components/adapters/postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -438,8 +438,8 @@ Two helpers convert between a Flow `Schema` and a PostgreSQL table definition:
- `pgsql_table_to_flow_schema()` turns a `Flow\PostgreSql\Schema\Table` back into a Flow `Schema`.

Column types are resolved through the shared `EntryTypesMap` (Flow type → PostgreSQL column type), and per-column
details primary keys, unique constraints, indexes, length, precision/scale, defaults, identity, generated columns,
and explicit type overrides are driven by `PostgreSqlMetadata` entries attached to each schema definition.
details - primary keys, unique constraints, indexes, length, precision/scale, defaults, identity, generated columns,
and explicit type overrides - are driven by `PostgreSqlMetadata` entries attached to each schema definition.

### Creating a Table from a Flow Schema

Expand Down Expand Up @@ -512,7 +512,7 @@ expressed by attaching the same name to several definitions.

For composite indexes and unique constraints, column order matters. By default columns are ordered the way they appear
in the schema. Pass an explicit `$position` (ascending, lower first) to `index()` / `indexUnique()` to control the
order independently of schema field order useful, for example, for a keyset-pagination index where the leading
order independently of schema field order - useful, for example, for a keyset-pagination index where the leading
column must serve `ORDER BY`:

```php
Expand All @@ -533,7 +533,7 @@ $table = to_pgsql_schema_table(
// => CREATE INDEX orders_created_at_id_idx ON public.orders (created_at, id)
```

Columns without an explicit position default to the end (`PHP_INT_MAX`), and schema order breaks ties so leaving
Columns without an explicit position default to the end (`PHP_INT_MAX`), and schema order breaks ties - so leaving
positions off keeps the schema-order behavior.

#### A Column in Multiple Indexes
Expand All @@ -549,7 +549,7 @@ $schema = schema(
);
```

> Index and unique-constraint names must not contain a colon (`:`) it is reserved internally as the name/position
> Index and unique-constraint names must not contain a colon (`:`) - it is reserved internally as the name/position
> separator and passing one throws an `InvalidArgumentException`.

### Reading a Flow Schema back from a Table
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ $aws = aws_s3_filesystem(
$fstab = fstab($aws);
```

The **mount protocol** the URI scheme under which the filesystem is registered in the
`FilesystemTable` defaults to `'aws-s3'`. Override by passing a fourth argument (e.g.
The **mount protocol** - the URI scheme under which the filesystem is registered in the
`FilesystemTable` - defaults to `'aws-s3'`. Override by passing a fourth argument (e.g.
`aws_s3_filesystem($bucket, $client, options: new Options(), protocol: 'warehouse')`) when you
need to mount the same bucket twice under distinct names or pick a scheme more meaningful to
your application.
Expand Down Expand Up @@ -59,5 +59,5 @@ data_frame($config)
```

`FileStatus` values returned from `list()` and `status()` carry `size` (from S3 `Size` / `ContentLength`)
and `lastModifiedAt` (from `LastModified`) populated directly from the S3 response no extra HEAD call
and `lastModifiedAt` (from `LastModified`) populated directly from the S3 response - no extra HEAD call
is issued when the CLI `flow:filesystem:ls --long` or `flow:filesystem:stat` prints them.
6 changes: 3 additions & 3 deletions documentation/components/bridges/filesystem-azure-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ $sdk = azure_blob_service(
To use the Azure Blob filesystem with Flow, you need to mount the filesystem to the configuration.
This operation will mount the Azure Blob filesystem to the fstab instance available in the DataFrame runtime.

The **mount protocol** the URI scheme under which the filesystem is registered in the
`FilesystemTable` defaults to `'azure-blob'`. Override via the third argument
The **mount protocol** - the URI scheme under which the filesystem is registered in the
`FilesystemTable` - defaults to `'azure-blob'`. Override via the third argument
(`azure_filesystem($blobService, $options, protocol: 'warehouse')`) when you need to mount the same
container under a different scheme.

Expand Down Expand Up @@ -81,5 +81,5 @@ data_frame($config)

`FileStatus` values returned from `list()` and `status()` carry `size` (from the `Content-Length`
header / listing property) and `lastModifiedAt` (from the `Last-Modified` header, parsed as RFC 7231)
populated directly from the Azure response no extra stream is opened when the CLI
populated directly from the Azure response - no extra stream is opened when the CLI
`flow:filesystem:ls --long` or `flow:filesystem:stat` prints them.
2 changes: 1 addition & 1 deletion documentation/components/bridges/phpstan-types-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,4 @@ includes:
- vendor/flow-php/phpstan-types-bridge/extension.neon
```

That's it `type_structure()` calls are now narrowed by PHPStan across your codebase.
That's it - `type_structure()` calls are now narrowed by PHPStan across your codebase.
Loading
Loading