diff --git a/.github/workflows/publish-docs.yml b/.github/workflows/publish-docs.yml new file mode 100644 index 0000000..40c56b0 --- /dev/null +++ b/.github/workflows/publish-docs.yml @@ -0,0 +1,67 @@ +name: Publish Documentation + +on: + push: + branches: + - main + paths: + - "docs/**" + - ".github/workflows/publish-docs.yml" + + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: github-pages + cancel-in-progress: false + +jobs: + build: + name: Build DocFX Site + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Setup .NET SDK + uses: actions/setup-dotnet@v4 + with: + dotnet-version: "10.0.x" + + - name: Install DocFX + run: dotnet tool update --global docfx + + - name: Configure GitHub Pages + uses: actions/configure-pages@v5 + + - name: Build documentation + run: docfx docs/docfx.json + + - name: Upload GitHub Pages artifact + uses: actions/upload-pages-artifact@v4 + with: + path: docs/_site + + deploy: + name: Deploy GitHub Pages + needs: build + + permissions: + pages: write + id-token: write + + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + + runs-on: ubuntu-latest + + steps: + - name: Deploy GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 \ No newline at end of file diff --git a/.gitignore b/.gitignore index 7302be4..b0368f7 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,4 @@ obj/ **/obj/ /.vs /DataArc.EntityFrameworkCore.Demo.Benchmark/BenchmarkDotNet.Artifacts +/docs/_site diff --git a/docs/bulk-and-parallel-operations.md b/docs/bulk-and-parallel-operations.md new file mode 100644 index 0000000..e6ccf22 --- /dev/null +++ b/docs/bulk-and-parallel-operations.md @@ -0,0 +1,750 @@ +# Bulk and Parallel Operations + +Bulk and parallel operations are intended for application workflows that need to process large entity collections or execute independent database operations across several registered execution contexts. + +DataArc keeps these concerns explicit: + +- `AddBulk(...)` describes a bulk operation for a selected execution context. +- `BuildAsync()` produces the executable command. +- `ExecuteAsync()` uses the command's normal execution path. +- `ExecuteParallelAsync()` allows independent command operations to execute concurrently. +- the returned command result reports the combined execution outcome. + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +## Why bulk operations exist + +Ordinary Entity Framework Core change tracking is appropriate for many application writes. + +Bulk operations become useful when a workflow needs to process large collections and the cost of handling each record individually becomes significant. + +Typical examples include: + +- importing employees from an external source; +- replicating prepared data to another database; +- applying a calculated update to a large entity set; +- seeding or refreshing reference data; +- distributing one prepared collection to several independent targets. + +A bulk operation should still represent a meaningful application task. It should not be used automatically for every write. + +```text +Small or ordinary write + -> normal tracked EF Core operation may be sufficient + +Large prepared collection + -> bulk operation may reduce execution overhead +``` + +The correct choice depends on the number of records, provider capabilities, transaction requirements, generated values, relationships, and the amount of application logic involved. + +## Creating a bulk command + +Inject `ICommandFactory` into the service or workflow that performs the operation: + +```csharp +private readonly ICommandFactory _commandFactory; + +public EmployeeReplicationService( + ICommandFactory commandFactory) +{ + _commandFactory = commandFactory; +} +``` + +Create the command builder: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); +``` + +Select the execution context and add the bulk operation: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +Build and execute the command: + +```csharp +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +The complete shape is: + +```text +ICommandFactory + -> CreateCommandBuilderAsync() + -> UseDbExecutionContext() + -> AddBulk(entities, batchSize) + -> BuildAsync() + -> ExecuteAsync() + -> inspect result +``` + +## The role of the batch size + +`AddBulk(...)` accepts the entity collection and a batch size: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +The batch size controls how the supplied collection is divided for processing by the bulk operation. + +Conceptually: + +```text +100,000 entities + | + +-- batch 1 + +-- batch 2 + +-- batch 3 + +-- ... +``` + +A larger batch size may reduce the number of database round trips, but it can also increase: + +- memory usage; +- transaction duration; +- command size; +- database locking pressure; +- the cost of retrying a failed batch. + +A smaller batch size may reduce the impact of each individual batch, but it can increase total database round trips. + +There is no universal batch size that is correct for every database and workload. + +Choose a starting value, benchmark it with realistic data, and confirm that the database remains stable under the expected production load. + +## Handling empty collections + +Do not build a database command when the workflow has no work to perform. + +```csharp +if (employees.Count == 0) +{ + return 0; +} +``` + +Then build the command only when records exist: + +```csharp +if (employees.Count == 0) +{ + return 0; +} + +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +This keeps the application intent explicit and avoids unnecessary command construction. + +## Single-context bulk execution + +A bulk operation can target one registered execution context: + +```csharp +public async Task ImportEmployeesAsync( + IReadOnlyList employees, + int batchSize) +{ + if (employees.Count == 0) + { + return 0; + } + + var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + var command = await commandBuilder.BuildAsync(); + + var commandResult = await command.ExecuteAsync(); + + if (!commandResult.Success) + { + throw new InvalidOperationException( + "The employee import failed.", + commandResult.Exception); + } + + return commandResult.TotalAffected; +} +``` + +The selected execution context remains visible: + +```text +Employee import + -> IGoogleDbContext + -> AddBulk(...) + -> execute +``` + +## Multi-context bulk commands + +One command builder can contain bulk operations for several execution contexts. + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); +``` + +Each operation remains associated with its own persistence boundary. + +```text +Built command + | + +-- IGoogleDbContext + | -> AddBulk(employees, batchSize) + | + +-- IMicrosoftDbContext + | -> AddBulk(employees, batchSize) + | + +-- IOpenAiDbContext + -> AddBulk(employees, batchSize) +``` + +This does not combine the contexts into one EF Core model. + +It creates one application command containing several explicitly routed operations. + +## Executing through the normal command path + +Use: + +```csharp +var commandResult = await command.ExecuteAsync(); +``` + +when the command should use its normal execution path. + +A complete multi-context example is: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +Use this path when concurrent execution is not required or when the workflow should not treat the operations as independent. + +## Parallel execution + +Use: + +```csharp +var commandResult = await command + .ExecuteParallelAsync(); +``` + +when the built command contains independent operations that are intentionally allowed to run concurrently. + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command + .ExecuteParallelAsync(); +``` + +Conceptually: + +```text +Built command + | + +-- Google bulk operation --------+ + | | + +-- Microsoft bulk operation -----+--> concurrent execution + | | + +-- OpenAI bulk operation --------+ + | + v + structured command result +``` + +## When parallel execution is appropriate + +Parallel execution is appropriate when: + +- each operation targets an independent execution context; +- no operation depends on the result of another; +- the same prepared data can be processed independently; +- the database servers and connection pools can support the added concurrency; +- the workflow can correctly handle combined success or failure information. + +A replication workflow is a typical example: + +```text +Prepared employee collection + | + +-- write to Google database + +-- write to Microsoft database + +-- write to OpenAI database +``` + +If each target is independent, concurrent execution may reduce the total elapsed time. + +## When parallel execution is not appropriate + +Do not use parallel execution when: + +- one operation requires an identifier produced by another; +- one operation must observe committed data from another; +- business ordering is significant; +- the contexts share a resource that cannot be used concurrently; +- the workflow requires one atomic transaction; +- database load is already constrained; +- partial completion would be unacceptable. + +For example: + +```text +Create employee + -> use generated EmployeeId + -> create payroll profile +``` + +The second step depends on the first. + +That is an ordered workflow, not an independent parallel operation. + +## Parallel does not mean transactional + +Parallel execution and transactional execution solve different problems. + +```text +Parallel execution + -> reduce elapsed time for independent operations + +Transaction + -> preserve consistency within a defined transaction boundary +``` + +A command containing operations for separate databases does not automatically become one distributed transaction. + +```text +One DataArc command + != +one atomic transaction across every database +``` + +If one target succeeds and another fails, the application must understand what that means for the workflow. + +Possible responses include: + +- return a failed structured result; +- log the failed target; +- retry the failed operation; +- initiate application-specific compensation; +- mark the operation for manual review. + +See [Transactional Workflows](transactions.md) for the transaction-boundary discussion. + +## Reading once and writing to several targets + +A common pattern is: + +1. read source records through a query pipeline; +2. apply application logic; +3. add bulk operations for several target contexts; +4. execute the independent target operations in parallel. + +```csharp +var employeesQuery = await _queryFactory + .CreateQueryAsync(); + +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => + employee.Salary > salaryThreshold); +``` + +Apply the required transformation: + +```csharp +foreach (var employee in employees) +{ + employee.Salary += + employee.Salary * salaryAdjustmentBaseRate; +} +``` + +Build the target operations: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +Execute them: + +```csharp +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command + .ExecuteParallelAsync(); +``` + +Inspect the result: + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "The parallel bulk operation failed.", + commandResult.Exception); +} + +return commandResult.TotalAffected; +``` + +## Complete parallel bulk example + +```csharp +public async Task ReplicateEmployeeAdjustmentsAsync( + decimal salaryAdjustmentBaseRate, + decimal salaryThreshold, + int batchSize) +{ + var employeesQuery = await _queryFactory + .CreateQueryAsync(); + + var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => + employee.Salary > salaryThreshold); + + if (employees.Count == 0) + { + return 0; + } + + foreach (var employee in employees) + { + employee.Salary += + employee.Salary * + salaryAdjustmentBaseRate; + } + + var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + var command = await commandBuilder.BuildAsync(); + + var commandResult = await command + .ExecuteParallelAsync(); + + if (!commandResult.Success) + { + throw new InvalidOperationException( + "Employee replication failed.", + commandResult.Exception); + } + + return commandResult.TotalAffected; +} +``` + +## Interpreting the combined result + +The current command workflow uses: + +```csharp +commandResult.Success +commandResult.Exception +commandResult.TotalAffected +``` + +Check the result before returning the affected count: + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "Bulk execution failed.", + commandResult.Exception); +} + +return commandResult.TotalAffected; +``` + +Do not assume that a non-zero affected count means that every operation completed successfully. + +Use `Success` as the primary outcome indicator and inspect `Exception` when the operation fails. + +Structured result behavior is covered in [Structured Execution Results](structured-results.md). + +## DbContext concurrency + +An EF Core `DbContext` instance is not intended to execute several database operations concurrently. + +Parallel DataArc operations should therefore remain isolated behind their independently registered execution contexts. + +Do not: + +- share one manually created `DbContext` instance across concurrent tasks; +- start several operations on the same active context; +- reuse tracked entities in ways that create conflicting state; +- assume that every database provider has identical concurrency behavior. + +The parallelism should exist between independent execution operations, not inside one active `DbContext` instance. + +## Connection-pool and database pressure + +Parallel execution can increase pressure on: + +- database connection pools; +- database CPU; +- transaction logs; +- network bandwidth; +- locks and latches; +- application memory. + +A workflow that is faster during a small local test may become less stable under production load. + +Benchmark with: + +- realistic record counts; +- realistic entity sizes; +- the production database provider; +- expected concurrent users or workers; +- representative network latency; +- the actual batch size. + +Measure both elapsed time and operational impact. + +## Avoid accidental nested parallelism + +Be careful when the application already processes several jobs concurrently. + +For example: + +```text +10 background jobs + x +3 parallel execution contexts + = +up to 30 concurrent database operations +``` + +Adding parallelism at several layers can overwhelm the database even when each individual command appears reasonable. + +Choose one deliberate concurrency boundary and control the total number of concurrent workflows at the application level. + +## Failure and retry considerations + +A retry should be based on the semantics of the operation. + +Before retrying, determine whether the operation is: + +- idempotent; +- safe to repeat; +- able to identify already processed records; +- protected by unique constraints; +- using generated values that may change; +- capable of creating duplicates. + +A parallel command may partially complete before a failure is returned. + +Retrying the entire workflow blindly can repeat operations that already succeeded. + +The application may need to: + +- retry only failed targets; +- record a workflow identifier; +- use idempotency keys; +- query the target state before retrying; +- run compensation logic. + +These are application-level decisions and should not be hidden behind a generic retry loop. + +## Bulk operations and relationships + +Bulk operations are best suited to collections whose required key values and relationships are already prepared. + +Before execution, confirm that: + +- required foreign keys are present; +- entity order does not hide a dependency; +- generated keys are not required by another parallel operation; +- navigation properties are not being relied on implicitly; +- the target model matches the supplied entity data. + +When dependent records require generated identifiers, use an ordered workflow. + +## Bulk operations and change tracking + +Bulk operations should not be treated as an automatic replacement for normal tracked EF Core behavior. + +A bulk operation may not provide the same workflow semantics as: + +- loading tracked entities; +- changing individual properties; +- relying on change detection; +- processing entity relationships; +- using application hooks around each entity. + +Use bulk processing when the collection is prepared and the application understands the operation as a set. + +Use normal tracked operations when entity-by-entity state and domain behavior are significant. + +## Recommended decision guide + +Use an ordinary command when: + +- the record count is small; +- change tracking is useful; +- relationships and generated values matter; +- the workflow contains substantial entity-level behavior. + +Use `AddBulk(...)` when: + +- a large collection is already prepared; +- the target context is explicit; +- set-oriented processing is appropriate; +- the provider and database have been tested with the workload. + +Use `ExecuteParallelAsync()` when: + +- several operations are independent; +- each operation has its own execution context; +- order does not matter; +- partial failure can be handled correctly; +- the database environment can support the concurrency. + +Use an ordered or transactional workflow when: + +- operations depend on one another; +- consistency is more important than elapsed time; +- one operation must observe another operation's committed result. + +## Summary + +Bulk and parallel execution follow this shape: + +```text +ICommandFactory + -> CreateCommandBuilderAsync() + -> UseDbExecutionContext() + -> AddBulk(entities, batchSize) + -> BuildAsync() + -> ExecuteAsync() + or ExecuteParallelAsync() + -> inspect structured result +``` + +The main principles are: + +- use bulk processing for large, prepared collections; +- select each execution context explicitly; +- benchmark the batch size with realistic data; +- use parallel execution only for independent operations; +- do not treat parallel execution as a distributed transaction; +- account for partial failure, retries, connection pressure, and idempotency; +- keep workflow consistency more important than raw throughput. + +## Next steps + +Continue with: + +- [Command Pipelines](command-pipelines.md) +- [Transactional Workflows](transactions.md) +- [Structured Execution Results](structured-results.md) diff --git a/docs/command-pipelines.md b/docs/command-pipelines.md new file mode 100644 index 0000000..cb1ee1e --- /dev/null +++ b/docs/command-pipelines.md @@ -0,0 +1,1027 @@ +# Command Pipelines + +Command pipelines provide a structured way to build and execute database write operations through registered DataArc execution contexts. + +A command pipeline follows four steps: + +1. Create a command builder through `ICommandFactory`. +2. Select an execution context with `UseDbExecutionContext()`. +3. Add one or more database operations to the builder. +4. Build and execute the resulting command. + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +The execution context identifies where the operation should run. + +The operation added after the execution-context selection identifies what work should be performed. + +```csharp +.UseDbExecutionContext() +.AddBulk(employees, batchSize); +``` + +## Why command pipelines exist + +Entity Framework Core allows application code to perform write operations directly through a `DbContext`. + +For a simple application with one database boundary, direct `DbContext` usage may be sufficient. + +More complex applications may need to: + +- target several registered `DbContext` instances; +- keep concrete persistence implementations outside the application layer; +- route writes through public execution-context contracts; +- build several related database operations before execution; +- execute operations sequentially or in parallel; +- return one structured result to the calling workflow; +- keep database-routing decisions visible in application code. + +DataArc provides a command-building model for these scenarios. + +```text +Application workflow + -> create command builder + -> select execution context + -> add database operation + -> build command + -> execute command + -> inspect structured result +``` + +The pipeline makes the execution boundary explicit rather than allowing the selected `DbContext` to remain an incidental implementation detail. + +## Injecting the command factory + +Inject `ICommandFactory` into the application service or workflow that needs to build database commands: + +```csharp +private readonly ICommandFactory _commandFactory; + +public SalaryAdjustmentService( + ICommandFactory commandFactory) +{ + _commandFactory = commandFactory; +} +``` + +The application service depends on the DataArc command contract rather than directly resolving a concrete `DbContext`. + +A service can combine `ICommandFactory` with `IQueryFactory` when a workflow needs to read data, apply application logic, and then persist the resulting changes: + +```csharp +private readonly IQueryFactory _queryFactory; +private readonly ICommandFactory _commandFactory; + +public SalaryAdjustmentService( + IQueryFactory queryFactory, + ICommandFactory commandFactory) +{ + _queryFactory = queryFactory; + _commandFactory = commandFactory; +} +``` + +This keeps query and command responsibilities distinct while allowing them to participate in the same application workflow. + +## Creating a command builder + +Create an asynchronous command builder with: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); +``` + +The builder collects the operations that will form the final command. + +At this point, no command has been executed. + +```text +CreateCommandBuilderAsync() + -> command builder created + -> execution contexts selected + -> operations added + -> command built + -> command executed +``` + +This separation allows the application to describe the required work before deciding how the completed command should execute. + +## Selecting an execution context + +Every database operation is associated with an explicitly selected execution context: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +The execution-context type answers: + +> Which registered database boundary should perform this operation? + +A public execution-context contract can be selected: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +A concrete `DbContext` that implements `IExecutionContext` directly can also be selected: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +The command pipeline remains the same in both cases. + +The difference lies in how the execution context was registered. + +For a public contract and hidden concrete implementation: + +```csharp +ef.AddDbExecutionContext< + IGoogleDbContext, + GoogleDbContext>( + options => + options.UseSqlServer(connectionString)); +``` + +For a directly exposed concrete context: + +```csharp +ef.AddDbExecutionContext( + options => + options.UseSqlServer(connectionString)); +``` + +For additional registration guidance, see [Execution Contexts](execution-contexts.md). + +## Adding an operation + +After selecting an execution context, add the required database operation. + +The current demo uses `AddBulk(...)` to add high-volume entity operations: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +The supplied values identify: + +- the entities to process; +- the batch size used by the operation; +- the execution context that should perform the work. + +```text +IGoogleDbContext + -> execution boundary + +employees + -> entities supplied to the operation + +batchSize + -> requested operation batch size +``` + +Bulk behavior and multi-target bulk workflows are covered in more detail in [Bulk and Parallel Operations](bulk-and-parallel-operations.md). + +## Building a command + +After all required operations have been added, build the executable command: + +```csharp +var command = await commandBuilder.BuildAsync(); +``` + +Building and executing are separate stages. + +```text +Command builder + -> describes the work + +BuildAsync() + -> produces the executable command + +ExecuteAsync() + -> runs the built command +``` + +The separation is intentional. + +It allows the application to finish defining the command before selecting its execution mode. + +## Executing a command sequentially + +Execute the built command with: + +```csharp +var commandResult = await command.ExecuteAsync(); +``` + +A complete single-context pipeline is: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +`ExecuteAsync()` is the standard execution path when the operations should be processed through the command’s normal ordered execution flow. + +The application should inspect the returned result rather than assuming that the command succeeded. + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + $"Command execution failed. " + + $"{commandResult.Exception?.Message}"); +} +``` + +## Targeting several execution contexts + +One command builder can contain operations for several registered execution contexts. + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +Each call selects its own execution boundary. + +```text +Command builder + | + +-- IGoogleDbContext + | -> AddBulk(...) + | + +-- IMicrosoftDbContext + | -> AddBulk(...) + | + +-- IOpenAiDbContext + -> AddBulk(...) +``` + +The repeated context selection is intentional. + +It keeps each target visible in the application workflow and prevents the database destination from being hidden inside generic application plumbing. + +The command can then be built normally: + +```csharp +var command = await commandBuilder.BuildAsync(); +``` + +## Sequential multi-context execution + +A command containing several context operations can use the standard execution path: + +```csharp +var commandResult = await command.ExecuteAsync(); +``` + +The complete shape is: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +Use this approach when the workflow requires the command’s normal ordered execution behavior rather than concurrent processing of independent targets. + +## Parallel multi-context execution + +When the command contains independent operations targeting separate execution contexts, it can be executed through: + +```csharp +var commandResult = await command + .ExecuteParallelAsync(); +``` + +For example: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command + .ExecuteParallelAsync(); +``` + +The intended shape is: + +```text +Built command + | + +-- Google operation --------+ + | | + +-- Microsoft operation -----+--> parallel execution + | | + +-- OpenAI operation --------+ + | + v + structured command result +``` + +Parallel execution is appropriate only when the operations are independent. + +Do not use parallel execution merely because several operations exist. + +If one operation depends on the successful output or committed state of another operation, the workflow requires ordered or transactional coordination instead. + +For more detail, see: + +- [Bulk and Parallel Operations](bulk-and-parallel-operations.md) +- [Transactional Workflows](transactions.md) + +## Sequential versus parallel execution + +The execution method expresses how the built command should process its operations. + +### Sequential execution + +```csharp +var commandResult = await command.ExecuteAsync(); +``` + +Use sequential execution when: + +- operations must follow their defined order; +- a later operation conceptually depends on earlier work; +- concurrent execution would make the workflow harder to reason about; +- the application does not require parallel processing. + +### Parallel execution + +```csharp +var commandResult = await command.ExecuteParallelAsync(); +``` + +Use parallel execution when: + +- operations target independent execution contexts; +- the targets do not depend on each other; +- the same prepared data can be written independently; +- concurrency is appropriate for the database and provider configuration; +- the calling workflow can correctly handle a combined result. + +The execution mode should be selected from the workflow’s consistency requirements, not purely from expected performance. + +## Reading before writing + +A common application workflow reads data through a query pipeline and then writes the transformed result through a command pipeline. + +The following example reads employees from a source execution context: + +```csharp +var employeesQuery = await _queryFactory + .CreateQueryAsync(); + +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => + employee.Salary > salaryThreshold); +``` + +The application applies its business operation to the materialized results: + +```csharp +foreach (var employee in employees) +{ + employee.Salary += + employee.Salary * salaryAdjustmentBaseRate; +} +``` + +A command builder is then created: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); +``` + +The transformed employees are added to one or more target execution contexts: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +The command is built and executed: + +```csharp +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command + .ExecuteParallelAsync(); +``` + +The responsibilities remain visible: + +```text +Query pipeline + -> select source execution context + -> read and materialize entities + +Application workflow + -> apply business rules or transformations + +Command pipeline + -> select target execution contexts + -> add database operations + -> build command + -> select execution mode + -> return structured result +``` + +## Complete salary-adjustment example + +The following example demonstrates the complete pattern used by the demo. + +```csharp +public async Task ProcessEmployeeSalaryAdjustmentsAsync( + decimal salaryAdjustmentBaseRate, + decimal salaryThreshold, + int batchSize) +{ + var employeesQuery = await _queryFactory + .CreateQueryAsync(); + + var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => + employee.Salary > salaryThreshold); + + foreach (var employee in employees) + { + employee.Salary += + employee.Salary * + salaryAdjustmentBaseRate; + } + + var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + var command = await commandBuilder.BuildAsync(); + + var commandResult = await command + .ExecuteParallelAsync(); + + if (!commandResult.Success) + { + throw new InvalidOperationException( + "Failed to process salary adjustments. " + + commandResult.Exception?.Message); + } + + return commandResult.TotalAffected; +} +``` + +This workflow: + +1. creates an asynchronous query; +2. reads employees from one source execution context; +3. applies the salary adjustment in application code; +4. creates a command builder; +5. adds operations for three independent target contexts; +6. builds the command; +7. executes the target operations in parallel; +8. checks the structured result; +9. returns the total affected count. + +## Structured command results + +Command execution returns a structured result. + +The current workflow uses the following members: + +```csharp +commandResult.Success +commandResult.Exception +commandResult.TotalAffected +``` + +### Checking success + +```csharp +if (!commandResult.Success) +{ + // Handle failed command execution. +} +``` + +### Inspecting the exception + +```csharp +if (commandResult.Exception is not null) +{ + Console.WriteLine( + commandResult.Exception.Message); +} +``` + +### Reading the affected count + +```csharp +var affectedRecords = + commandResult.TotalAffected; +``` + +A typical result-handling block is: + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + $"Command execution failed. " + + $"{commandResult.Exception?.Message}"); +} + +return commandResult.TotalAffected; +``` + +The calling application remains responsible for deciding what a failed result means to the business workflow. + +For example, it may: + +- throw an application exception; +- return an error response; +- log the failure; +- retry through a higher-level process; +- stop a background workflow; +- compensate through application-specific logic. + +Command-result behavior is covered in more detail in [Structured Execution Results](structured-results.md). + +## Commands and public execution-context contracts + +Command pipelines can target a public execution-context contract while keeping the concrete `DbContext` internal to the persistence layer. + +For example: + +```csharp +public interface IGoogleDbContext : IExecutionContext +{ + DbSet? Employer { get; set; } + + DbSet? Employee { get; set; } +} +``` + +The concrete context remains inside persistence: + +```csharp +internal sealed class GoogleDbContext : + DbContext, + IGoogleDbContext +{ + public GoogleDbContext( + DbContextOptions options) + : base(options) + { + } + + public DbSet? Employer { get; set; } + + public DbSet? Employee { get; set; } +} +``` + +Application code targets the public contract: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +The application does not need to reference `GoogleDbContext` directly. + +DataArc resolves the registered implementation associated with the selected execution-context contract. + +This provides a clear distinction: + +```text +Application layer + -> IGoogleDbContext + +Persistence layer + -> GoogleDbContext + +DataArc registration + -> maps the contract to the implementation +``` + +## Commands across separate databases + +A multi-context command does not imply that all selected contexts share one database. + +Each execution context can represent: + +- a separate database; +- a separate schema; +- a separate application module; +- a bounded persistence context; +- a different EF Core model; +- a different connection string. + +For example: + +```text +IGoogleDbContext + -> SAS_GoogleDb + +IMicrosoftDbContext + -> SAS_MicrosoftDb + +IOpenAiDbContext + -> SAS_OpenAiDb + +ISolidArcDbContext + -> SAS_Db +``` + +The application can build one command containing explicit operations for these boundaries. + +That does not turn the databases into one joined EF Core model. + +Each operation continues to run through its own registered execution context. + +## Command pipelines are not cross-database joins + +A command pipeline coordinates operations. + +It does not create a cross-database entity relationship or a shared `DbContext`. + +```text +One command pipeline + | + +-- operation routed to database A + +-- operation routed to database B + +-- operation routed to database C +``` + +The execution contexts remain separate. + +The command pipeline provides an application-level execution model above those contexts. + +Any required data transformation, consolidation, or mapping should be made explicit in the application workflow before the command operations are added. + +## Command pipelines and repositories + +DataArc command pipelines do not require a repository for every table. + +An application may use a command pipeline directly inside a focused application service: + +```csharp +public async Task ReplicateEmployeesAsync( + IReadOnlyList employees, + int batchSize) +{ + var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + var command = await commandBuilder.BuildAsync(); + + var result = await command.ExecuteAsync(); + + if (!result.Success) + { + throw new InvalidOperationException( + result.Exception?.Message); + } + + return result.TotalAffected; +} +``` + +A repository may also encapsulate a meaningful persistence operation when that improves the application design. + +The choice should follow the application’s boundaries rather than a requirement to wrap every entity in generic repository infrastructure. + +DataArc handles execution routing. + +The application remains responsible for naming and organising meaningful business operations. + +## Keeping command workflows explicit + +Prefer a command pipeline in which each execution boundary can be identified from the application code: + +```csharp +commandBuilder + .UseDbExecutionContext() + .AddBulk(googleEmployees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(microsoftEmployees, batchSize); +``` + +Avoid hiding all context selection behind an unrelated generic helper unless that abstraction expresses a genuine application concept. + +For example, this may make the execution boundary difficult to see: + +```csharp +await _genericPersistenceProcessor + .ProcessEverythingAsync(items); +``` + +The DataArc model is strongest when the workflow clearly communicates: + +- which contexts participate; +- which operations are added; +- when the command is built; +- how the command is executed; +- how the result is handled. + +## Empty input collections + +Application workflows should determine how empty input collections should be handled. + +For example: + +```csharp +if (employees.Count == 0) +{ + return 0; +} +``` + +This can prevent the application from building a command when no database work is required. + +```csharp +if (employees.Count == 0) +{ + return 0; +} + +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +Whether an empty operation is meaningful depends on the application workflow and the selected operation. + +Making the decision before command construction keeps the workflow’s intention clear. + +## Failure handling + +A failed command result should be handled at the appropriate application boundary. + +```csharp +var commandResult = await command.ExecuteAsync(); + +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "The database command failed.", + commandResult.Exception); +} +``` + +For a background worker: + +```csharp +if (!commandResult.Success) +{ + logger.LogError( + commandResult.Exception, + "Employee replication failed."); + + return; +} +``` + +For an application service returning its own result model: + +```csharp +if (!commandResult.Success) +{ + return ApplicationResult.Failure( + commandResult.Exception?.Message ?? + "Command execution failed."); +} +``` + +DataArc provides the execution outcome. + +The application decides how that outcome should be represented to: + +- an API caller; +- a user interface; +- a background worker; +- a scheduled process; +- an event consumer; +- another application service. + +## Transaction boundaries + +Building several operations into one command does not by itself prove that all participating execution contexts share one atomic database transaction. + +This distinction is especially important when the command targets separate databases. + +```text +One application command + != +one distributed database transaction +``` + +Transaction behavior depends on: + +- the selected execution contexts; +- the underlying database connections; +- the execution mode; +- the configured transaction workflow; +- the capabilities of the EF Core provider and database platform. + +Do not assume that parallel operations across separate databases are atomically committed as one unit. + +For transaction-specific guidance, see [Transactional Workflows](transactions.md). + +## Choosing an execution mode + +Select the execution mode after considering: + +- whether operations depend on one another; +- whether each target is independent; +- whether ordering is significant; +- whether concurrent database work is appropriate; +- whether the workflow requires transactional coordination; +- how partial failures should be handled. + +Use: + +```csharp +await command.ExecuteAsync(); +``` + +for the command’s standard sequential execution path. + +Use: + +```csharp +await command.ExecuteParallelAsync(); +``` + +for independent operations that are intentionally allowed to execute concurrently. + +Do not select parallel execution solely because it appears faster. + +Correct workflow semantics take priority over throughput. + +## Recommended application shape + +A focused application workflow can use command pipelines without becoming a command-class-per-operation architecture. + +```text +SalaryAdjustmentService + | + +-- IQueryFactory + | -> read source employees + | + +-- application logic + | -> adjust salaries + | + +-- ICommandFactory + -> build target operations + -> execute command + -> inspect result +``` + +The workflow remains named after its application purpose. + +The DataArc command pipeline provides the persistence execution model inside that workflow. + +## Summary + +A DataArc command pipeline consists of: + +```text +ICommandFactory + -> CreateCommandBuilderAsync() + -> UseDbExecutionContext() + -> add operation + -> BuildAsync() + -> ExecuteAsync() + or ExecuteParallelAsync() + -> inspect structured result +``` + +The core responsibilities are: + +- `ICommandFactory` creates the command builder; +- the command builder collects database operations; +- `UseDbExecutionContext()` selects each database boundary; +- `BuildAsync()` produces the executable command; +- `ExecuteAsync()` executes through the standard ordered path; +- `ExecuteParallelAsync()` executes independent operations concurrently; +- the structured result reports success, failure information, and affected records. + +Command pipelines are most useful when application workflows need explicit control over writes across one or more registered EF Core execution contexts. + +## Next steps + +Continue with: + +- [Bulk and Parallel Operations](bulk-and-parallel-operations.md) +- [Transactional Workflows](transactions.md) +- [Structured Execution Results](structured-results.md) \ No newline at end of file diff --git a/docs/compatibility.md b/docs/compatibility.md new file mode 100644 index 0000000..bef9076 --- /dev/null +++ b/docs/compatibility.md @@ -0,0 +1,339 @@ +# Compatibility + +DataArc.EntityFrameworkCore is designed to support modern .NET and Entity Framework Core applications across multiple framework versions. + +The current release multi-targets .NET and Entity Framework Core versions 6 through 10. + +## Supported framework versions + +| .NET target | Entity Framework Core | +|---|---| +| .NET 6 | EF Core 6 | +| .NET 7 | EF Core 7 | +| .NET 8 | EF Core 8 | +| .NET 9 | EF Core 9 | +| .NET 10 | EF Core 10 | + +Applications should use the DataArc target that matches their own .NET and Entity Framework Core version. + +For example: + +```xml +net8.0 +``` + +should use the .NET 8-compatible DataArc assemblies and EF Core 8 packages. + +A .NET 10 application should similarly use EF Core 10: + +```xml +net10.0 +``` + +## Multi-targeted applications + +DataArc.EntityFrameworkCore packages contain target-specific assemblies for supported .NET versions. + +A project may target one framework: + +```xml + + net10.0 + +``` + +or multiple frameworks: + +```xml + + net6.0;net7.0;net8.0;net9.0;net10.0 + +``` + +When a project is multi-targeted, the .NET SDK selects the appropriate DataArc assembly for each target framework during restore and build. + +## Running a multi-targeted project + +When a project targets several frameworks, specify the framework when running it: + +```bash +dotnet run --framework net8.0 +``` + +or: + +```bash +dotnet run -f net10.0 +``` + +This is especially important when the machine only has one of the targeted SDKs or runtimes installed. + +For example, a project targeting .NET 6 through .NET 10 may not run without an explicit framework selection on a machine that only has the .NET 10 runtime available: + +```bash +dotnet run -f net10.0 +``` + +The same applies to other commands: + +```bash +dotnet build -f net10.0 +``` + +```bash +dotnet test -f net10.0 +``` + +```bash +dotnet publish -c Release -f net10.0 +``` + +## Entity Framework Core version alignment + +The application's Entity Framework Core packages should align with its target framework and with the DataArc assembly selected for that target. + +For example, a .NET 8 project would typically reference EF Core 8 packages: + +```xml + + + + + +``` + +A .NET 10 project would use the corresponding EF Core 10 packages: + +```xml + + + + + +``` + +Avoid mixing incompatible major versions of: + +- the target framework; +- Entity Framework Core; +- the selected database provider; +- Entity Framework Core tooling. + +For example, an EF Core 8 runtime package should normally be used with the EF Core 8 SQL Server provider and EF Core 8 design package. + +## Database providers + +The public DataArc.EntityFrameworkCore demo uses Microsoft SQL Server: + +```csharp +services.ConfigureDataArc(provider => +{ + provider.UseEntityFrameworkCore(entityFrameworkCore => + { + entityFrameworkCore.AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString( + "ApplicationDatabase")); + }); + }); +}); +``` + +DataArc registers Entity Framework Core execution contexts through standard `DbContextOptions` configuration. + +The database provider is configured in the same registration callback used by a normal Entity Framework Core application: + +```csharp +options.UseSqlServer(connectionString); +``` + +Provider-specific behavior remains the responsibility of Entity Framework Core and the selected provider. + +Capabilities such as: + +- SQL generation; +- transaction behavior; +- stored procedure support; +- batching; +- parameter handling; +- database type mapping; +- provider-specific limitations; + +may differ between providers and versions. + +The current public examples and compatibility guidance are based on SQL Server. + +## SQL Server + +The public demo uses: + +```xml + +``` + +and configures contexts with: + +```csharp +options.UseSqlServer(connectionString); +``` + +DataArc features demonstrated with SQL Server include: + +- command pipelines; +- query pipelines; +- bulk and parallel operations; +- transactional workflows; +- multiple execution contexts; +- database generation; +- script generation; +- stored procedure execution. + +Applications should test provider-specific behavior against the database and EF Core version they intend to use. + +## EF Core tools + +Entity Framework Core command-line tools should normally match the major version of the EF Core packages used by the project. + +Check the installed tool version with: + +```bash +dotnet ef --version +``` + +Install or update a version that matches the project: + +```bash +dotnet tool install --global dotnet-ef --version 8.* +``` + +or: + +```bash +dotnet tool update --global dotnet-ef --version 10.* +``` + +Projects that use design-time EF Core operations may also require: + +```xml + + all + + runtime; + build; + native; + contentfiles; + analyzers; + buildtransitive + + +``` + +Use the version that matches the project's EF Core major version. + +## C# language versions + +The available C# language version is determined by the selected .NET SDK and project configuration. + +DataArc does not require application code to use the newest available C# syntax. + +Execution-context contracts can use a conventional declaration: + +```csharp +public interface IApplicationDbContext : IExecutionContext +{ +} +``` + +Concrete contexts can also use standard syntax supported by the target framework: + +```csharp +public sealed class ApplicationDbContext + : DbContext, IExecutionContext +{ + public ApplicationDbContext( + DbContextOptions options) + : base(options) + { + } +} +``` + +Applications may use newer C# syntax where supported by their selected SDK. + +## Operating systems + +DataArc.EntityFrameworkCore runs within the supported .NET and Entity Framework Core runtime environments. + +The application must also satisfy the requirements of: + +- the selected .NET runtime; +- the selected Entity Framework Core provider; +- the target database; +- any native or operating-system dependencies introduced by the application. + +The public demo can be built and run through the standard .NET CLI: + +```bash +dotnet restore +dotnet build +dotnet run -f net10.0 +``` + +Database connectivity remains dependent on the configured provider, connection string, network access, authentication method, and database server. + +## Dependency versioning + +Keep related DataArc packages on the same release version. + +For example: + +```xml + + + + + + + +``` + +Mixing different DataArc package versions may introduce incompatible contracts or runtime behavior. + +When upgrading, update the related DataArc packages together and then rebuild the application. + +## Current compatibility scope + +The current release is intended for: + +- .NET 6 through .NET 10; +- EF Core 6 through EF Core 10; +- single-targeted or multi-targeted applications; +- public execution-context contracts; +- direct concrete `DbContext` execution contexts; +- SQL Server-based applications and demos; +- applications with one or multiple registered execution contexts; +- contexts targeting one shared database or separate databases. + +The public demo is the reference implementation for the currently documented setup. + +Where provider or framework behavior differs, the behavior of the selected .NET runtime, Entity Framework Core version, and database provider should be treated as authoritative. \ No newline at end of file diff --git a/docs/database-generation.md b/docs/database-generation.md new file mode 100644 index 0000000..aad8f14 --- /dev/null +++ b/docs/database-generation.md @@ -0,0 +1,835 @@ +# Database and Script Generation + +DataArc.EntityFrameworkCore includes database-building support for generating database creation and deletion scripts from one or more EF Core `DbContext` models. + +The database builder is created through `IDatabaseFactory`: + +```csharp +private readonly IDatabaseFactory _databaseFactory; +``` + +A database definition is then composed by: + +1. creating a database builder; +2. including one or more concrete `DbContext` types; +3. building the database operation; +4. choosing whether scripts should be generated; +5. choosing whether the generated changes should be applied; +6. executing the create or drop operation. + +```csharp +var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); + +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + +database.ExecuteCreate(); +``` + +## Why database generation exists + +EF Core applications commonly use migrations or `EnsureCreated()` to create database structures. + +DataArc's database builder provides another option for applications that need: + +- database creation from the current EF Core model; +- database deletion through the same builder model; +- reviewable SQL script output; +- several database contexts handled through a consistent API; +- repeatable database setup for demos, development, testing, or onboarding; +- model-based database generation without requiring migration files. + +```text +EF Core DbContext model + -> DataArc database builder + -> generated SQL script + -> optionally apply database changes +``` + +The generated scripts make the resulting database operations visible and reviewable before or after execution. + +## Injecting the database factory + +Inject `IDatabaseFactory` into the database creator or persistence component responsible for database generation: + +```csharp +private readonly IDatabaseFactory _databaseFactory; + +public FinanceDbCreator( + IDatabaseFactory databaseFactory) +{ + _databaseFactory = databaseFactory; +} +``` + +The factory provides the entry point for creating a database builder: + +```csharp +var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); +``` + +## Including a DbContext + +Use `IncludeDbContext()` to add a concrete EF Core context model to the database definition: + +```csharp +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); +``` + +The included context provides the EF Core model used to generate the database structure. + +```text +FinanceDbContext + -> schemas + -> tables + -> columns + -> keys + -> constraints +``` + +The database builder uses the concrete `DbContext` type because database generation requires access to its complete EF Core model and provider configuration. + +## Including several DbContext models + +A database builder can include more than one context when several EF Core models contribute to the same database-generation workflow. + +```csharp +var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); + +var database = databaseBuilder + .IncludeDbContext() + .IncludeDbContext() + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); +``` + +Use this only when the included models are intended to participate in the same generated database operation. + +Do not combine unrelated contexts merely because they are registered in the same application. + +Before including several contexts, confirm: + +- whether they target the same physical database; +- whether their schemas and object names are compatible; +- whether duplicate table definitions exist; +- whether the provider configuration is consistent; +- whether the generated script reflects the intended deployment boundary. + +For separate databases, build and execute separate database definitions. + +## Building the database operation + +Call `Build(...)` after all required contexts have been included: + +```csharp +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); +``` + +The current builder accepts: + +```csharp +generateScripts +applyChanges +``` + +These flags control script output and execution behavior. + +## Generating scripts + +Set: + +```csharp +generateScripts: true +``` + +to generate SQL scripts for the database operation: + +```csharp +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: false); +``` + +This is useful when the SQL should be reviewed or deployed separately. + +```text +DbContext model + -> build database operation + -> generate SQL files + -> review scripts + -> apply through deployment process +``` + +Generated scripts can be inspected for: + +- database creation; +- schema creation; +- table creation; +- primary keys; +- foreign keys; +- indexes; +- constraints; +- database deletion statements. + +## Applying changes + +Set: + +```csharp +applyChanges: true +``` + +when the database operation should execute the generated changes: + +```csharp +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + +database.ExecuteCreate(); +``` + +This configuration both generates the scripts and applies the database creation operation. + +For script-only generation: + +```csharp +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: false); + +database.ExecuteCreate(); +``` + +The exact runtime behavior should be validated in the target environment before using database generation in an automated production deployment. + +## Creating a database + +Use `ExecuteCreate()` to execute the create operation: + +```csharp +var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); + +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + +database.ExecuteCreate(); +``` + +The operation generates the database structure from the included EF Core model. + +Conceptually: + +```text +ExecuteCreate() + | + +-- create database + +-- create schemas + +-- create tables + +-- create keys and constraints +``` + +The exact SQL depends on: + +- the EF Core provider; +- the configured connection string; +- the included context model; +- entity mappings; +- schema mappings; +- column types; +- key and relationship configuration. + +## Deleting a database + +Use `ExecuteDrop()` to execute the database deletion operation: + +```csharp +var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); + +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + +database.ExecuteDrop(); +``` + +Database deletion is destructive. + +Use it only in an environment where the target database is known and deletion is intentional. + +Typical uses include: + +- local development; +- integration testing; +- repeatable demo setup; +- disposable test environments. + +Do not run automated database deletion against a production connection string. + +## Database creator pattern + +A project can place database creation behind a focused creator interface. + +```csharp +public interface IFinanceDbCreator +{ + bool EnsureCreated(); + + bool EnsureDeleted(); +} +``` + +The implementation uses `IDatabaseFactory`: + +```csharp +public sealed class FinanceDbCreator : + IFinanceDbCreator +{ + private readonly IDatabaseFactory _databaseFactory; + + public FinanceDbCreator( + IDatabaseFactory databaseFactory) + { + _databaseFactory = databaseFactory; + } + + public bool EnsureCreated() + { + var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); + + var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + + database.ExecuteCreate(); + + return true; + } + + public bool EnsureDeleted() + { + var databaseBuilder = + _databaseFactory.CreateDatabaseBuilder(); + + var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + + database.ExecuteDrop(); + + return true; + } +} +``` + +This keeps database lifecycle logic inside the persistence layer. + +The application or demo host can then depend on the creator contract: + +```csharp +_financeDbCreator.EnsureDeleted(); +_financeDbCreator.EnsureCreated(); +``` + +## Separate database creators + +For applications with several separate databases, use one creator per database boundary. + +```text +FinanceDbCreator +HrDbCreator +ItDbCreator +OperationsDbCreator +``` + +Each creator includes the context associated with its own database: + +```csharp +var database = databaseBuilder + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); +``` + +This keeps the target database explicit and prevents unrelated database models from being combined accidentally. + +## Complete multi-database setup example + +```csharp +public sealed class DemoDatabaseInitializer +{ + private readonly IFinanceDbCreator _finance; + private readonly IHrDbCreator _hr; + private readonly IItDbCreator _it; + private readonly IOperationsDbCreator _operations; + + public DemoDatabaseInitializer( + IFinanceDbCreator finance, + IHrDbCreator hr, + IItDbCreator it, + IOperationsDbCreator operations) + { + _finance = finance; + _hr = hr; + _it = it; + _operations = operations; + } + + public void RecreateDatabases() + { + _finance.EnsureDeleted(); + _hr.EnsureDeleted(); + _it.EnsureDeleted(); + _operations.EnsureDeleted(); + + _finance.EnsureCreated(); + _hr.EnsureCreated(); + _it.EnsureCreated(); + _operations.EnsureCreated(); + } +} +``` + +The workflow is explicit: + +```text +Delete Finance database +Delete HR database +Delete IT database +Delete Operations database + +Create Finance database +Create HR database +Create IT database +Create Operations database +``` + +This pattern is suitable for disposable demo or integration-test environments. + +## Generated script location + +Generated SQL scripts are written beneath the application output directory in a `Scripts` folder. + +Example: + +```text +bin/Debug/net8.0/Scripts +``` + +The target framework segment depends on the application being built. + +For a .NET 10 application, the path may resemble: + +```text +bin/Debug/net10.0/Scripts +``` + +The precise output location also depends on the selected build configuration and host project. + +Typical configurations include: + +```text +bin/Debug//Scripts +bin/Release//Scripts +``` + +## Example generated SQL + +A generated creation script can contain SQL such as: + +```sql +CREATE DATABASE [FinanceDb]; +GO + +USE [FinanceDb]; +GO + +IF SCHEMA_ID('employees') IS NULL + EXEC('CREATE SCHEMA [employees]'); +GO + +CREATE TABLE [employees].[Employees] ( + [Id] int IDENTITY(1,1) NOT NULL, + [EmployeeName] varchar(50) NULL, + [EmployeeSalary] decimal(18,2) NOT NULL, + CONSTRAINT [PK_Employees] + PRIMARY KEY ([Id]) +); +GO +``` + +The actual generated SQL is determined by the EF Core model and provider. + +Do not copy this example as a replacement for the generated output of the application's own context model. + +## Script review + +Generated scripts should be reviewed before production use. + +Check: + +- the target database name; +- schema names; +- table names; +- column types and lengths; +- decimal precision and scale; +- nullable columns; +- primary keys; +- foreign keys; +- cascade-delete behavior; +- indexes; +- unique constraints; +- destructive statements; +- provider-specific SQL. + +A script can be technically valid while still representing an incorrect model configuration. + +Database generation reflects the model it receives. + +It does not decide whether that model is correct for the business. + +## Database generation and migrations + +DataArc database generation does not require EF Core migration files for the demonstrated create-and-drop workflow. + +```text +Current DbContext model + -> generate complete database script +``` + +EF Core migrations solve a different problem: + +```text +Previous schema version + -> ordered migration changes + -> new schema version +``` + +Use model-based database generation when: + +- creating a new database; +- recreating a disposable environment; +- producing a complete baseline script; +- running demos or integration tests; +- onboarding a new empty database from the current model. + +Use migrations when: + +- upgrading an existing database that contains data; +- applying incremental schema changes; +- preserving a schema-change history; +- supporting controlled version-to-version deployment. + +Do not drop and recreate a production database merely to avoid migrations. + +## Baseline generation + +A generated create script can be used as a reviewable baseline for a new environment. + +```text +Current model + -> generate create script + -> review + -> approve + -> deploy to empty database +``` + +This can be useful when a database administrator requires SQL scripts rather than direct application-driven database creation. + +In that case, use: + +```csharp +.Build( + generateScripts: true, + applyChanges: false); +``` + +and deploy the reviewed script through the organisation's database-release process. + +## Development and test environments + +Automatic create-and-drop workflows are useful in disposable environments: + +```csharp +database.ExecuteDrop(); +database.ExecuteCreate(); +``` + +Suitable environments include: + +- developer workstations; +- local demos; +- automated integration tests; +- short-lived test databases; +- ephemeral CI environments. + +Use environment checks before allowing destructive operations: + +```csharp +if (!hostEnvironment.IsDevelopment()) +{ + throw new InvalidOperationException( + "Database recreation is allowed only in Development."); +} +``` + +A stronger implementation can also validate the target database name before deletion. + +## Production safety + +Database generation should be handled conservatively in production. + +Recommended controls include: + +- generate scripts without applying them; +- review scripts through source control or release tooling; +- require database-owner approval; +- back up existing databases; +- verify the target connection string; +- run under a least-privileged account; +- separate create permissions from normal application permissions; +- prohibit automated `ExecuteDrop()` calls; +- record script hashes and deployment results. + +A production application account usually should not have permission to create or delete databases. + +## Provider-specific behavior + +The generated SQL depends on the configured EF Core provider. + +For example: + +```csharp +options.UseSqlServer(connectionString); +``` + +produces SQL Server-specific output. + +Other providers may differ in: + +- database creation support; +- schema support; +- identifier quoting; +- data types; +- identity columns; +- computed columns; +- default constraints; +- index syntax; +- deletion behavior. + +Validate generated scripts against the exact provider and database version used by the target environment. + +## Model configuration affects generated scripts + +The builder generates scripts from the EF Core model. + +For example: + +```csharp +modelBuilder + .Entity() + .Property(employee => employee.EmployeeSalary) + .HasPrecision(18, 2); +``` + +affects the generated column definition. + +Likewise: + +```csharp +modelBuilder + .Entity() + .ToTable( + "Employees", + "employees"); +``` + +affects the generated schema and table names. + +Review EF Core model warnings before trusting generated output. + +Warnings about unspecified decimal precision, required relationships, or provider compatibility can indicate that the generated schema will not match the intended design. + +## Registration requirements + +The concrete contexts used by the database builder must have valid EF Core configuration. + +For example: + +```csharp +services.ConfigureDataArc(dataArc => +{ + dataArc.UseEntityFrameworkCore(ef => + { + ef.AddDbExecutionContext< + IFinanceDbContext, + FinanceDbContext>( + options => + options.UseSqlServer( + financeConnectionString)); + }); +}); +``` + +The database builder uses the configured context model and provider. + +Verify that: + +- the connection string targets the intended database; +- the provider package is installed; +- the context constructor accepts the correct options; +- the context model builds successfully; +- required services are registered. + +## Recommended creator implementation + +A focused creator keeps the operation clear: + +```csharp +public sealed class FinanceDbCreator : + IFinanceDbCreator +{ + private readonly IDatabaseFactory _databaseFactory; + + public FinanceDbCreator( + IDatabaseFactory databaseFactory) + { + _databaseFactory = databaseFactory; + } + + public bool EnsureCreated() + { + var database = _databaseFactory + .CreateDatabaseBuilder() + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + + database.ExecuteCreate(); + + return true; + } + + public bool EnsureDeleted() + { + var database = _databaseFactory + .CreateDatabaseBuilder() + .IncludeDbContext() + .Build( + generateScripts: true, + applyChanges: true); + + database.ExecuteDrop(); + + return true; + } +} +``` + +The application can create one implementation for each separate database boundary. + +## Choosing the correct mode + +Generate and apply changes: + +```csharp +.Build( + generateScripts: true, + applyChanges: true); +``` + +Use this for controlled disposable environments where automatic creation is intended. + +Generate scripts without applying changes: + +```csharp +.Build( + generateScripts: true, + applyChanges: false); +``` + +Use this when scripts must be reviewed or deployed separately. + +Apply changes without retaining scripts: + +```csharp +.Build( + generateScripts: false, + applyChanges: true); +``` + +Use this only when direct application is intentional and reviewable script output is not required. + +The preferred production posture is normally script generation and review rather than granting the running application broad database-creation permissions. + +## Summary + +The DataArc database-generation flow is: + +```text +IDatabaseFactory + -> CreateDatabaseBuilder() + -> IncludeDbContext() + -> Build( + generateScripts, + applyChanges) + -> ExecuteCreate() + or ExecuteDrop() +``` + +The central principles are: + +- database structure is generated from the current EF Core model; +- one or more concrete contexts can be included; +- scripts can be generated without immediately applying changes; +- `ExecuteCreate()` creates the generated database structure; +- `ExecuteDrop()` performs destructive database deletion; +- separate databases should normally have separate creator implementations; +- generated scripts are written beneath the application's `Scripts` output folder; +- complete model generation is not a replacement for incremental production migrations; +- destructive operations should be restricted to disposable environments; +- generated SQL must be reviewed against the intended provider and schema. + +## Next steps + +Continue with: + +- [Execution Contexts](execution-contexts.md) +- [Transactional Workflows](transactions.md) +- [Trial and Licensing](licensing-and-trial.md) diff --git a/docs/docfx.json b/docs/docfx.json new file mode 100644 index 0000000..51b9297 --- /dev/null +++ b/docs/docfx.json @@ -0,0 +1,33 @@ +{ + "$schema": "https://raw.githubusercontent.com/dotnet/docfx/main/schemas/docfx.schema.json", + "build": { + "content": [ + { + "files": [ + "**/*.{md,yml}" + ], + "exclude": [ + "_site/**" + ] + } + ], + "resource": [ + { + "files": [ + "images/**" + ] + } + ], + "output": "_site", + "template": [ + "default", + "modern" + ], + "globalMetadata": { + "_appName": "DataArc.EntityFrameworkCore", + "_appTitle": "DataArc.EntityFrameworkCore", + "_enableSearch": true, + "pdf": false + } + } +} \ No newline at end of file diff --git a/docs/execution-contexts.md b/docs/execution-contexts.md new file mode 100644 index 0000000..de0724a --- /dev/null +++ b/docs/execution-contexts.md @@ -0,0 +1,583 @@ +# Execution Contexts + +Execution contexts define the database boundaries used by DataArc.EntityFrameworkCore. + +They allow application workflows to select a specific persistence boundary explicitly before executing a command or query. + +IExecutionContext is not intended to be another repository-style abstraction over Entity Framework Core. It provides DataArc with a stable execution boundary that can be used to build commands, route queries, coordinate parallel operations across multiple registered contexts, and support transactional workflows. + +The execution-context contract also separates application workflows from the concrete provider registered behind that boundary. In the current release, these contexts are backed by Entity Framework Core. The same execution model can support additional persistence providers in future without requiring workflows to abandon the execution-context patterns they already use. + +DataArc supports two execution-context styles: + +1. A public execution-context contract backed by a concrete `DbContext`. +2. A concrete `DbContext` that directly implements `IExecutionContext`. + +Hiding the concrete `DbContext` is optional. DataArc supports both approaches so that applications can choose the boundary that fits their architecture. + +## Why execution contexts exist + +Entity Framework Core applications can access a `DbContext` directly and implement reads and writes in many different ways. + +That flexibility is useful, but it can also result in different developers or modules creating their own persistence patterns: + +- direct `DbSet` access; +- repository abstractions; +- custom query services; +- handler-specific persistence logic; +- locally defined transaction patterns; +- different approaches to projections, paging, and bulk operations. + +DataArc introduces a consistent command and query contract over registered Entity Framework Core contexts. + +Application workflows: + +1. obtain a command or query from a DataArc factory; +2. select an execution context explicitly; +3. execute an operation through the DataArc command or query API. + +For example: + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.Surname == "Doe"); +``` + +The execution context identifies the database boundary, while `ReadWhere` identifies the model and operation being executed. + +This keeps developers using the same query and command patterns across an application instead of each feature introducing its own Entity Framework Core read and write implementation. + +## Public execution-context contracts + +A public execution-context contract can be used when application workflows should depend on an abstraction rather than a concrete Entity Framework Core context. + +The contract implements `IExecutionContext`: + +```csharp +using DataArc.Core.Abstractions; + +public interface IHrDbContext : IExecutionContext +{ + DbSet? Employee { get; set; } +} +``` + +The concrete Entity Framework Core context implements that contract: + +```csharp +using Microsoft.EntityFrameworkCore; + +internal sealed class HrDbContext : DbContext, IHrDbContext +{ + public HrDbContext(DbContextOptions options) + : base(options) + { + } + + public DbSet Employees => Set(); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.HasKey(employee => employee.Id); + + entity.Property(employee => employee.Name) + .HasMaxLength(200) + .IsRequired(); + }); + } +} +``` + +Register the public contract and its concrete implementation with: + +```csharp +AddDbExecutionContext(...) +``` + +For example: + +```csharp +services.ConfigureDataArc(provider => +{ + provider.UseEntityFrameworkCore(entityFrameworkCore => + { + entityFrameworkCore.AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("HrDatabase")); + }); + }); +}); +``` + +Application workflows can then select the public execution boundary without referencing `HrDbContext`: + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.IsActive); +``` + +This style is useful when: + +- the application layer should not reference the concrete `DbContext`; +- the concrete context should remain internal to an infrastructure project; +- the execution boundary represents an application module or domain; +- multiple workflows should share a stable persistence contract; +- Entity Framework Core implementation details should remain behind an abstraction. + +The execution-context contract does not need to expose `DbSet` properties. Its purpose is to identify the registered execution boundary used by DataArc. + +## Direct concrete DbContext usage + +A concrete `DbContext` can also act as the execution context directly. + +The context implements `IExecutionContext`: + +```csharp +using DataArc.Core.Abstractions; +using Microsoft.EntityFrameworkCore; + +public sealed class ApplicationDbContext + : DbContext, IExecutionContext +{ + public ApplicationDbContext( + DbContextOptions options) + : base(options) + { + } + + public DbSet Customers => Set(); +} +``` + +Register it with: + +```csharp +AddDbExecutionContext(...) +``` + +For example: + +```csharp +services.ConfigureDataArc(provider => +{ + provider.UseEntityFrameworkCore(entityFrameworkCore => + { + entityFrameworkCore.AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("ApplicationDatabase")); + }); + }); +}); +``` + +The concrete context is then selected directly: + +```csharp +var customers = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + customer => customer.IsActive); +``` + +This style is useful when: + +- the application already references its concrete `DbContext`; +- hiding the context would not create a meaningful architectural boundary; +- the solution is intentionally lightweight; +- a separate execution-context interface would add unnecessary ceremony. + +Both registration styles use the same DataArc command and query contracts. + +## Explicit execution routing + +DataArc does not infer which context should execute a command or query. + +The workflow selects the required boundary explicitly with: + +```csharp +UseDbExecutionContext() +``` + +For example: + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadAll(); +``` + +The execution context and entity type are specified separately: + +```csharp +UseDbExecutionContext() +ReadAll() +``` + +The execution-context type answers: + +> Which registered database boundary should execute this operation? + +The entity type answers: + +> Which model should the operation read or modify? + +Keeping these concerns separate makes the selected persistence boundary visible while retaining a consistent operation contract. + +## Synchronous queries + +Create a synchronous query with `CreateQuery()`: + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.Surname == "Doe"); +``` + +The entity type is supplied to the read operation: + +```csharp +ReadWhere(...) +``` + +Other synchronous query operations follow the same pattern, such as: + +```csharp +var employee = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadOne(employeeId); +``` + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadAll(); +``` + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadPaged( + startIndex: 0, + recordCount: 50); +``` + +## Asynchronous queries + +Create an asynchronous query with `CreateQueryAsync()`: + +```csharp +var employeesQuery = await _queryFactory.CreateQueryAsync(); + +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold); +``` + +The asynchronous factory and operation are both awaited: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var results = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +The asynchronous query contract returns materialized results such as: + +```csharp +Task +``` + +or: + +```csharp +Task> +``` + +depending on the selected operation. + +For example: + +```csharp +var employee = await employeesQuery + .UseDbExecutionContext() + .ReadOneAsync(employeeId); +``` + +```csharp +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadPagedAsync( + startIndex: 0, + recordCount: 50); +``` + +```csharp +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.IsActive); +``` + +## Query projections + +DataArc query operations support projections without exposing the underlying `IQueryable`. + +For example: + +```csharp +var employeeSummaries = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.IsActive, + employee => new EmployeeSummaryDto + { + Id = employee.Id, + Name = employee.Name, + Salary = employee.Salary + }); +``` + +The first generic type identifies the persisted entity: + +```csharp +Employee +``` + +The second identifies the projected result: + +```csharp +EmployeeSummaryDto +``` + +The projection remains part of the DataArc query contract and is translated through Entity Framework Core. + +This allows workflows to select only the data they require while keeping the query shape consistent across the application. + +## Why DataArc does not expose IQueryable directly + +The current DataArc query contract executes reads through operations such as: + +```csharp +ReadOne(...) +ReadAll() +ReadPaged(...) +ReadWhere(...) +ReadWhere(...) +``` + +and their asynchronous equivalents. + +DataArc does not currently return an `IQueryable` for application code to compose with arbitrary Entity Framework Core operations. + +A direct `IQueryable` API could feel more familiar to Entity Framework Core developers: + +```csharp +var customers = await queryFactory + .CreateQuery() + .UseDbExecutionContext() + .Where(customer => customer.IsActive) + .ToListAsync(); +``` + +However, that is not the current DataArc contract. + +The current API deliberately keeps filtering, projection, paging, and execution behind a shared set of command and query operations. + +This provides a consistent pattern across teams and modules: + +```csharp +UseDbExecutionContext() + .ReadWhereAsync(...) +``` + +rather than allowing each workflow to define its own persistence composition and terminal execution pattern. + +A future version could introduce a more directly composable Entity Framework Core query surface, but applications should use the documented DataArc contracts for the current release. + +## Multiple execution contexts + +An application can register multiple execution contexts: + +```csharp +services.ConfigureDataArc(provider => +{ + provider.UseEntityFrameworkCore(entityFrameworkCore => + { + entityFrameworkCore + .AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("HrDatabase")); + }) + .AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("FinanceDatabase")); + }); + }); +}); +``` + +Each workflow selects the context required for its operation: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.IsActive); +``` + +```csharp +var payrollRecords = await query + .UseDbExecutionContext() + .ReadWhereAsync( + record => record.EmployeeId == employeeId); +``` + +The execution-context type acts as: + +- the registration key used by DataArc; +- the selected Entity Framework Core boundary; +- an explicit architectural marker in the workflow. + +## One database with multiple contexts + +Several execution contexts may target the same physical database. + +For example, an application may contain separate HR, finance, IT, and operations contexts while storing their tables in one SQL Server database: + +```csharp +entityFrameworkCore + .AddDbExecutionContext( + options => + { + options.UseSqlServer(sharedConnectionString); + }) + .AddDbExecutionContext( + options => + { + options.UseSqlServer(sharedConnectionString); + }) + .AddDbExecutionContext( + options => + { + options.UseSqlServer(sharedConnectionString); + }); +``` + +The physical connection is shared, but workflows continue to select the relevant logical boundary explicitly: + +```csharp +var employee = await query + .UseDbExecutionContext() + .ReadOneAsync(employeeId); +``` + +```csharp +var payroll = await query + .UseDbExecutionContext() + .ReadWhereAsync( + record => record.EmployeeId == employeeId); +``` + +The shared database does not remove the architectural distinction between the registered contexts. + +## Separate databases + +Execution contexts may also target separate physical databases: + +```csharp +entityFrameworkCore + .AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("GoogleDatabase")); + }) + .AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("MicrosoftDatabase")); + }) + .AddDbExecutionContext( + options => + { + options.UseSqlServer( + configuration.GetConnectionString("OpenAiDatabase")); + }); +``` + +Each operation routes to its intended database: + +```csharp +var googleEmployees = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +```csharp +var microsoftEmployees = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +```csharp +var openAiEmployees = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +No global or ambient context selector is required. + +The selected execution context is visible directly in the workflow. + +## Choosing a registration style + +Use a public execution-context contract when the abstraction represents a useful application or module boundary: + +```csharp +AddDbExecutionContext(...) +``` + +Use the concrete `DbContext` directly when an additional interface would provide little value: + +```csharp +AddDbExecutionContext(...) +``` + +Both approaches follow the same execution model: + +- the selected type implements `IExecutionContext`; +- the context is registered with DataArc; +- workflows select it explicitly; +- entity types are supplied to command and query operations; +- reads and writes use DataArc’s shared operation contracts. + +The choice between an interface and a concrete context is architectural rather than technical. + +DataArc does not require applications to hide their concrete `DbContext`. It provides that option while preserving the same command and query patterns for both styles. \ No newline at end of file diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..f3e3101 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,333 @@ +# Getting Started + +This guide introduces the basic DataArc.EntityFrameworkCore execution model. + +The setup is: + +```text +Install DataArc.EntityFrameworkCore + ↓ +Create or expose an EF Core execution context + ↓ +Register the execution context + ↓ +Inject a command or query factory + ↓ +Select the execution context + ↓ +Execute the operation +``` + +## Prerequisites + +You need: + +- a supported .NET SDK +- a supported Entity Framework Core version +- an EF Core database provider +- a database connection string +- the DataArc.EntityFrameworkCore NuGet package + +The public demo uses SQL Server. + +See [Compatibility](compatibility.md) for supported versions. + +## Install the package + +Install DataArc.EntityFrameworkCore in the project that configures persistence: + +```bash +dotnet add package DataArc.EntityFrameworkCore +``` + +Install the EF Core provider required by the application. + +For SQL Server: + +```bash +dotnet add package Microsoft.EntityFrameworkCore.SqlServer +``` + +## Choose an execution-context style + +DataArc supports two valid styles: + +1. a public execution-context contract implemented by an internal concrete `DbContext` +2. a concrete `DbContext` that implements `IExecutionContext` directly + +Use a contract when application code should not depend on the concrete persistence implementation. + +Use the concrete context directly when hiding it adds no value. + +## Option 1: Execution-context contract + +Create a public contract that extends `IExecutionContext`: + +```csharp +using DataArc.Core; +using DataArc.EntityFrameworkCore.Demo.Persistence.Database.DBModels; + +using Microsoft.EntityFrameworkCore; + +namespace DataArc.EntityFrameworkCore.Demo.Persistence.Contracts +{ + public interface IGoogleDbContext : IExecutionContext + { + DbSet? Employer { get; set; } + + DbSet? Employee { get; set; } + } +} +``` + +Implement the contract with the concrete EF Core `DbContext`: + +```csharp +using DataArc.EntityFrameworkCore.Demo.Persistence.Contracts; +using DataArc.EntityFrameworkCore.Demo.Persistence.Database.DBModels; + +using Microsoft.EntityFrameworkCore; + +namespace DataArc.EntityFrameworkCore.Demo.Persistence.Database.DBContexts +{ + internal class GoogleDbContext + : DbContext, + IGoogleDbContext + { + public GoogleDbContext( + DbContextOptions dbContextOptions) + : base(dbContextOptions) + { + } + + public DbSet? Employer { get; set; } + + public DbSet? Employee { get; set; } + + protected override void OnModelCreating( + ModelBuilder modelBuilder) + { + base.OnModelCreating(modelBuilder); + + modelBuilder + .Entity() + .Property(employer => employer.Description) + .HasColumnType("text"); + } + } +} +``` + +Application workflows target the contract: + +```csharp +.UseDbExecutionContext() +``` + +## Option 2: Direct concrete DbContext + +A concrete context can implement `IExecutionContext` directly: + +```csharp +using DataArc.Core; +using DataArc.EntityFrameworkCore.Demo.Persistence.Database.DBModels; + +using Microsoft.EntityFrameworkCore; + +namespace DataArc.EntityFrameworkCore.Demo.Persistence.Database.DBContexts +{ + public class GoogleDbContext + : DbContext, + IExecutionContext + { + public GoogleDbContext( + DbContextOptions dbContextOptions) + : base(dbContextOptions) + { + } + + public DbSet? Employer { get; set; } + + public DbSet? Employee { get; set; } + + protected override void OnModelCreating( + ModelBuilder modelBuilder) + { + base.OnModelCreating(modelBuilder); + + modelBuilder + .Entity() + .Property(employer => employer.Description) + .HasColumnType("text"); + } + } +} +``` + +Application workflows target the concrete context: + +```csharp +.UseDbExecutionContext() +``` + +## Register DataArc + +Call `AddDataArcCore` once at the application root. + +Register EF Core execution contexts through `ConfigureDataArc`. + +### Contract-based registration + +```csharp +services.AddDataArcCore(); + +services.ConfigureDataArc(dataArc => +{ + dataArc.UseEntityFrameworkCore(ef => + { + ef.AddDbExecutionContext< + IGoogleDbContext, + GoogleDbContext>(options => + { + options.UseSqlServer( + configuration.GetConnectionString( + "GoogleDb")); + }); + }); +}); +``` + +### Direct DbContext registration + +```csharp +services.AddDataArcCore(); + +services.ConfigureDataArc(dataArc => +{ + dataArc.UseEntityFrameworkCore(ef => + { + ef.AddDbExecutionContext(options => + { + options.UseSqlServer( + configuration.GetConnectionString( + "GoogleDb")); + }); + }); +}); +``` + +## Execute a query + +Inject `IQueryFactory` into an application service or repository: + +```csharp +private readonly IQueryFactory _queryFactory; + +public SalaryAdjustmentService( + IQueryFactory queryFactory) +{ + _queryFactory = queryFactory; +} +``` + +Create a query and select the execution boundary explicitly: + +```csharp +var employeesQuery = await _queryFactory.CreateQueryAsync(); + +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold); +``` + +A contract-based context can be selected in the same way: + +```csharp +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Rating > 4.5); +``` + +## Execute a command + +Inject `ICommandFactory` into the application workflow: + +```csharp +private readonly ICommandFactory _commandFactory; + +public SalaryAdjustmentService( + ICommandFactory commandFactory) +{ + _commandFactory = commandFactory; +} +``` + +Create a command builder and add work to an execution context: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +Build and execute the command: + +```csharp +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +## Execute work across several contexts + +A single command pipeline can target several registered execution contexts: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + +var command = await commandBuilder.BuildAsync(); + +var commandResult = await command.ExecuteAsync(); +``` + +Each target remains explicit and independently routed. + +## Check the result + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + $"Execution failed. {commandResult.Exception?.Message}"); +} + +return commandResult.TotalAffected; +``` + +## Next steps + +Continue with: + +- [Execution Contexts](execution-contexts.md) +- [Query Pipelines](query-pipelines.md) +- [Command Pipelines](command-pipelines.md) +- [Bulk and Parallel Operations](bulk-and-parallel-operations.md) +- [Structured Execution Results](structured-results.md) \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..33b2133 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,109 @@ +# DataArc.EntityFrameworkCore + +DataArc.EntityFrameworkCore provides an explicit execution layer for Entity Framework Core applications. + +It is designed for workflows that need to coordinate queries, commands, bulk operations, transactions, or database tooling across one or more EF Core `DbContext` boundaries. + +EF Core remains responsible for: + +- entity mapping +- change tracking +- LINQ translation +- database-provider integration +- database communication + +DataArc controls how application workflows execute that work. + +## What DataArc provides + +DataArc.EntityFrameworkCore supports: + +- explicit database execution boundaries +- command and query factories +- command and query pipelines +- contract-based or direct `DbContext` execution +- bulk operations +- parallel execution across multiple contexts +- transactional command workflows +- structured execution results +- database creation and deletion +- SQL script generation +- multi-`DbContext` and multi-database workflows + +## Core execution model + +```text +Application workflow + ↓ +Command or query factory + ↓ +Explicit execution-context selection + ↓ +EF Core DbContext + ↓ +Database +``` + +A workflow can select a contract-based execution boundary explicitly: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Rating > 4.5); +``` + +DataArc can also target a concrete `DbContext` directly: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Rating > 4.5); +``` + +Execution-context contracts are optional architectural boundaries. They are not required when targeting the concrete `DbContext` directly is appropriate. + +## Example scenarios + +DataArc.EntityFrameworkCore can support: + +- background workers +- scheduled jobs +- high-volume data imports +- batch processing +- salary or pricing adjustments +- multi-database workflows +- company or tenant database onboarding +- bulk data distribution +- controlled transactional execution +- database creation and reset tooling + +## Documentation + +Continue with: + +- [Getting Started](getting-started.md) +- [Compatibility](compatibility.md) +- [Execution Contexts](execution-contexts.md) +- [Query Pipelines](query-pipelines.md) +- [Command Pipelines](command-pipelines.md) +- [Bulk and Parallel Operations](bulk-and-parallel-operations.md) +- [Transactional Workflows](transactions.md) +- [Structured Execution Results](structured-results.md) +- [Database and Script Generation](database-generation.md) +- [Trial and Licensing](licensing-and-trial.md) + +## Public resources + +- [GitHub repository](https://github.com/pierriemostert/DataArc.EntityFrameworkCore) +- [NuGet package](https://www.nuget.org/packages/DataArc.EntityFrameworkCore) +- [DataArc website](https://www.dataarc.dev) + +--- + +**EF Core owns data access.** + +**DataArc controls execution.** \ No newline at end of file diff --git a/docs/introduction.md b/docs/introduction.md new file mode 100644 index 0000000..f6ecaa6 --- /dev/null +++ b/docs/introduction.md @@ -0,0 +1 @@ +# Introduction \ No newline at end of file diff --git a/docs/licensing-and-trial.md b/docs/licensing-and-trial.md new file mode 100644 index 0000000..cca24a8 --- /dev/null +++ b/docs/licensing-and-trial.md @@ -0,0 +1,383 @@ +# Trial and Licensing + +DataArc.EntityFrameworkCore is commercially licensed software. + +The package can be evaluated through a limited trial before a paid license is activated. + +This page explains the intended licensing model at a practical level. The applicable license agreement and current commercial terms published by Solid Arc Software remain the authoritative source. + +## Trial period + +A DataArc trial is valid for seven days. + +The trial is intended to allow a developer or organisation to: + +- install the package; +- configure execution contexts; +- run the demonstration solution; +- test query and command pipelines; +- evaluate bulk and multi-context workflows; +- determine whether the framework fits the target application. + +The trial is for evaluation. + +It should not be treated as a permanent free production license. + +```text +Install package + -> activate trial + -> evaluate for seven days + -> purchase and activate a paid license + or remove the commercial package +``` + +## Paid licensing + +A paid DataArc.EntityFrameworkCore license permits continued use after the trial period, subject to the purchased license terms. + +The current product, pricing, and licensing options should be obtained from the DataArc website because commercial terms can change independently of the documentation version. + +Use the product page for: + +- current price; +- included rights; +- license duration; +- organisation or developer scope; +- supported deployment environments; +- purchasing and activation instructions. + +Do not rely on a price copied into source code or old documentation as the current commercial offer. + +## Package installation does not grant a paid license + +Installing the NuGet package does not, by itself, grant unrestricted commercial use. + +```text +NuGet package available + != +paid commercial license granted +``` + +The package distribution mechanism and the commercial license are separate concerns. + +A valid trial or paid license is required according to the applicable product terms. + +## Trial activation + +Trial activation associates the evaluation with the relevant runtime or machine information required by the licensing process. + +The exact activation flow is provided through the DataArc portal and package runtime. + +Depending on the product version, activation data may include machine or runtime identifiers such as network-adapter information. + +Only provide activation information through the official DataArc website or the documented package activation process. + +Do not submit licensing or payment information to unverified third-party domains. + +## Runtime environment + +DataArc supports explicit runtime-environment identification through: + +```text +DataArcRuntimeEnvironment +``` + +This value can be supplied as an environment variable where required by the deployed application. + +Example: + +```powershell +$env:DataArcRuntimeEnvironment = "Development" +``` + +For a deployed environment: + +```powershell +$env:DataArcRuntimeEnvironment = "Production" +``` + +Use an environment name that matches the application's actual deployment context. + +Typical values may include: + +- `Development` +- `QA` +- `Staging` +- `Production` + +The exact value should follow the environment configuration used by the organisation and the licensing instructions for the installed package version. + +## Local development + +A developer evaluating or using DataArc locally should: + +1. install the required NuGet packages; +2. configure the supported execution contexts; +3. activate the trial or paid license; +4. set the runtime environment where required; +5. start the application; +6. confirm that license validation succeeds before continuing development. + +Keep licensing configuration outside source control when it contains private activation information. + +Do not commit: + +- private activation keys; +- machine-specific license files; +- purchase details; +- secrets; +- private portal credentials. + +## Configuration and secrets + +Licensing values that behave like credentials should be stored through normal secure configuration mechanisms. + +Suitable options include: + +- environment variables; +- .NET user secrets for local development; +- a managed secret store; +- deployment-platform secret configuration; +- protected server configuration. + +For local .NET development: + +```powershell +dotnet user-secrets set "DataArc:LicenseKey" "YOUR-LICENSE-VALUE" +``` + +The exact configuration key must match the licensing instructions for the installed product version. + +The example demonstrates secure storage, not a guaranteed package key name. + +Do not invent configuration names in production code. Use the names provided by the product activation instructions. + +## Containers + +Paid DataArc licenses can be used in containerised deployments subject to the purchased license terms and runtime configuration. + +Container environments require deliberate configuration because container instances may be: + +- recreated; +- scaled horizontally; +- assigned different network identities; +- deployed across several hosts; +- started by an orchestrator. + +Do not assume that a machine-bound development activation can simply be copied into an arbitrary container fleet. + +Before deployment, confirm: + +- that the purchased license permits container use; +- how runtime identification is configured; +- how many environments or instances are covered; +- where activation values are stored; +- how secrets are injected; +- how scaling and replacement instances are handled. + +## Container configuration + +Supply required licensing configuration through the container runtime rather than baking private values into the image. + +Example: + +```yaml +services: + api: + image: your-application:latest + environment: + DataArcRuntimeEnvironment: Production +``` + +Private licensing values should be provided through the platform's secret mechanism. + +Avoid: + +```dockerfile +ENV DataArcLicenseKey=private-value +``` + +inside a committed Dockerfile. + +A secret embedded in an image can be recovered from image layers and copied into unauthorised environments. + +## CI and build servers + +A CI build that restores and compiles the package is different from a deployed runtime that executes licensed functionality. + +Keep build pipelines free of unnecessary production license values. + +Where tests execute licensed runtime behavior, provide only the configuration required for the authorised test environment. + +Use separate configuration for: + +- developer workstations; +- CI tests; +- QA; +- staging; +- production. + +Do not reuse one private activation value across every environment unless the purchased license explicitly permits it. + +## Production deployment + +Before deploying to production, verify: + +- the trial has not expired; +- the correct paid product has been purchased; +- the production runtime is covered by the license; +- the runtime environment is correctly identified; +- required activation data is available; +- private values are stored securely; +- container or scale-out rights are understood; +- operational teams know how to respond to a license-validation failure. + +A production release should not depend on an evaluation trial. + +## License validation failures + +When license validation fails, investigate the configuration rather than repeatedly reinstalling packages. + +Check: + +- whether the trial expired; +- whether the correct product license was activated; +- whether the runtime environment is configured; +- whether machine or runtime identity changed; +- whether a container was recreated without required configuration; +- whether the license is valid for the current deployment; +- whether private configuration values are available to the process; +- whether the system clock is correct. + +Do not disable or bypass license validation. + +If the configuration appears correct, use the official support or contact channel published by Solid Arc Software. + +## Product boundaries + +DataArc.EntityFrameworkCore and DataArc.Orchestration.Framework are distinct products. + +A license for one product should not be assumed to grant rights to the other unless the purchased commercial terms explicitly state that it does. + +```text +DataArc.EntityFrameworkCore license + -> covers the EF Core product according to its terms + +DataArc.Orchestration.Framework license + -> covers the orchestration product according to its terms +``` + +Confirm the product selected during purchase and activation. + +## Open source repository and commercial licensing + +The public repository allows developers to inspect the demonstration, documentation, and available source published by Solid Arc Software. + +Public visibility does not automatically make every component permissively licensed. + +```text +Publicly visible source + != +MIT license + != +unrestricted commercial use +``` + +Always read the repository license and commercial terms before: + +- copying source; +- modifying package internals; +- redistributing binaries; +- embedding the product in another commercial package; +- publishing a fork; +- offering hosted access to the licensed software. + +## Redistribution + +Do not redistribute DataArc packages, activation material, license files, or private licensed source unless the applicable agreement explicitly permits it. + +A customer application may reference the package according to its license. + +That does not necessarily grant the customer the right to: + +- republish the package under a different name; +- include the package in a public package feed; +- share activation keys; +- distribute license files; +- resell the framework itself. + +## Evaluation checklist + +During the seven-day trial, evaluate the product against a real technical scenario. + +A useful evaluation should confirm: + +- execution-context registration works with the target provider; +- application code can target the required context boundaries; +- query pipelines support the required reads; +- command pipelines support the required writes; +- bulk operations behave correctly with realistic data; +- parallel execution is suitable for independent contexts; +- transaction boundaries are understood; +- structured results provide enough operational feedback; +- performance is measured using representative workloads; +- the architecture is simpler or safer than the existing alternative. + +A trial should answer a concrete adoption question. + +It should not be consumed only by installing the package and reading the API surface. + +## Purchasing decision + +A purchase is justified when DataArc materially improves the target system through: + +- clearer execution-context boundaries; +- consistent multi-context routing; +- reduced custom command and query infrastructure; +- explicit bulk and parallel execution; +- improved workflow visibility; +- reduced architecture-remediation effort; +- lower risk in complex EF Core operations. + +Do not purchase the framework merely because the application uses EF Core. + +Many simple applications do not need an additional execution layer. + +DataArc is intended for systems where explicit execution control provides enough value to justify the dependency and license cost. + +## Current terms and support + +For authoritative and current information, use the official DataArc resources: + +- DataArc product page; +- DataArc pricing page; +- the license included with the repository or package; +- the official purchase and activation workflow; +- Solid Arc Software's published contact channel. + +When this documentation conflicts with the current signed license agreement or official commercial terms, the agreement and official terms take precedence. + +## Summary + +The licensing model is: + +```text +Seven-day evaluation trial + -> evaluate technical fit + +Paid license + -> continued authorised use + -> deployment rights according to purchased terms +``` + +Remember: + +- package installation is not the same as a paid license; +- a trial is for evaluation and expires after seven days; +- paid licensing is required for continued use; +- container use must match the purchased terms; +- runtime environment configuration may be required; +- private activation information must be stored securely; +- DataArc.EntityFrameworkCore and DataArc.Orchestration.Framework are separate products; +- public source visibility does not imply unrestricted redistribution; +- the current official license and commercial terms are authoritative. diff --git a/docs/query-pipelines.md b/docs/query-pipelines.md new file mode 100644 index 0000000..b30fc28 --- /dev/null +++ b/docs/query-pipelines.md @@ -0,0 +1,724 @@ +# Query Pipelines + +Query pipelines provide a consistent way to execute read operations through a registered DataArc execution context. + +A query pipeline follows three steps: + +1. Create a query through `IQueryFactory`. +2. Select an execution context with `UseDbExecutionContext()`. +3. Execute one of DataArc’s defined read operations. + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.Surname == "Doe"); +``` + +The execution context identifies where the query runs, while the generic type supplied to the read operation identifies the model being queried. + +```csharp +UseDbExecutionContext() +ReadWhere(...) +``` + +## Why query pipelines exist + +Entity Framework Core allows developers to read data directly through a `DbContext` or `DbSet`. + +That flexibility can lead to different features or developers implementing reads in different ways: + +- direct `DbSet` access; +- custom repository methods; +- feature-specific query services; +- arbitrary `IQueryable` composition; +- inconsistent projection and paging patterns; +- different approaches to synchronous and asynchronous execution. + +DataArc provides a defined query contract so that reads follow the same recognizable pattern throughout an application. + +```csharp +_queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.IsActive); +``` + +This keeps the following decisions visible in the workflow: + +- which execution context is selected; +- which model is being queried; +- which read operation is being performed; +- whether execution is synchronous or asynchronous; +- whether the result contains full entities or a selected projection. + +The current DataArc contract returns materialized results rather than exposing the underlying `IQueryable` to application workflows. + +## Synchronous query pipelines + +Create a synchronous query with `CreateQuery()`: + +```csharp +var query = _queryFactory.CreateQuery(); +``` + +Select the required execution context and execute a read operation: + +```csharp +var employees = query + .UseDbExecutionContext() + .ReadWhere( + employee => employee.Surname == "Doe"); +``` + +The query can also be written as one fluent pipeline: + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.Surname == "Doe"); +``` + +Synchronous query operations are appropriate when the application intentionally uses synchronous database access. + +For web applications, services, background workers, and other scalable workloads, asynchronous query operations are generally the preferred path. + +## Asynchronous query pipelines + +Create an asynchronous query with `CreateQueryAsync()`: + +```csharp +var employeesQuery = await _queryFactory.CreateQueryAsync(); +``` + +Select the execution context and execute an asynchronous read operation: + +```csharp +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold); +``` + +The factory operation and the database operation are both asynchronous: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +Depending on the selected operation, asynchronous queries return materialized results such as: + +```csharp +Task +``` + +or: + +```csharp +Task> +``` + +## Selecting an execution context + +Every query pipeline explicitly selects an execution context: + +```csharp +UseDbExecutionContext() +``` + +For a public execution-context contract: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +For a concrete `DbContext` that implements `IExecutionContext` directly: + +```csharp +var customers = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +The query contract remains the same in both cases. + +The execution-context type answers: + +> Which registered database boundary should execute this query? + +The generic type supplied to the read operation answers: + +> Which model should be queried? + +```csharp +.UseDbExecutionContext() +.ReadWhereAsync(...) +``` + +The execution context and entity type are therefore selected independently. + +## Reading one entity + +Use `ReadOne` or `ReadOneAsync` to retrieve an entity by its key values. + +```csharp +var employee = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadOne(employeeId); +``` + +The asynchronous equivalent is: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employee = await query + .UseDbExecutionContext() + .ReadOneAsync(employeeId); +``` + +The asynchronous operation returns: + +```csharp +Task +``` + +A null result indicates that no matching entity was found. + +Composite keys can be supplied as multiple key values: + +```csharp +var record = await query + .UseDbExecutionContext() + .ReadOneAsync( + tenantId, + recordId); +``` + +The key values must be supplied in the order defined by the Entity Framework Core model. + +## Reading all entities + +Use `ReadAll` or `ReadAllAsync` to retrieve all rows for a model. + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadAll(); +``` + +The asynchronous equivalent is: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadAllAsync(); +``` + +The asynchronous operation returns: + +```csharp +Task> +``` + +`ReadAll` should be used carefully for tables that may contain a large number of records. + +For large datasets, use filtering or paging instead. + +## Filtering entities + +Use `ReadWhere` or `ReadWhereAsync` to filter entities with an expression. + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadWhere( + employee => employee.Surname == "Doe"); +``` + +The asynchronous equivalent is: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold); +``` + +The predicate is supplied as: + +```csharp +Expression> +``` + +Entity Framework Core translates supported expressions through the database provider configured for the selected execution context. + +For example: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => + employee.IsActive && + employee.Salary >= minimumSalary); +``` + +## Entity-shaped projections + +DataArc supports projections that return the same entity type. + +```csharp +var employeesQuery = await _queryFactory.CreateQueryAsync(); + +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold, + employee => new Employee + { + Name = employee.Name, + Surname = employee.Surname + }); +``` + +The second expression is a projection selector: + +```csharp +employee => new Employee +{ + Name = employee.Name, + Surname = employee.Surname +} +``` + +Because the selector is supplied as an expression tree, Entity Framework Core can translate it as part of the database query. + +Only the referenced columns are selected from the database. + +Conceptually, the generated query is equivalent to: + +```sql +SELECT Name, Surname +FROM Employees +WHERE Salary > @salaryThreshold; +``` + +The returned objects are therefore entity-shaped query results containing only the selected values. + +They are not fully populated entity records. + +The results can then be mapped to an application DTO after materialization: + +```csharp +var employeeDtos = employees + .Select(employee => new EmployeeDto + { + Name = employee.Name, + Surname = employee.Surname + }) + .ToList(); +``` + +The complete flow is: + +```text +Database + -> filter by Salary + -> select Name and Surname + -> materialize entity-shaped results + -> map results to EmployeeDto in memory +``` + +This approach is useful when: + +- the workflow requires only a subset of entity columns; +- unnecessary data transfer should be reduced; +- the current query contract should continue returning `TEntity`; +- DTO mapping is intentionally performed after materialization. + +Because the result may contain only selected columns, consumers should treat these objects as read results rather than complete entities intended for later updates. + +## DTO and read-model projections + +DataArc also supports projecting an entity directly into a different result type. + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employeeSummaries = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.IsActive, + employee => new EmployeeSummaryDto + { + Id = employee.Id, + Name = employee.Name, + Surname = employee.Surname, + Salary = employee.Salary + }); +``` + +The first generic type identifies the persisted entity: + +```csharp +Employee +``` + +The second generic type identifies the projected result: + +```csharp +EmployeeSummaryDto +``` + +The selector is supplied as: + +```csharp +Expression> +``` + +The operation returns: + +```csharp +Task> +``` + +A result model could be defined as: + +```csharp +public sealed class EmployeeSummaryDto +{ + public int Id { get; init; } + + public string Name { get; init; } = string.Empty; + + public string Surname { get; init; } = string.Empty; + + public decimal Salary { get; init; } +} +``` + +Direct result projections are useful when: + +- returning API or application response models; +- selecting only the required database columns; +- creating workflow-specific read models; +- avoiding an additional in-memory mapping step; +- preventing persistence entities from crossing application boundaries. + +Both projection styles are supported: + +```csharp +ReadWhereAsync( + predicate, + entitySelector) +``` + +and: + +```csharp +ReadWhereAsync( + predicate, + resultSelector) +``` + +Use an entity-shaped projection when the result should remain `TEntity`. + +Use the two-type projection when the query should return a separate DTO or read model. + +## Paging query results + +Use `ReadPaged` or `ReadPagedAsync` to retrieve a fixed range of records. + +```csharp +var employees = _queryFactory + .CreateQuery() + .UseDbExecutionContext() + .ReadPaged( + startIndex: 0, + recordCount: 50); +``` + +The asynchronous equivalent is: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadPagedAsync( + startIndex: 0, + recordCount: 50); +``` + +The `startIndex` specifies how many records should be skipped. + +The `recordCount` specifies the maximum number of records returned. + +For example, the second page of a fifty-record result set begins at index `50`: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadPagedAsync( + startIndex: 50, + recordCount: 50); +``` + +## Filtering with paging + +Use `ReadWherePaged` or `ReadWherePagedAsync` to combine filtering and paging. + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadWherePagedAsync( + employee => employee.IsActive, + startIndex: 0, + recordCount: 50); +``` + +Filtering is applied before the requested range is materialized. + +Projected results can also be returned from a filtered and paged query: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadWherePagedAsync( + employee => employee.IsActive, + employee => new EmployeeSummaryDto + { + Id = employee.Id, + Name = employee.Name, + Salary = employee.Salary + }, + startIndex: 0, + recordCount: 50); +``` + +This combines: + +- an execution context; +- a persisted entity type; +- a filter expression; +- a result projection; +- a paging range; +- asynchronous materialization. + +The asynchronous operation returns: + +```csharp +Task> +``` + +## Stored procedure queries + +The asynchronous query contract supports stored procedure execution. + +A typed stored procedure result can be returned with: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var results = await query + .UseDbExecutionContext() + .ExecuteStoredProcedureAsync( + "dbo.GetEmployeeReport", + new + { + DepartmentId = departmentId, + MinimumSalary = minimumSalary + }); +``` + +The result type must be a reference type with a parameterless constructor: + +```csharp +public sealed class EmployeeReportRow +{ + public int EmployeeId { get; set; } + + public string EmployeeName { get; set; } = string.Empty; + + public decimal Salary { get; set; } +} +``` + +Stored procedure results can also be returned dynamically: + +```csharp +var results = await query + .UseDbExecutionContext() + .ExecuteStoredProcedureAsync( + "dbo.GetEmployeeReport", + new + { + DepartmentId = departmentId + }); +``` + +A cancellation token can be supplied: + +```csharp +var results = await query + .UseDbExecutionContext() + .ExecuteStoredProcedureAsync( + "dbo.GetEmployeeReport", + parameters, + cancellationToken); +``` + +Stored procedure names, parameter handling, result mapping, and supported behavior remain dependent on the configured Entity Framework Core database provider. + +## Using query results in a workflow + +Query pipelines can be combined with command pipelines in an application workflow. + +The following example reads employees from one execution context: + +```csharp +var employeesQuery = await _queryFactory.CreateQueryAsync(); + +var employees = await employeesQuery + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold); +``` + +The application then applies its business operation to the materialized results: + +```csharp +foreach (var employee in employees) +{ + employee.Salary += + employee.Salary * salaryAdjustmentBaseRate; +} +``` + +The resulting collection can then be supplied to a command pipeline: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); +``` + +This keeps the responsibilities visible: + +```text +Query pipeline + -> select execution context + -> execute read operation + -> materialize results + +Application workflow + -> apply business changes + +Command pipeline + -> select target execution contexts + -> build database operations + -> execute commands +``` + +A complete workflow may therefore read from one execution context and write to several others while using the same DataArc execution model throughout. + +## Materialized results + +The current DataArc query contract returns materialized results. + +Collection operations return: + +```csharp +IReadOnlyList +``` + +or: + +```csharp +IReadOnlyList +``` + +DataArc does not currently expose an `IQueryable` for application code to continue composing after selecting an execution context. + +The supported pattern is: + +```csharp +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.IsActive); +``` + +rather than: + +```csharp +var employees = await query + .UseDbExecutionContext() + .Where(employee => employee.IsActive) + .ToListAsync(); +``` + +The second form may be familiar to Entity Framework Core users, but it is not the current DataArc query contract. + +The defined read operations ensure that developers use consistent patterns for: + +- execution-context selection; +- filtering; +- projection; +- paging; +- stored procedure execution; +- result materialization. + +## Query pipeline summary + +A DataArc query pipeline consists of: + +```text +IQueryFactory + -> CreateQuery() or CreateQueryAsync() + -> UseDbExecutionContext() + -> Read operation + -> Materialized result +``` + +For example: + +```csharp +var query = await _queryFactory.CreateQueryAsync(); + +var employees = await query + .UseDbExecutionContext() + .ReadWhereAsync( + employee => employee.Salary > salaryThreshold); +``` + +The factory creates the query pipeline. + +The execution context selects the registered database boundary. + +The read operation identifies the entity, filter, projection, paging, or stored procedure behavior. + +Entity Framework Core executes the resulting database query through the provider registered for that execution context. \ No newline at end of file diff --git a/docs/structured-results.md b/docs/structured-results.md new file mode 100644 index 0000000..7d1d004 --- /dev/null +++ b/docs/structured-results.md @@ -0,0 +1,629 @@ +# Structured Execution Results + +DataArc command execution returns a structured result instead of requiring the calling workflow to infer success from control flow alone. + +The current command result exposes: + +```csharp +commandResult.Success +commandResult.Exception +commandResult.TotalAffected +``` + +These values allow the application to determine: + +- whether execution completed successfully; +- whether an exception was captured; +- how many records were affected across the executed operations. + +```csharp +var commandResult = await command.ExecuteAsync(); + +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "Command execution failed.", + commandResult.Exception); +} + +return commandResult.TotalAffected; +``` + +## Why structured results exist + +A database operation can complete in several ways: + +```text +Execution + | + +-- succeeds + | -> Success = true + | -> TotalAffected contains the affected count + | + +-- fails + -> Success = false + -> Exception contains failure information +``` + +Returning a structured result keeps the execution outcome explicit and gives the application one place to inspect command completion. + +This is particularly useful when a command contains: + +- several operations; +- bulk operations; +- several execution contexts; +- parallel execution; +- a workflow that must report one combined result. + +## The result does not replace application decisions + +DataArc reports the execution outcome. + +The application still decides what that outcome means. + +For example, a failed command may cause an API to return an error response: + +```csharp +if (!commandResult.Success) +{ + return Results.Problem( + title: "Database operation failed", + detail: commandResult.Exception?.Message); +} +``` + +A background worker may log the failure and stop processing: + +```csharp +if (!commandResult.Success) +{ + logger.LogError( + commandResult.Exception, + "Employee replication failed."); + + return; +} +``` + +An application service may convert the result into its own contract: + +```csharp +if (!commandResult.Success) +{ + return ApplicationResult.Failure( + commandResult.Exception?.Message + ?? "The command could not be completed."); +} + +return ApplicationResult.Success( + commandResult.TotalAffected); +``` + +The DataArc result belongs to the execution layer. + +The public API, user interface, worker, or application service should translate it into the form required by that boundary. + +## Success + +`Success` indicates whether the command execution completed successfully. + +```csharp +if (commandResult.Success) +{ + // Continue the workflow. +} +``` + +A typical failure check is: + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "Command execution failed.", + commandResult.Exception); +} +``` + +Use `Success` as the primary execution outcome. + +Do not use `TotalAffected > 0` as a substitute for checking success. + +A successful command may legitimately affect zero records. + +Examples include: + +- no rows matched the operation; +- the input collection was empty and handled upstream; +- the target state already matched the requested state; +- an idempotent operation had nothing new to apply. + +Likewise, a command that affected some records before another operation failed should not automatically be treated as successful. + +## Exception + +`Exception` contains failure information when command execution fails. + +```csharp +if (commandResult.Exception is not null) +{ + logger.LogError( + commandResult.Exception, + "Command execution failed."); +} +``` + +Preserve the original exception when raising a higher-level exception: + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "The employee update could not be completed.", + commandResult.Exception); +} +``` + +This retains the underlying cause for diagnostics. + +Avoid returning raw internal exception details directly to end users. + +For a public API, log the exception and return an appropriate application-safe message: + +```csharp +if (!commandResult.Success) +{ + logger.LogError( + commandResult.Exception, + "Salary adjustment execution failed."); + + return Results.Problem( + title: "Salary adjustment failed", + detail: "The operation could not be completed."); +} +``` + +## TotalAffected + +`TotalAffected` reports the total number of records affected by the command execution. + +```csharp +var affectedRecords = + commandResult.TotalAffected; +``` + +A typical successful workflow returns the count: + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "Command execution failed.", + commandResult.Exception); +} + +return commandResult.TotalAffected; +``` + +For a command containing several operations, `TotalAffected` represents the combined affected count reported by the command result. + +Conceptually: + +```text +Operation A -> 100 affected +Operation B -> 100 affected +Operation C -> 100 affected + | + v +TotalAffected -> combined command count +``` + +The affected count is an execution metric. + +It is not, by itself, proof that the business workflow reached the expected state. + +The application may still need to verify: + +- that the expected targets participated; +- that the required entities were processed; +- that no business rule was violated; +- that external follow-up work completed; +- that a partial failure did not occur across separate boundaries. + +## Complete command result example + +```csharp +public async Task ReplicateEmployeesAsync( + IReadOnlyList employees, + int batchSize) +{ + if (employees.Count == 0) + { + return 0; + } + + var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + var command = await commandBuilder.BuildAsync(); + + var commandResult = await command + .ExecuteParallelAsync(); + + if (!commandResult.Success) + { + throw new InvalidOperationException( + "Employee replication failed.", + commandResult.Exception); + } + + return commandResult.TotalAffected; +} +``` + +This workflow: + +1. avoids building a command when no work exists; +2. adds operations to explicit execution contexts; +3. builds the command; +4. executes the independent operations in parallel; +5. checks `Success`; +6. preserves `Exception` when raising a higher-level failure; +7. returns `TotalAffected` only after successful execution. + +## Result handling in an application service + +An application service should normally convert the DataArc result into a use-case-specific result. + +```csharp +public async Task> + ProcessSalaryAdjustmentsAsync( + IReadOnlyList employees, + int batchSize) +{ + var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + + commandBuilder + .UseDbExecutionContext() + .AddBulk(employees, batchSize); + + var command = await commandBuilder.BuildAsync(); + + var commandResult = await command.ExecuteAsync(); + + if (!commandResult.Success) + { + return ApplicationResult.Failure( + commandResult.Exception?.Message + ?? "Salary adjustment execution failed."); + } + + return ApplicationResult.Success( + commandResult.TotalAffected); +} +``` + +This separates: + +```text +DataArc execution result + -> technical command outcome + +Application result + -> use-case outcome exposed to the caller +``` + +The application result may include additional information such as: + +- a business error code; +- validation messages; +- a workflow identifier; +- a user-safe description; +- the affected record count; +- retry guidance. + +## Result handling in an API endpoint + +An endpoint can map the application outcome to HTTP responses: + +```csharp +var result = await salaryAdjustmentService + .ProcessSalaryAdjustmentsAsync( + employees, + batchSize); + +if (!result.Success) +{ + return Results.Problem( + title: "Salary adjustment failed", + detail: result.ErrorMessage); +} + +return Results.Ok(new +{ + Affected = result.Value +}); +``` + +The endpoint should not need to understand the internal database exception model. + +That translation belongs in the application boundary. + +## Result handling in a background worker + +A worker may choose to log and retry: + +```csharp +var commandResult = await command.ExecuteAsync(); + +if (!commandResult.Success) +{ + logger.LogError( + commandResult.Exception, + "Import command failed."); + + throw new InvalidOperationException( + "The import will be retried.", + commandResult.Exception); +} +``` + +Automatic retry is only appropriate when repeating the operation is safe. + +Before retrying, determine whether the command is: + +- idempotent; +- protected against duplicate inserts; +- able to detect already completed work; +- safe after partial completion; +- using a stable workflow identifier. + +## Results from multi-context execution + +A command may contain operations for several execution contexts: + +```text +Command + | + +-- Google operation + +-- Microsoft operation + +-- OpenAI operation +``` + +The returned result represents the command execution as one application-level outcome. + +However, one application-level result does not imply one distributed transaction. + +If the contexts use separate databases, the workflow must still account for partial completion. + +```text +Google operation succeeds +Microsoft operation succeeds +OpenAI operation fails +``` + +The result may report failure, but the successful database operations may already have committed. + +The application may need to: + +- identify the failed target; +- retry only incomplete work; +- record a reconciliation task; +- run compensation logic; +- flag the workflow for manual review. + +See [Transactional Workflows](transactions.md). + +## Successful execution with zero affected rows + +A successful result can report: + +```csharp +commandResult.Success == true +commandResult.TotalAffected == 0 +``` + +This can be valid. + +The application should decide whether zero affected records is acceptable. + +For example: + +```csharp +if (!commandResult.Success) +{ + return ApplicationResult.Failure( + "The command failed."); +} + +if (commandResult.TotalAffected == 0) +{ + return ApplicationResult.Failure( + "No employee records were updated."); +} +``` + +In another workflow, zero affected rows may simply mean that the data was already current. + +The correct interpretation belongs to the use case. + +## Failed execution with partial effects + +When operations cross separate execution contexts or databases, a failed command can still leave partial effects. + +Do not assume: + +```text +Success = false + -> every database change was rolled back +``` + +That is only guaranteed when all protected changes share a verified transaction boundary. + +For multi-database workflows, record enough operational information to reconcile the outcome. + +Useful information can include: + +- workflow identifier; +- execution-context name; +- operation name; +- start and completion timestamps; +- affected count per target; +- captured exception; +- retry status; +- compensation status. + +The current result members provide the overall command outcome. Application-level telemetry can add the business and operational context required by the system. + +## Logging structured results + +Log the result at the boundary where the workflow is understood. + +```csharp +if (!commandResult.Success) +{ + logger.LogError( + commandResult.Exception, + "Employee replication failed."); +} +else +{ + logger.LogInformation( + "Employee replication completed. " + + "Affected records: {TotalAffected}", + commandResult.TotalAffected); +} +``` + +Avoid logging sensitive entity data unnecessarily. + +Prefer identifiers and execution metadata over full payloads. + +## Do not hide failure + +Avoid code that discards the result: + +```csharp +await command.ExecuteAsync(); +``` + +when the workflow requires confirmation that the operation succeeded. + +Prefer: + +```csharp +var commandResult = await command.ExecuteAsync(); + +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "Command execution failed.", + commandResult.Exception); +} +``` + +A structured result is only useful when the calling workflow inspects it. + +## Do not expose internal exceptions directly + +This is weak API handling: + +```csharp +return Results.BadRequest( + commandResult.Exception?.ToString()); +``` + +It may expose: + +- database details; +- connection information; +- SQL fragments; +- internal type names; +- implementation details. + +Instead: + +```csharp +logger.LogError( + commandResult.Exception, + "Command execution failed."); + +return Results.Problem( + title: "Operation failed", + detail: "The requested operation could not be completed."); +``` + +## Recommended handling pattern + +Use this general shape: + +```csharp +var commandResult = await command.ExecuteAsync(); + +if (!commandResult.Success) +{ + logger.LogError( + commandResult.Exception, + "Command execution failed."); + + return ApplicationResult.Failure( + "The operation could not be completed."); +} + +return ApplicationResult.Success( + commandResult.TotalAffected); +``` + +The responsibilities remain clear: + +```text +DataArc + -> executes command + -> returns structured execution result + +Application + -> interprets result + -> applies business meaning + -> maps failure or success to its own contract + +Presentation or worker + -> exposes or reacts to the application result +``` + +## Summary + +The current DataArc command result exposes: + +```csharp +Success +Exception +TotalAffected +``` + +Use them as follows: + +- check `Success` before treating the command as completed; +- inspect or log `Exception` when execution fails; +- preserve the original exception when raising a higher-level failure; +- use `TotalAffected` as an execution metric after success; +- do not equate affected rows with business success; +- do not assume a failed multi-context command rolled back every target; +- translate the DataArc result into an application-specific result; +- avoid exposing internal exception details to public callers. + +## Next steps + +Continue with: + +- [Command Pipelines](command-pipelines.md) +- [Bulk and Parallel Operations](bulk-and-parallel-operations.md) +- [Transactional Workflows](transactions.md) diff --git a/docs/toc.yml b/docs/toc.yml new file mode 100644 index 0000000..404a29e --- /dev/null +++ b/docs/toc.yml @@ -0,0 +1,38 @@ +- name: Overview + href: index.md + +- name: Getting Started + href: getting-started.md + +- name: Compatibility + href: compatibility.md + +- name: Core Concepts + items: + - name: Execution Contexts + href: execution-contexts.md + + - name: Query Pipelines + href: query-pipelines.md + + - name: Command Pipelines + href: command-pipelines.md + + - name: Bulk and Parallel Operations + href: bulk-and-parallel-operations.md + + - name: Transactional Workflows + href: transactions.md + + - name: Structured Execution Results + href: structured-results.md + +- name: Database Tooling + items: + - name: Database and Script Generation + href: database-generation.md + +- name: Licensing + items: + - name: Trial and Licensing + href: licensing-and-trial.md \ No newline at end of file diff --git a/docs/transactions.md b/docs/transactions.md new file mode 100644 index 0000000..06997df --- /dev/null +++ b/docs/transactions.md @@ -0,0 +1,775 @@ +# Transactional Workflows + +Transactional workflows protect a defined set of database changes from being committed in an inconsistent state. + +DataArc command pipelines can coordinate application operations across registered execution contexts, but the application must still understand the transaction boundary of each workflow. + +The most important distinction is: + +```text +One application workflow + != +one database transaction + +One DataArc command + != +one distributed transaction across every execution context +``` + +A transaction is only as broad as the database connection and provider capabilities that participate in it. + +## Why transaction boundaries matter + +Consider an employee-onboarding operation: + +```text +Create employee + -> create payroll profile + -> provision IT account + -> assign equipment +``` + +The application may describe this as one business operation. + +The underlying technical work may span: + +- one `DbContext`; +- several `DbContext` instances connected to one database; +- several schemas; +- several databases; +- external services that do not participate in database transactions. + +Those scenarios do not have the same consistency guarantees. + +Before implementing a transactional workflow, identify: + +1. which operations must succeed together; +2. which database connection each operation uses; +3. whether all operations can share one transaction; +4. what happens when an external or separate-database operation fails; +5. whether compensation or retry is required. + +## Transaction scope and application scope + +A business operation can be wider than a database transaction. + +```text +Business operation + | + +-- local database transaction + +-- external API call + +-- message publication + +-- second database operation +``` + +The application workflow coordinates all four steps. + +The database transaction may protect only the local database changes. + +DataArc helps make the participating execution contexts and command operations explicit, but it does not remove the physical limits of the underlying databases. + +## EF Core's default transaction behavior + +For relational providers, a single `SaveChanges` call is normally executed within a transaction when the provider supports transactions. + +Conceptually: + +```text +tracked changes + -> SaveChanges + -> transaction begins + -> SQL operations execute + -> commit on success + -> rollback on failure +``` + +This is sufficient for many ordinary single-context writes. + +An explicit transaction is useful when the workflow needs to group several persistence steps into one local transaction boundary. + +The exact behavior remains provider-dependent and should be tested with the database used by the application. + +## Single-context transactional workflow + +The strongest transaction boundary is normally one execution context using one database connection. + +```text +Application service + -> one execution context + -> one database connection + -> one transaction +``` + +Within that boundary, the application can: + +- read the required state; +- apply business rules; +- add or update several entities; +- save the changes; +- commit only when every required operation succeeds. + +A representative EF Core shape is: + +```csharp +await using var transaction = await dbContext + .Database + .BeginTransactionAsync(); + +try +{ + // Perform the required database operations. + + await dbContext.SaveChangesAsync(); + + await transaction.CommitAsync(); +} +catch +{ + await transaction.RollbackAsync(); + throw; +} +``` + +This example shows the underlying EF Core transaction principle. + +The concrete transaction implementation should remain inside the persistence boundary when the application layer depends only on an execution-context contract. + +## Keep provider-specific transaction code in persistence + +When the application uses a public execution-context contract such as: + +```csharp +public interface IHrDbContext : IExecutionContext +{ + DbSet? Employee { get; set; } + + DbSet? EmploymentRecord { get; set; } +} +``` + +the concrete `DbContext` can remain internal: + +```csharp +internal sealed class HrDbContext : + DbContext, + IHrDbContext +{ + public HrDbContext( + DbContextOptions options) + : base(options) + { + } + + public DbSet? Employee { get; set; } + + public DbSet? EmploymentRecord { get; set; } +} +``` + +Transaction-specific EF Core code should not be forced into the application layer merely to gain access to `Database.BeginTransactionAsync()`. + +A focused persistence operation can own the local transaction: + +```csharp +public interface IEmployeePersistence +{ + Task CreateEmployeeRecordAsync( + Employee employee, + EmploymentRecord employmentRecord); +} +``` + +The implementation can use the concrete context: + +```csharp +internal sealed class EmployeePersistence : + IEmployeePersistence +{ + private readonly HrDbContext _dbContext; + + public EmployeePersistence( + HrDbContext dbContext) + { + _dbContext = dbContext; + } + + public async Task CreateEmployeeRecordAsync( + Employee employee, + EmploymentRecord employmentRecord) + { + await using var transaction = await _dbContext + .Database + .BeginTransactionAsync(); + + try + { + _dbContext.Set().Add(employee); + _dbContext.Set() + .Add(employmentRecord); + + var affected = await _dbContext + .SaveChangesAsync(); + + await transaction.CommitAsync(); + + return affected; + } + catch + { + await transaction.RollbackAsync(); + throw; + } + } +} +``` + +The application service coordinates the business use case while the persistence implementation owns the provider-specific transaction. + +## Several operations in one context + +Several related changes can be committed together when they use the same context and transaction. + +```text +IHrDbContext + | + +-- add Employee + +-- add EmploymentRecord + +-- update Vacancy + | + -> one local transaction +``` + +This is the preferred shape when the records must be atomically consistent and belong to the same persistence boundary. + +Avoid splitting tightly coupled records across separate database boundaries without a clear consistency strategy. + +## Multiple `DbContext` instances against one database + +A modular application may use several `DbContext` types while still connecting to one physical database. + +```text +HrDbContext --------+ + | +FinanceDbContext ---+--> same database + | +OperationsDbContext-+ +``` + +This does not automatically mean that the contexts share one transaction. + +To participate in one local transaction, they generally need compatible provider behavior and coordinated use of the same underlying connection and transaction. + +That arrangement is more complex than a normal single-context transaction. + +Use it only when the modular boundaries justify the added coordination. + +Do not infer atomicity merely because the connection strings point to the same database. + +## Separate databases + +When execution contexts connect to separate databases: + +```text +IHrDbContext + -> HR database + +IFinanceDbContext + -> Finance database + +IInformationTechnologyDbContext + -> IT database +``` + +a command can coordinate operations for all three contexts, but those operations do not automatically share one local transaction. + +```text +One command builder + | + +-- HR operation + +-- Finance operation + +-- IT operation + +Each target + -> its own connection + -> its own transaction boundary +``` + +If the HR operation commits and the IT operation fails, a normal local database transaction cannot roll back the already committed HR database operation. + +That is a distributed consistency problem. + +## DataArc command coordination is not a distributed transaction + +A command can contain several operations: + +```csharp +var commandBuilder = await _commandFactory + .CreateCommandBuilderAsync(); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(hrEmployees, batchSize); + +commandBuilder + .UseDbExecutionContext() + .AddBulk(financeEmployees, batchSize); + +var command = await commandBuilder.BuildAsync(); +``` + +Executing the command: + +```csharp +var commandResult = await command.ExecuteAsync(); +``` + +coordinates the application operations. + +It does not prove that both databases participate in one atomic commit. + +The application must not present a multi-database command as transactionally atomic unless that guarantee is explicitly implemented and verified. + +## Parallel execution and transactions + +Parallel execution is intended for independent operations: + +```csharp +var commandResult = await command + .ExecuteParallelAsync(); +``` + +It should not be used as a substitute for a transaction. + +```text +ExecuteParallelAsync() + -> concurrent independent work + +Transaction + -> commit or roll back a defined unit of work +``` + +If operations must succeed together, concurrency does not solve the consistency requirement. + +Parallel multi-database operations may also increase the possibility of partial completion because several targets can progress independently. + +## Ordered workflows + +When operations depend on one another, execute them in the required application order. + +```text +Step 1: create employee + | + v +Step 2: use employee identifier + | + v +Step 3: create payroll profile +``` + +Do not run these steps in parallel. + +If all steps belong to one local persistence boundary, place them in one local transaction. + +If they cross databases or external systems, use a broader workflow strategy. + +## Local transaction plus external operation + +External services do not normally participate in an EF Core database transaction. + +```text +Local database transaction + -> commit employee + +External identity provider + -> create account +``` + +If the database commits and the external call fails, the workflow is partially complete. + +Possible strategies include: + +- call the external system before committing, when safe; +- commit locally and retry the external operation; +- record a pending provisioning state; +- publish an outbox message within the local transaction; +- execute compensation when the external step fails; +- require manual intervention for exceptional cases. + +The correct strategy depends on the business process. + +## Transactional outbox pattern + +When a local database update must result in a message or external action, an outbox can preserve the local atomic boundary. + +```text +One local transaction + | + +-- save business data + +-- save outbox message + | + -> commit +``` + +A separate worker later publishes the outbox message: + +```text +Outbox worker + -> read pending message + -> publish + -> mark as completed +``` + +This prevents the application from committing business data and then losing the intent to publish when the process crashes between those steps. + +DataArc can coordinate the application workflow around this pattern, while the local database transaction protects the business data and outbox record. + +## Compensation + +When a workflow crosses transaction boundaries, compensation may be more appropriate than attempting one distributed transaction. + +Example: + +```text +1. Create HR employee +2. Create finance profile +3. Provision IT account +``` + +If step 3 fails after steps 1 and 2 commit, compensation could: + +```text +1. mark employee onboarding as incomplete +2. disable or remove the finance profile +3. queue IT provisioning for retry +``` + +Compensation is business logic. + +It should be explicit, observable, and safe to repeat where possible. + +Do not describe compensation as a rollback unless it truly restores the previous business state. + +## Idempotency + +Retries are common in cross-boundary workflows. + +An operation is idempotent when repeating it does not create an incorrect additional effect. + +Examples of idempotency controls include: + +- unique workflow identifiers; +- unique database constraints; +- operation-status tables; +- idempotency keys; +- checking for an existing target record; +- recording completed workflow stages. + +```text +Retry request + -> check workflow identifier + -> already completed? + -> return existing outcome + -> do not create a duplicate +``` + +Idempotency should be designed before enabling automatic retries. + +## Savepoints and partial recovery + +Some relational providers support savepoints inside an existing transaction. + +A savepoint can allow a workflow to roll back part of a local transaction without abandoning the entire transaction. + +Conceptually: + +```text +begin transaction + -> operation A + -> create savepoint + -> operation B + -> operation B fails + -> roll back to savepoint + -> continue or abort +``` + +Savepoint support and behavior are provider-specific. + +Use savepoints only when the added complexity is justified and tested with the target provider. + +## Isolation levels + +A transaction isolation level controls how concurrent database operations can observe one another. + +Higher isolation may improve consistency but can increase: + +- locking; +- blocking; +- deadlocks; +- transaction duration; +- reduced concurrency. + +Lower isolation may improve concurrency but allow more concurrent-state effects. + +Do not select an isolation level from a generic recommendation. + +Choose it from: + +- the business consistency requirement; +- the database provider; +- expected concurrent load; +- the exact read and write pattern. + +Keep transactions as short as practical. + +## Keep transactions short + +Do not hold a database transaction open while performing slow unrelated work. + +Avoid placing these operations inside a local database transaction unless required: + +- remote HTTP calls; +- file uploads; +- long calculations; +- user interaction; +- artificial delays; +- waiting for another process. + +A better shape is: + +```text +Prepare and validate + -> begin transaction + -> perform required database work + -> commit + -> continue with external work +``` + +When the external work must be reliably triggered, use an outbox or another durable workflow mechanism. + +## Error handling + +A local transaction should roll back when the protected operation fails. + +```csharp +await using var transaction = await dbContext + .Database + .BeginTransactionAsync(); + +try +{ + // Database work. + + await dbContext.SaveChangesAsync(); + + await transaction.CommitAsync(); +} +catch +{ + await transaction.RollbackAsync(); + throw; +} +``` + +The application boundary can then convert the failure into its own result: + +```csharp +try +{ + var affected = await _employeePersistence + .CreateEmployeeRecordAsync( + employee, + employmentRecord); + + return ApplicationResult.Success(affected); +} +catch (Exception exception) +{ + logger.LogError( + exception, + "Employee creation failed."); + + return ApplicationResult.Failure( + "Employee creation could not be completed."); +} +``` + +Do not silently catch a transaction failure and return success. + +## Structured command results + +DataArc command execution returns a structured outcome: + +```csharp +commandResult.Success +commandResult.Exception +commandResult.TotalAffected +``` + +A failed result should stop the calling workflow unless the application has an explicit recovery strategy. + +```csharp +if (!commandResult.Success) +{ + throw new InvalidOperationException( + "The command failed.", + commandResult.Exception); +} +``` + +`TotalAffected` is an execution metric. + +It is not proof that every business invariant was satisfied. + +The business workflow should still validate its expected outcome. + +See [Structured Execution Results](structured-results.md). + +## A transactional onboarding shape + +A realistic onboarding workflow may combine local transactions with broader orchestration: + +```text +Onboard employee + | + +-- HR local transaction + | -> create employee + | -> create employment record + | -> update vacancy + | + +-- save outbox event + | + -> commit HR transaction + | + +-- finance provisioning worker + +-- IT provisioning worker + +-- operations allocation worker +``` + +This preserves one strong local transaction for HR ownership while allowing the other bounded contexts to process durable follow-up work. + +The complete business process is eventually consistent rather than one distributed atomic transaction. + +## Choosing the correct consistency model + +Use one local transaction when: + +- all changes belong to one database boundary; +- the records must commit together; +- the provider supports the required behavior; +- the transaction can remain short. + +Use ordered application orchestration when: + +- steps depend on one another; +- each step has a clear outcome; +- a later step should not begin before an earlier step succeeds. + +Use parallel execution when: + +- operations are independent; +- order does not matter; +- partial failure can be handled; +- no shared atomic transaction is required. + +Use an outbox when: + +- a local database commit must reliably trigger messaging or external work. + +Use compensation when: + +- the workflow crosses boundaries; +- already committed work cannot be rolled back technically; +- the business can define a corrective action. + +Use idempotent retries when: + +- transient failures are expected; +- repeating the operation can be made safe. + +## Common mistakes + +### Assuming one command means one transaction + +```text +One command + != +one atomic transaction across all contexts +``` + +### Using parallel execution for dependent steps + +Parallelism is not appropriate when later work requires earlier output. + +### Holding transactions open during external calls + +This increases lock duration and failure risk. + +### Retrying non-idempotent operations blindly + +A retry may create duplicates or repeat already committed work. + +### Treating compensation as a technical rollback + +Compensation is a new business action and may not restore the exact previous state. + +### Hiding the transaction boundary + +The code and documentation should make clear which operations are protected together. + +## Recommended design questions + +Before implementing a transactional workflow, answer: + +1. What is the business operation? +2. Which records must be atomically consistent? +3. Which execution context owns those records? +4. Do all writes use the same database connection? +5. Can the transaction remain short? +6. Are external calls involved? +7. What happens after partial completion? +8. Can failed operations be retried safely? +9. Is an outbox required? +10. What operational evidence is recorded? + +If these questions do not have clear answers, adding a transaction API alone will not make the workflow reliable. + +## Summary + +Transactional workflows require an explicit understanding of boundaries. + +```text +Single execution context + -> strongest local transaction boundary + +Several contexts on one database + -> transaction sharing must be explicitly coordinated + +Separate databases + -> no automatic local atomic transaction + +External services + -> require retries, outbox, compensation, + or another durable workflow strategy +``` + +DataArc can coordinate the application workflow and keep the participating execution contexts visible. + +The underlying database transaction still belongs to the relevant provider, connection, and persistence boundary. + +The central rules are: + +- do not equate one application command with one distributed transaction; +- keep local transactions inside the persistence boundary; +- use parallel execution only for independent operations; +- use ordered execution for dependent steps; +- design for partial failure across separate databases and external systems; +- use idempotency, outbox processing, and compensation where appropriate; +- keep transaction duration short and observable. + +## Next steps + +Continue with: + +- [Command Pipelines](command-pipelines.md) +- [Bulk and Parallel Operations](bulk-and-parallel-operations.md) +- [Structured Execution Results](structured-results.md)