Skip to content

Commit a501acb

Browse files
committed
add more fixes for parity
1 parent 3fb6cce commit a501acb

41 files changed

Lines changed: 723 additions & 129 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

exporthistory/README.md

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,20 +76,56 @@ supplied, all three are exported.
7676
- **Permissions** — the storage credential needs blob write on the container; the DTS credential needs
7777
orchestration read.
7878

79+
## Export format
80+
81+
Each blob holds the instance's full history, and the **blob body is byte-for-byte identical to the .NET
82+
`Microsoft.DurableTask.ExportHistory` output** (pinned by a test against golden output captured from
83+
`Microsoft.Azure.DurableTask.Core`):
84+
85+
- **JSONL** (default, gzipped) is one JSON object per line; **JSON** is a single array.
86+
- Each event is `{"eventType": "...", <type-specific fields>, "eventId": N, "isPlayed": false, "timestamp": "..."}`.
87+
- camelCase field names, null fields omitted, empty maps as `{}`, enum values in PascalCase (e.g. `"Completed"`),
88+
timestamps as trimmed ISO-8601 ending in `Z`, and the same HTML-safe string escaping (`"``\u0022`,
89+
`& < > ' +` and all non-ASCII → `\uXXXX`).
90+
91+
Blob **names**: a lowercase-hex SHA-256 of `"<completedTimestamp>|<instanceId>"` plus the format extension.
92+
7993
## Differences from .NET
8094

8195
1. **Worker registration takes an explicit client**`useExportHistory(workerBuilder, storage, client)`. Java has
8296
no dependency injection, so the export activities require a `DurableTaskClient` for the same backend.
83-
2. **Export format** — history is serialized from the structured `com.microsoft.durabletask.history` domain model
84-
(camelCase, null fields omitted, one event per line for JSONL). Byte-level parity with .NET's
85-
protobuf-`HistoryEvent`-JSON output is an open item.
97+
2. **Entity events** — the .NET export folds durable-entity operations into core events via a stateful converter, so
98+
it has no distinct JSON for them; Java emits entity events in a Java-native shape instead (a reflective projection
99+
with an `eventType` discriminator, e.g. `"EntityLockGranted"` — matching the Python SDK, which also keeps entity
100+
events as first-class types). Every **non-entity** event is byte-for-byte identical to .NET, so only orchestrations
101+
that call entities differ, and only on those entity-specific lines.
86102

87103
## Backend requirement
88104

89105
The export feature relies on the `ListInstanceIds` and `StreamInstanceHistory` gRPC operations. Managed DTS serves
90106
both; the emulator / self-hosted sidecar needs **≥ v0.4.22**. Against an older backend, a raw gRPC `UNIMPLEMENTED`
91107
surfaces (matching .NET).
92108

109+
## Validating the export
110+
111+
Locally, with the DTS emulator and Azurite:
112+
113+
1. Start the backends:
114+
```
115+
docker run --name durabletask-emulator -p 4001:8080 -d mcr.microsoft.com/dts/dts-emulator:latest
116+
docker run --name azurite -p 10000:10000 -d mcr.microsoft.com/azure-storage/azurite azurite-blob --blobHost 0.0.0.0
117+
```
118+
2. Point the app at them: DTS connection `Endpoint=http://localhost:4001;Authentication=None`, storage = the Azurite
119+
dev connection string, container `orchestration-history`.
120+
3. Run an orchestration to a terminal state, then create a `BATCH` export job whose window covers its completion time.
121+
4. Confirm the job reaches `COMPLETED` and inspect progress:
122+
```java
123+
ExportJobDescription d = job.describe();
124+
// d.getStatus() == ExportJobStatus.COMPLETED, d.getExportedInstances() >= 1
125+
```
126+
5. Download the blob from the container (gunzip for JSONL) and inspect it. Every line carries an `eventType`
127+
discriminator, `isPlayed:false`, and a trailing `timestamp`.
128+
93129
## Sample
94130

95131
See [`HistoryExportSample`](../samples/src/main/java/io/durabletask/samples/HistoryExportSample.java):

exporthistory/build.gradle

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ archivesBaseName = 'durabletask-exporthistory'
1212
def grpcVersion = '1.78.0'
1313
def azureCoreVersion = '1.57.1'
1414
def azureStorageBlobVersion = '12.29.1'
15-
def jacksonVersion = '2.18.3'
1615

1716
// Java 11 is used to compile and run all tests. Set the JDK_11 env var to your
1817
// local JDK 11 home directory, e.g. C:/Program Files/Java/openjdk-11.0.12_7/
@@ -34,10 +33,6 @@ dependencies {
3433
// Azure Storage Blobs — export destination for serialized history.
3534
implementation "com.azure:azure-storage-blob:${azureStorageBlobVersion}"
3635

37-
// Jackson — serialize the history domain model to JSONL/JSON for export.
38-
implementation "com.fasterxml.jackson.core:jackson-databind:${jacksonVersion}"
39-
implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:${jacksonVersion}"
40-
4136
// TokenCredential abstraction (from azure-core) — 'api' because
4237
// ExportHistoryStorageOptions exposes TokenCredential in its public API.
4338
api "com.azure:azure-core:${azureCoreVersion}"

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/BlobExportWriter.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
* <p>
2424
* Built once from {@link ExportHistoryStorageOptions} (connection-string or identity auth) and reused across export
2525
* activities. The target container is taken from each {@link ExportDestination}; the container is created on first
26-
* use. Mirrors the upload behavior of the .NET {@code ExportInstanceHistoryActivity}.
26+
* use.
2727
*/
2828
final class BlobExportWriter {
2929

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/CommitCheckpointRequest.java

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,8 @@
88
/**
99
* Request to commit a checkpoint with progress updates and optional failures.
1010
* <p>
11-
* Mirrors the .NET {@code Microsoft.DurableTask.ExportHistory.CommitCheckpointRequest}. When
12-
* {@link #getCheckpoint()} is non-null the cursor moves forward (successful batch); when {@code null} the cursor is
13-
* retained (failed batch eligible for retry).
11+
* When {@link #getCheckpoint()} is non-null the cursor moves forward (successful batch); when {@code null} the
12+
* cursor is retained (failed batch eligible for retry).
1413
*/
1514
public final class CommitCheckpointRequest {
1615

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExecuteExportJobOperationOrchestrator.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
* Orchestrator that executes a single operation on an export job entity and returns its result.
1010
* <p>
1111
* The client schedules this orchestrator (rather than signaling the entity directly) so it can await completion and
12-
* surface validation errors. Mirrors the .NET {@code ExecuteExportJobOperationOrchestrator}.
12+
* surface validation errors.
1313
*/
1414
public final class ExecuteExportJobOperationOrchestrator implements TaskOrchestration {
1515

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportBlobNaming.java

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,22 @@
66
import java.security.MessageDigest;
77
import java.security.NoSuchAlgorithmException;
88
import java.time.Instant;
9+
import java.time.OffsetDateTime;
10+
import java.time.ZoneOffset;
911
import java.time.format.DateTimeFormatter;
1012

1113
/**
12-
* Computes export blob names and paths. The blob name is a SHA-256 hash of
13-
* {@code "<completedTimestamp ISO-8601>|<instanceId>"} plus the format-specific extension, mirroring the .NET
14-
* {@code ExportInstanceHistoryActivity} naming scheme.
14+
* Computes export blob names and paths. The blob name is a lowercase-hex SHA-256 hash of
15+
* {@code "<completedTimestamp>|<instanceId>"} plus the format-specific extension. The timestamp is rendered with a
16+
* fixed round-trip format ({@code yyyy-MM-ddTHH:mm:ss.fffffff+00:00}) so blob names are stable and idempotent under
17+
* retry.
1518
*/
1619
final class ExportBlobNaming {
1720

21+
// Date-time portion of the timestamp format; the fractional seconds and offset are appended manually.
22+
private static final DateTimeFormatter TIMESTAMP_DATE_TIME =
23+
DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss");
24+
1825
private ExportBlobNaming() {
1926
}
2027

@@ -27,10 +34,26 @@ private ExportBlobNaming() {
2734
* @return the blob file name, e.g. {@code "<hex>.jsonl.gz"}
2835
*/
2936
static String blobFileName(Instant completedTimestamp, String instanceId, ExportFormat format) {
30-
String hashInput = DateTimeFormatter.ISO_INSTANT.format(completedTimestamp) + "|" + instanceId;
37+
String hashInput = formatTimestamp(completedTimestamp) + "|" + instanceId;
3138
return sha256Hex(hashInput) + "." + HistoryEventSerializer.fileExtension(format);
3239
}
3340

41+
/**
42+
* Formats an instant as {@code yyyy-MM-ddTHH:mm:ss.fffffff+00:00} (seven fractional digits, explicit UTC
43+
* offset). The instant is treated as UTC.
44+
* <p>
45+
* Note: instance timestamps are truncated to milliseconds upstream, so the sub-millisecond fractional digits
46+
* are always zero here.
47+
*
48+
* @param instant the timestamp (treated as UTC)
49+
* @return the formatted timestamp string
50+
*/
51+
static String formatTimestamp(Instant instant) {
52+
OffsetDateTime utc = instant.atOffset(ZoneOffset.UTC);
53+
long ticks = utc.getNano() / 100L;
54+
return TIMESTAMP_DATE_TIME.format(utc) + "." + String.format("%07d", ticks) + "+00:00";
55+
}
56+
3457
/**
3558
* Combines an optional prefix with a blob file name.
3659
*

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportCheckpoint.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,7 @@
77
/**
88
* Checkpoint information used to resume an export.
99
* <p>
10-
* Mirrors the .NET {@code Microsoft.DurableTask.ExportHistory.ExportCheckpoint}. The
11-
* {@code lastInstanceKey} is the pagination cursor returned by the client {@code listInstanceIds} wrapper.
10+
* The {@code lastInstanceKey} is the pagination cursor returned by the client {@code listInstanceIds} wrapper.
1211
*/
1312
public final class ExportCheckpoint {
1413

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportDestination.java

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66

77
/**
88
* Export destination settings for Azure Blob Storage.
9-
* <p>
10-
* Mirrors the .NET {@code Microsoft.DurableTask.ExportHistory.ExportDestination}.
119
*/
1210
public final class ExportDestination {
1311

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFailure.java

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66

77
/**
88
* Failure of a specific instance export.
9-
* <p>
10-
* Mirrors the .NET {@code Microsoft.DurableTask.ExportHistory.ExportFailure}.
119
*/
1210
public final class ExportFailure {
1311

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFilter.java

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,6 @@
1010

1111
/**
1212
* Filter criteria for selecting orchestration instances to export.
13-
* <p>
14-
* Mirrors the .NET {@code Microsoft.DurableTask.ExportHistory.ExportFilter}.
1513
*/
1614
public final class ExportFilter {
1715

0 commit comments

Comments
 (0)