Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@
- Session: daemon-owned state for a selected target and opened app or surface.
- Script recording: opt-in session mode armed before actions so a persisted `.ad` can carry portable
action inputs and recording-time target identity evidence. It is distinct from screen/video recording.
- Recorded input parameterization: explicit fill authoring contract that sends literal text only to the
live interaction while the recorder stores `${VAR}` before any durable recording/event/publication
boundary. The caller owns the uppercase variable name; no selector or field-name heuristic infers
sensitivity. Replay resolves the placeholder immediately before dispatch and preserves the authored
placeholder if that run is recorded again.
- Open-to-destination script: self-contained `.ad` script with exactly one initial `open`, a destination
guard after its last app-state mutation, no `close`, and an app session left active for subsequent work.
Avoid: replay (artifact noun), fragment (reserved for lifecycle-free composition), partial script.
Expand Down
23 changes: 11 additions & 12 deletions docs/adr/0016-active-session-script-publication.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,15 +111,14 @@ absence of `close` changes neither action dispatch nor the success response shap

### Sensitive inputs

Executable `.ad` artifacts serialize action inputs literally. A recorded `fill` therefore writes its
text to the published file; diagnostic and event-log redaction cannot protect an input that the replay
engine must later execute. V1 does not claim secret-bearing login flows are safe to publish.
Executable `.ad` artifacts serialize ordinary action inputs literally; diagnostic and event-log
redaction cannot protect an input that replay must later execute. ADR 0017 resolves #1348 for explicit
sensitive fills: `fill ... --record-as PASSWORD` sends the live text to the app while recording only
`${PASSWORD}`. Replay receives the value through `AD_VAR_PASSWORD` or `--env PASSWORD=<value>`.

Native `.ad` replay already supports late-bound `${VAR}` values, but ordinary recording cannot yet
execute with a real value while publishing only its placeholder. That safe-authoring capability is
tracked in [#1348](https://github.com/callstack/agent-device/issues/1348). Until it ships, authors must not
record a journey that enters a secret: use pre-authenticated test state or deliberately non-secret fixture
credentials that are safe to persist. CLI help must state this warning next to the authoring workflow.
This is opt-in and fill-only. Unparameterized fill/type inputs remain literal artifact content, so
authors must use ADR 0017 for each sensitive fill and avoid secret-bearing `type` steps. CLI help states
both the safe workflow and the remaining literal-input warning next to publication guidance.

### Artifact contract

Expand Down Expand Up @@ -167,8 +166,8 @@ executing that script, not the artifact being saved.
bootstrap; authoring resumes only in a fresh session.
- Intermediate lifecycle-free fragments, entry guards, include semantics, composed digests, and shared
fragment pinning remain entirely under #1336.
- Secret-bearing authoring remains unsafe until #1348; the initial workflow is limited to journeys that
do not enter secrets. Arbitrary history ranges remain out of scope.
- Secret-bearing fill authoring uses ADR 0017's explicit `--record-as <VAR>` contract. Unparameterized
fill/type inputs remain literal script content. Arbitrary history ranges remain out of scope.

## Alternatives Considered

Expand Down Expand Up @@ -215,7 +214,7 @@ executing that script, not the artifact being saved.
request-sensitive read-only/mutating subcommands, and destination-guard ordering consumes only that
trait.
- Repair-armed sessions refuse this action without changing repair state.
- CLI help warns that literal `fill` inputs are persisted and tells authors not to record secret-bearing
journeys until #1348's parameterized-input mechanism is available.
- CLI help documents ADR 0017's `--record-as` workflow and warns that unparameterized fill/type inputs
remain literal script content.
- Provider-backed integration scenarios cover the public daemon route, and live iOS and Android runs
prove the saved artifact and post-save session behavior on real backends.
91 changes: 91 additions & 0 deletions docs/adr/0017-parameterized-recorded-inputs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# ADR 0017: Parameterized Recorded Inputs

## Status

Accepted

## Context

Ordinary `.ad` recording persists executable inputs. A literal `fill` value therefore becomes part of
the published script, including any password or token supplied during authoring. Diagnostic and event
redaction cannot make that executable artifact safe because replay still needs the value.

Native replay already resolves `${VAR}` from `--env`/`-e` and `AD_VAR_*` immediately before dispatch.
The missing boundary is authoring: the live app needs the literal, while recording state and every
publication write must receive only the placeholder. Issue
[#1348](https://github.com/callstack/agent-device/issues/1348) defines this requirement; ADR 0016's
active-session publication deliberately excluded secret-bearing journeys until it was solved.

## Decision

Add a fill-only `recordAs` input, spelled `--record-as <VAR>` on the CLI and exposed as `recordAs` by
the Node and MCP fill contracts:

```sh
export AD_VAR_PASSWORD='<secret>'
agent-device open com.example.app --save-script=login.ad
agent-device fill 'id="password"' "$AD_VAR_PASSWORD" --record-as PASSWORD
agent-device wait 'role="heading" label="Home"'
agent-device session save-script
```

The live fill dispatch receives the literal. Its public response, recording state, session event, and
published script use `${PASSWORD}`. Replay later resolves the placeholder from `AD_VAR_PASSWORD` or
`--env PASSWORD=<value>` before the fill is dispatched.

Variable names use replay's uppercase `[A-Z_][A-Z0-9_]*` grammar. `AD_*` remains reserved for runtime
built-ins. `--record-as` is rejected unless script recording is armed, and it cannot be combined with
`--no-record`. Ordinary fills without the option keep literal recording behavior. Sensitivity is never
inferred from a selector, label, field name, or value shape.

### Mapping contract

The caller names the mapping explicitly. Reusing `PASSWORD` deterministically emits `${PASSWORD}`;
using `API_TOKEN` emits `${API_TOKEN}`. There is no secret-to-name table and no literal comparison or
deduplication state. This both avoids retaining a second copy of the secret and makes distinct-value
naming an authoring decision visible in the script.

### Data-flow boundary

The literal exists only on the live request path long enough to execute the interaction:

1. The request scope registers it as an explicit diagnostics secret before platform work. This is in
addition to key-name redaction, because an opaque value need not look secret.
2. The platform interaction receives the literal.
3. Before response/event/recording retention, the fill result's semantic `text` field becomes
`${VAR}` and every occurrence in a post-fill `refLabel` is parameterized. Stable selector/ref
provenance is preserved byte-for-byte.
Untrusted backend extras and nested settle output parameterize every occurrence of the supplied
literal in both keys and values; post-fill selector-chain candidates are dropped only when a
parsed `text`, `label`, or `value` term semantically contains it. Known response and settle
schema fields are the explicit structural boundary and retain their names and provenance values.
4. At `recordActionEntry`, the recorder reconstructs the fill's literal positional, writes only the
placeholder into `SessionAction`, and parameterizes the semantic result field plus arbitrary
backend/settle echoes again. ADR 0012 `target-v1` evidence keeps its structure; exact
value-bearing accessibility labels become the placeholder without rewriting identity fragments
that merely contain the same characters. The `recordAs` control flag itself is not serialized
into the script.
5. The existing ADR 0012/0016 writer receives an already-parameterized action. Its selector provenance,
portability checks, same-directory temp write, and atomic publication algorithm are unchanged.

Because the target and temp file are written from the same prepared script string, neither can contain
the literal once this pre-publication boundary has run.

### Replay and repair

Missing variables retain replay's existing fail-loud behavior: `${VAR}` resolution throws before the
corresponding action is invoked. When replay dispatches an authored fill whose complete text token is a
`${VAR}` placeholder, its daemon-only provenance channel derives `recordAs=VAR` for that live request.
This derivation uses the unresolved source action, independently of whether the resolved value is empty
or whitespace. If the replay is itself being recorded or repaired, the recorder therefore retains the
original placeholder rather than serializing the expanded value. Embedded interpolation remains
ordinary script input; safe authoring emits one complete placeholder token.

## Consequences

- Secret-bearing login/bootstrap scripts can use ADR 0016 active publication safely.
- The caller must opt in for every sensitive fill; unparameterized fill/type values remain literal
script content and help warns accordingly.
- `type` and mutating `find ... fill|type` parameterization are not added by this decision. They require
their own surface and provenance design if needed.
- No new publication format, variable store, secret registry, or sensitivity heuristic is introduced.
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
| [0014 Session Ref-Frame Lifetime](0014-session-ref-frame-lifetime.md) | ref authorization epochs, complete/partial issuance, pre-side-effect expiration, replay/batch compatibility, and cross-platform stale-mutation policy |
| [0015 Direct Maestro Compatibility Engine](0015-direct-maestro-engine.md) | Maestro YAML parsing/execution, compatibility observation policy, conformance, performance gates, gesture integration |
| [0016 Active-Session Script Publication](0016-active-session-script-publication.md) | publishing an armed open-to-destination `.ad` script without closing its live session |
| [0017 Parameterized Recorded Inputs](0017-parameterized-recorded-inputs.md) | safely authoring sensitive fill inputs as `${VAR}` placeholders across recording, replay, and repair |

ADRs record *why*; the registries and gates they describe are the living source of truth — when
prose and a registry disagree, the registry wins and the ADR needs a follow-up.
1 change: 1 addition & 0 deletions scripts/integration-progress-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ function summarizeProviderScenarioFlagCoverage(files) {
['recordingScope', 'recording app vs whole-screen scope', ['scope']],
['intervalMs', 'repeated press interval'],
['delayMs', 'typing/fill delay'],
['recordAs', 'parameterized fill publication for recorded scripts'],
['durationMs', 'scroll, gesture, and TV remote duration'],
['holdMs', 'press hold duration'],
['jitterPx', 'press jitter'],
Expand Down
10 changes: 6 additions & 4 deletions src/cli/parser/__tests__/args-parse-interaction.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -175,20 +175,22 @@ test('parseArgs recognizes gesture subcommand positionals', () => {
assert.deepEqual(transform.positionals, ['transform', '200', '420', '80', '-40', '2', '35']);
});

test('parseArgs recognizes type and fill delay flags', () => {
test('parseArgs recognizes type delay and fill recording parameter flags', () => {
const typeParsed = parseArgs(['type', 'hello', '--delay-ms', '75'], {
strictFlags: true,
});
assert.equal(typeParsed.command, 'type');
assert.deepEqual(typeParsed.positionals, ['hello']);
assert.equal(typeParsed.flags.delayMs, 75);

const fillParsed = parseArgs(['fill', '@e5', 'search', '--delay-ms', '40'], {
strictFlags: true,
});
const fillParsed = parseArgs(
['fill', '@e5', 'search', '--delay-ms', '40', '--record-as', 'SEARCH_TERM'],
{ strictFlags: true },
);
assert.equal(fillParsed.command, 'fill');
assert.deepEqual(fillParsed.positionals, ['@e5', 'search']);
assert.equal(fillParsed.flags.delayMs, 40);
assert.equal(fillParsed.flags.recordAs, 'SEARCH_TERM');
});

test('parseArgs recognizes record --fps flag', () => {
Expand Down
1 change: 1 addition & 0 deletions src/cli/parser/__tests__/cli-help-command-usage.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,7 @@ test('command usage describes delayed typing flags', async () => {
}
assert.match(typeHelp, /--delay-ms <ms>/);
assert.match(fillHelp, /--delay-ms <ms>/);
assert.match(fillHelp, /--record-as <VAR>/);
});

test('snapshot command usage documents diff alias', async () => {
Expand Down
3 changes: 3 additions & 0 deletions src/cli/parser/__tests__/cli-help-topics.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,9 @@ test('usageForCommand resolves workflow help topic', async () => {
assert.match(help, /snapshot -s @e7/);
assert.match(help, /Use plain fill\/type first for ordinary login and form fields/);
assert.match(help, /--delay-ms intentionally paces character entry/);
assert.match(help, /agent-device fill 'id="password"' "\$AD_VAR_PASSWORD" --record-as PASSWORD/);
assert.match(help, /published script contain only \$\{PASSWORD\}/);
assert.match(help, /Do not record passwords, tokens, or other secrets without --record-as/);
assert.match(help, /Read-only visible\/state question: use snapshot\/get\/is\/find/);
assert.match(help, /Use snapshot -i only when refs are needed/);
assert.match(help, /install-from-source --github-actions-artifact org\/repo:app-debug/);
Expand Down
6 changes: 5 additions & 1 deletion src/cli/parser/cli-help.ts
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,11 @@ Reusable open-to-destination scripts:
agent-device wait 'role="heading" label="Screen X"'
agent-device session save-script
session save-script [path] [--force] publishes the sole recorded open through the destination guard, omits close, and leaves the session active. A duration wait, wait stable, or wait @ref is not a destination guard. A second successful open aborts publication; start a fresh session to author again.
Recorded fill/type inputs are written literally to the .ad file. Do not record passwords, tokens, or other secrets; use pre-authenticated test state or non-secret fixture credentials until parameterized input authoring is available.
Unparameterized fill/type inputs are literal .ad script content. For a sensitive fill, arm recording first, keep the live value in an environment variable, and name its replay placeholder explicitly:
export AD_VAR_PASSWORD='<secret>'
agent-device fill 'id="password"' "$AD_VAR_PASSWORD" --record-as PASSWORD
The live app receives the value, while recording state and the published script contain only \${PASSWORD}. Reuse the same name for repeated values and choose distinct names for distinct inputs. --record-as accepts uppercase replay variable names, is fill-only, requires an armed recording, and cannot be combined with --no-record.
Publish with session save-script, then replay with AD_VAR_PASSWORD still set or pass --env PASSWORD=<value>. A missing value fails before that fill runs. Do not record passwords, tokens, or other secrets without --record-as; their literal text will be written to the .ad target.

Snapshots and refs:
snapshot reads visible state. snapshot -i gets current interactive refs only; it is the fast path when the next step is an interaction.
Expand Down
4 changes: 4 additions & 0 deletions src/client/client-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -783,6 +783,8 @@ export type FillOptions = DeviceCommandBaseOptions &
SettleCommandOptions & {
text: string;
delayMs?: number;
/** Publish this fill value as `${VAR}` when script recording is armed. */
recordAs?: string;
verify?: boolean;
};

Expand Down Expand Up @@ -1102,6 +1104,8 @@ export type InternalRequestOptions = AgentDeviceClientConfig &
deviceHub?: boolean;
testIme?: boolean;
noRecord?: boolean;
/** Fill-only script parameter name used to publish `${VAR}` instead of literal text. */
recordAs?: string;
/** #1271 stage 2: force-record this action; mutually exclusive with `noRecord`. */
record?: boolean;
backMode?: BackMode;
Expand Down
7 changes: 7 additions & 0 deletions src/commands/cli-grammar/flag-definitions-action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,13 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
usageLabel: '--delay-ms <ms>',
usageDescription: 'Delay between typed characters',
},
{
key: 'recordAs',
names: ['--record-as'],
type: 'string',
usageLabel: '--record-as <VAR>',
usageDescription: 'Fill: send the live text but publish ${VAR} in an armed .ad recording',
},
{
key: 'durationMs',
names: ['--duration-ms'],
Expand Down
1 change: 1 addition & 0 deletions src/commands/command-flags.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ function buildFlags(options: InternalRequestOptions): CommandFlags {
deviceHub: options.deviceHub,
testIme: options.testIme,
noRecord: options.noRecord,
recordAs: options.recordAs,
record: options.record,
backMode: options.backMode,
metroHost: options.metroHost,
Expand Down
8 changes: 7 additions & 1 deletion src/commands/interaction/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,12 @@ const interactionCliSchemas = {
usageOverride: 'fill <x> <y> <text> | fill <@ref|selector> <text>',
positionalArgs: ['targetOrX', 'yOrText', 'text?'],
allowsExtraPositionals: true,
allowedFlags: [...SELECTOR_SNAPSHOT_FLAGS, 'delayMs', ...postActionObservationCliFlags('fill')],
allowedFlags: [
...SELECTOR_SNAPSHOT_FLAGS,
'delayMs',
'recordAs',
...postActionObservationCliFlags('fill'),
],
},
scroll: {
usageOverride: 'scroll <direction|top|bottom> [amount] [--pixels <n>] [--duration-ms <ms>]',
Expand Down Expand Up @@ -389,6 +394,7 @@ function toFillOptions(input: FillInput): FillOptions {
...toSelectorSnapshotOptions(input),
text: input.text,
delayMs: input.delayMs,
recordAs: input.recordAs,
verify: input.verify,
...toSettleOptions(input),
};
Expand Down
11 changes: 11 additions & 0 deletions src/commands/interaction/interactions.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,14 @@ test('swipe writes only typed daemon input', () => {
pattern: 'ping-pong',
});
});

test('fill projects recordAs through the typed daemon flags', () => {
const request = interactionDaemonWriters.fill({
selector: 'id="password"',
text: 'live-secret',
recordAs: 'PASSWORD',
});

expect(request.positionals).toEqual(['id="password"', 'live-secret']);
expect(request.options.recordAs).toBe('PASSWORD');
});
1 change: 1 addition & 0 deletions src/commands/interaction/interactions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ export const interactionCliReaders = {
target: targetInputFromClientTarget(decoded.target),
text: decoded.text,
delayMs: flags.delayMs,
recordAs: flags.recordAs,
verify: flags.verify,
};
},
Expand Down
3 changes: 3 additions & 0 deletions src/commands/interaction/metadata.ts
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,9 @@ const fillFields = {
target: requiredField(interactionTargetField()),
text: requiredField(stringField('Text to enter into the target.')),
delayMs: integerField('Delay between typed characters.', { min: 0 }),
recordAs: stringField(
'When script recording is armed, send text to the live app but publish it as ${VAR}. Use an uppercase replay variable name such as PASSWORD.',
),
...selectorSnapshotFields(),
...postActionObservationFields('fill'),
};
Expand Down
2 changes: 2 additions & 0 deletions src/contracts/cli-flags.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ export type CliFlags = CloudProviderProfileFields &
recordingScope?: RecordingScope;
intervalMs?: number;
delayMs?: number;
/** Fill: publish the live text as a late-bound ${VAR} in a recorded .ad script. */
recordAs?: string;
durationMs?: number;
holdMs?: number;
jitterPx?: number;
Expand Down
Loading
Loading