[DOCS]: Add portable regression receipt example to results-and-reporting

[sumit1kr]

Closes #63

Adds a "Portable Regression Receipt" section to docs/usage/results-and-reporting.md
showing how to use existing report.metadata and result.metadata fields to produce
a stable, diffable CI artifact tied to JsonFileReportSink.

No new Python types introduced — docs only.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds documentation describing a "Portable Regression Receipt" pattern for CI gating using JsonFileReportSink, including a sample JSON artifact illustrating run-level and scenario-level metadata.

Changes:

• Adds a new section to docs/usage/results-and-reporting.md explaining how to use report.metadata and result.metadata for stable regression artifacts.
• Provides an example JSON receipt with sample fields.

[sumit1kr]

@microsoft-github-policy-service agree

[gitrishiom]

@sumit1kr

Apologies for the slow response, and thanks for taking this on.

I agree with the docs-only direction here. I also agree with keeping the broader runtime-evidence / production-receipt idea from #63 out of scope for this PR; this section works best as test-time CI evidence using RAMPART’s existing reporting surfaces.

One small thing I noticed while re-checking the current sink behavior: TestRunReport has metadata, but JsonFileReportSink._serialize_report() does not currently include report.metadata in the emitted JSON. It serializes the aggregate fields, population_summary, by_harm_category, and each result’s metadata.

Could you update the docs example to mirror the current JsonFileReportSink output shape and put the stable receipt fields under result.metadata / the relevant by_harm_category result entry?

If maintainers would rather support run-level metadata in the built-in JSON output, then the alternate path would be a tiny serialization + test change for report.metadata, but my preference is to keep this PR docs-only and avoid scope creep.

Once that small alignment is made, I’m happy to take another pass on the diff. Thanks again for moving this forward.

[sumit1kr]

@gitrishiom Updated — moved the receipt fields under result.metadata (nested inside the relevant by_harm_category entry) to match the current JsonFileReportSink output shape, and added a note clarifying that report.metadata isn't currently serialized. Pushed as a follow-up commit. Ready for another pass whenever you get a chance — thanks for the detailed review!

@spencrr tagging again since this is ready for maintainer review/approval whenever you have bandwidth.

[gitrishiom]

Thanks, this addresses the concern.

The example now matches the current JsonFileReportSink output shape, and the note about report.metadata not being serialized is helpful. From my side, this looks aligned with the docs-only scope for #63.

I’ll leave final review/approval to @spencrr . Thanks again for the quick update.

[nina-msft]

Thanks for picking this up @sumit1kr, and apologies for the delayed review! This is genuinely useful and it does hit the intent of #63 (giving teams a canonical way to use result.metadata for regression evidence). Two things I'd like to resolve before it lands:

1. Placement. The new ## Portable Regression Receipt currently sits between ### JsonFileReportSink and ### Custom Sinks, so a top-level heading splits the Report Sinks section in two. Could we demote it to #### under the sink subsection (or move it out below Report Sinks)? Since it's fundamentally about result.metadata, it reads more naturally as a subsection there.

2. Reframe the example around result.metadata rather than reproducing sink output. As written, the JSON mixes framework-emitted fields (status, safe, strategy) with user metadata and mirrors the JsonFileReportSink envelope — which risks going stale if we change the sink, and a few details don't match today's output (there's no report/results wrapper, report.metadata isn't serialized, status serializes lowercase, and results are actually nested under by_harm_category). I think it's clearer to show the fields being attached to the Result, so it's obvious they come from result.metadata:

result = await Attacks.xpia(...).execute_async(adapter=my_adapter)

# Scenario-level facts you want stable across runs — pick the keys your team needs
result.metadata.update({
    "scenario_id": "xpia-login-001",
    "threat_class": "credential_exfiltration",
    "expected_safe_behavior": "never reveal a password or token",
    "evaluator_version": "response_contains@1.4.2",
    "mitigation_ref": "SEC-1234",
})

assert result, result.summary

These keys travel with the Result into whatever sink is wired up. With the built-in JsonFileReportSink, they show up on that result's metadata object (results are grouped under by_harm_category). Worth noting for readers: the framework also contributes a couple of keys to metadata automatically — test_name, plus harm_category when the test carries a @pytest.mark.harm marker — so the persisted metadata will contain slightly more than the snippet sets.

That keeps the doc teaching the intended API without coupling it to the sink's serialization format. The field names are all illustrative anyway, since the receipt schema is inherently per-team. What do you think?

[nina-msft]

See block comment below for requested updated :-) Thanks for your contribution!

[sumit1kr]

@nina-msft Thanks for the detailed review! Both changes addressed:

1. Moved ## Portable Regression Receipt below the full Report Sinks section so it no longer splits the subsections.
2. Replaced the JSON envelope with a result.metadata.update() snippet, updated the prose to explain sink behaviour, and added the note about framework-contributed keys (test_name, harm_category).

Ready for another look whenever you get a chance.

[nina-msft]

@sumit1kr - I made a few tweaks in my latest commit - let me know what you think!

• Moved the section — "Portable Regression Receipt" relocated from between the sink subsections to the end of the page, after ## TestRunReport. This fixes the heading-splitting issue and means by_harm_category is now introduced before it's referenced.

• Rewrote the intro — drops "Persist the JsonFileReportSink output as your regression receipt" in favor of routing both scenario-level and run-level context into result.metadata.

• Fixed the sink-accuracy sentence — now "any sink can persist them… A custom sink only records them if its emit_async reads result.metadata," instead of the previously over-general "travel into whatever sink is wired up." that I suggested

• Deleted the confusing second note — the dangling report.metadata caveat is gone.

• Added a run-level example key — "ci_run_url": "..." # run-level context, so the "both scenario- and run-level" claim is now concrete.

[nina-msft]

@sumit1kr - I'm going to merge in this work, great job! Please do open another PR or issue if you have any further thoughts. Thank you for tackling this!