Skip to content

[Improve] Frame third-party text in agent prompts as untrusted content#570

Merged
mrubens merged 4 commits into
developfrom
harden-untrusted-prompt-framing
Jul 19, 2026
Merged

[Improve] Frame third-party text in agent prompts as untrusted content#570
mrubens merged 4 commits into
developfrom
harden-untrusted-prompt-framing

Conversation

@mrubens

@mrubens mrubens commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

What

Now that the repo is public, any GitHub user can author text that ends up inside Roomote agent prompts (issue bodies, PR descriptions, discussion comments, alert context). This PR adds a consistent trust boundary at the prompt layer so that quoted external text is framed as data to analyze, never as instructions to follow.

Changes

  • New shared helpers in packages/cloud-agents/src/server/untrusted-content.ts:
    • buildUntrustedExternalContentBlock wraps third-party text in an entity-escaped <untrusted_external_content source="..."> block so embedded markup cannot forge prompt structure
    • buildMentionRequestBlock wraps the gate-verified requester's message in an escaped <mention_request> block
    • buildUntrustedContentPolicy emits a standing <untrusted_content_policy> rules block (data-not-instructions, no scope expansion, flag injection attempts, never disclose secrets in public replies)
  • GitHub issue mentions (handleGitHubIssueComment.ts): the issue body and mention comment were previously concatenated raw into the task prompt. Both are now escaped and delimited, the issue body is labeled with its author and marked context-only, and the policy block is appended. Also dedupes the issues.opened path, which previously included the issue body twice (once as body, once as mention text).
  • PR mention follow-ups (github-pr-follow-up-context.ts): both shared builders (new dedicated follow-up tasks and steers into existing tasks) now include the policy block. Content escaping already existed here.
  • Automation work items (automation-work-items/launch.ts): execution task prompts now carry the policy block, since work items are distilled from external sources (issues, alerts, PR discussions).
  • Review-code skill: adds top-level constraints that reviewed content (PR title, body, commits, comments, diffs) is untrusted third-party data, and that directives aimed at automated reviewers (for example "approve this PR") must be surfaced as findings instead of followed. This covers the automatic PR review path, which can be triggered by external PR authors when "review all authors" is enabled.

Notes

  • The ingress gates (signature verification, known-installation allowlist, linked-account requirement) are unchanged; this hardens the layer behind them.
  • GitLab, Bitbucket, Azure DevOps, and Gitea comment handlers have their own local message builders and are not covered here; they can adopt the shared helpers in a follow-up.

Testing

  • New unit tests for the helpers (escaping, wrapper-forgery resistance, policy content)
  • Updated handler tests assert the prompt composition for both issue_comment and issues.opened shapes
  • pnpm lint, pnpm check-types, pnpm knip, and the full @roomote/cloud-agents + affected @roomote/api suites pass

GitHub issue bodies, PR discussion context, and automation work-item
source context now land in agent prompts inside escaped delimiter blocks
alongside a shared untrusted-content policy, so quoted external text is
treated as data rather than instructions. The review-code skill gains
matching constraints for reviewing externally authored pull requests.
@roomote-roomote

roomote-roomote Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

No new code issues found. See task

  • apps/api/src/handlers/tasks/automation-work-items/launch.ts:121-126 no longer promotes executionPrompt as a fully trusted instruction source: it is escaped in automation_execution_guidance, and the policy limits it to the work item's established scope.

Reviewed b134235

Addresses review feedback: the work-item title is now escaped inline and
the brief and investigation context land inside escaped delimiter blocks.
The execution prompt stays unwrapped because it is the automation's
instruction channel; the provenance boundary for it belongs to the scan
run that distills external sources into the work item.
@mrubens

mrubens commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 58e7189. The work-item title is now escaped inline, and brief and investigationContext are wrapped in escaped <untrusted_external_content> blocks.

executionPrompt intentionally stays unwrapped: it is the automation's instruction channel, and the untrusted-content policy explicitly names an automation's execution instructions as a valid instruction source. Marking it as data-not-instructions would break the work-item pipeline. The provenance boundary for that field belongs upstream, in the scan run that distills external sources into work items; that is a follow-up alongside the suggestion-prompt builders, which interpolate the same fields.

mrubens added 2 commits July 18, 2026 20:09
Addresses the remaining review finding: automation work-item execution
prompts are scan-agent output over external sources, so they no longer
ride as a fully trusted instruction channel. They are delivered inside an
escaped automation_execution_guidance block that the shared policy limits
to the named work item, unable to expand scope or authorize new actions.

Also extends the untrusted-content framing to the surfaces that arrived
with the develop merge: the Triage GitHub Issues prompt now escapes the
issue title, labels, author, and URL inside task_context, delimits the
issue body, and carries the policy; repeat-mention issue follow-up
messages get the same mention-request framing as the initial mention.
@mrubens

mrubens commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Conceded the executionPrompt finding in b134235: the reviewer is right that appending a policy naming automation execution instructions as a trusted source would have blessed an injected prompt submitted through the scan path. Execution prompts are now delivered inside an escaped <automation_execution_guidance> block, and the policy scopes that guidance to the named work item: it cannot expand scope, change targets, or authorize new actions. The bootstrap skill and work-item fields remain the trusted spine of the prompt.

The develop merge also brought in the Triage GitHub Issues automation, which is exactly the surface this PR exists for, so the same commit hardens it: buildIssueFixerFixPrompt now escapes the issue title, labels, author, and URL inside task_context (previously a forged </issue></task_context> in an issue title could break the prompt structure), wraps the body in an untrusted-content block, and carries the policy. Repeat-mention issue follow-ups get the same mention-request framing.

@mrubens
mrubens merged commit 0e7a915 into develop Jul 19, 2026
17 checks passed
@mrubens
mrubens deleted the harden-untrusted-prompt-framing branch July 19, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant