Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
360 changes: 350 additions & 10 deletions Cargo.lock

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,15 @@ h2 = "=0.4.17"
httpdate = "=1.0.3"
hyper = "=1.11.0"
hyper-util = { version = "=0.1.20", features = ["server", "http1", "http2", "tokio"] }
image = { version = "=0.25.10", default-features = false, features = ["gif", "jpeg", "png", "webp"] }
jsonwebtoken = { version = "=11.0.0", default-features = false, features = ["aws_lc_rs"] }
keyring = { version = "=4.1.6", default-features = false, features = ["v1"] }
jsonschema = { version = "=0.50.1", default-features = false }
opentelemetry = { version = "=0.32.0", default-features = false, features = ["trace"] }
opentelemetry-otlp = { version = "=0.32.0", default-features = false, features = ["grpc-tonic", "tls-webpki-roots", "trace"] }
opentelemetry_sdk = { version = "=0.32.1", default-features = false, features = ["trace"] }
ratatui = { version = "=0.30.2", default-features = false, features = ["crossterm", "layout-cache"] }
ratatui-image = { version = "=11.0.6", default-features = false, features = ["crossterm"] }
reqwest = { version = "=0.13.4", default-features = false, features = ["blocking", "form", "rustls", "stream"] }
rmcp = { version = "=3.1.4", default-features = false, features = ["auth", "client"] }
runlet = "=0.4.0"
Expand Down
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -631,8 +631,11 @@ only when every pasted token is a supported file; otherwise it preserves the
whole paste as text. A prompt can contain at most 8 attachments, 10 MiB each
and 20 MiB total. Kit sends the bytes through OpenRouter or OpenAI subscription
while retaining canonical local `file://` links in model-facing text. Model
modality support varies. Video, terminal image rendering, and audio playback are
not supported, and Kit never displays base64 or `data:` URLs. See
modality support varies. In terminals detected as supporting Kitty, Sixel, or
iTerm2 graphics, user-attached images render inline as a bounded static first
frame; other terminals and decode failures keep the safe clickable attachment
label. Video and audio playback are not supported, and Kit never displays
base64 or `data:` URLs. See
[the TUI guide](docs/user/tui-and-sessions.md#attach-local-images-and-audio).

## Deliberate limits
Expand Down
2 changes: 1 addition & 1 deletion docs/user/tui-and-sessions.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ An accepted file appears in the editor as `[Image #N]` or `[Audio #N]`. Add surr

The model-facing prompt retains canonical `file://` Markdown links, while Kit also reads and sends the file bytes because remote providers cannot access local files. Image and audio acceptance remains model-dependent. Kit supports these request shapes through OpenRouter and OpenAI subscription; an individual model can still reject a modality it does not support. Video is not supported.

Assistant- and tool-produced media appears as portable Markdown placeholders or links. Kit does not render images in the terminal or play audio. Only bounded `file://`, `http://`, and `https://` links are displayed; base64 and `data:` URLs are never copied into terminal text or Markdown links.
User-attached images render inline as a bounded static first frame when Kit detects Kitty, Sixel, or iTerm2 graphics support. No setting is required. Unsupported terminals, malformed or oversized images, and decode failures retain the safe clickable attachment label. Animated GIF and WebP files currently show only their first frame. Assistant- and tool-produced media remains portable Markdown placeholders or links, and audio is not played. Only bounded `file://`, `http://`, and `https://` links are displayed; base64 and `data:` URLs are never copied into terminal text or Markdown links.

### Interrupt a running turn or quit

Expand Down
49 changes: 28 additions & 21 deletions src/protocols/acp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -160,24 +160,15 @@ pub(super) fn tool_output_raw(output: &ToolOutput) -> Option<serde_json::Value>
}

fn media_replay_content(media: &MediaPart) -> ContentBlock {
match media.modality {
Modality::Image
if matches!(media.data, DataRef::InlineText(_) | DataRef::InlineBytes(_)) =>
{
ContentBlock::Image(ImageContent::new(
data_ref_base64_payload(&media.data),
media.mime_type.clone(),
))
let payload = data_ref_base64_payload(&media.data);
match (media.modality, payload) {
(Modality::Image, Some(payload)) => {
ContentBlock::Image(ImageContent::new(payload, media.mime_type.clone()))
}
Modality::Audio
if matches!(media.data, DataRef::InlineText(_) | DataRef::InlineBytes(_)) =>
{
ContentBlock::Audio(AudioContent::new(
data_ref_base64_payload(&media.data),
media.mime_type.clone(),
))
(Modality::Audio, Some(payload)) => {
ContentBlock::Audio(AudioContent::new(payload, media.mime_type.clone()))
}
Modality::Image | Modality::Audio | Modality::Video | Modality::Binary => {
(Modality::Image | Modality::Audio | Modality::Video | Modality::Binary, _) => {
data_ref_replay_content(None, Some(&media.mime_type), &media.data)
}
}
Expand Down Expand Up @@ -218,7 +209,7 @@ fn data_ref_replay_content(
}
_ => {
let mut resource = BlobResourceContents::new(
data_ref_base64_payload(data),
data_ref_base64_payload(data).unwrap_or_default(),
format!("agentkit://session-replay/{}", name.unwrap_or("content")),
);
if let Some(mime_type) = mime_type {
Expand All @@ -231,13 +222,14 @@ fn data_ref_replay_content(
}
}

fn data_ref_base64_payload(data: &DataRef) -> String {
fn data_ref_base64_payload(data: &DataRef) -> Option<String> {
match data {
DataRef::InlineText(text) => {
data_url_base64_payload(text).unwrap_or_else(|| BASE64.encode(text.as_bytes()))
Some(data_url_base64_payload(text).unwrap_or_else(|| BASE64.encode(text.as_bytes())))
}
DataRef::InlineBytes(bytes) => BASE64.encode(bytes),
DataRef::Uri(_) | DataRef::Handle(_) => String::new(),
DataRef::InlineBytes(bytes) => Some(BASE64.encode(bytes)),
DataRef::Uri(uri) => data_url_base64_payload(uri),
DataRef::Handle(_) => None,
}
}

Expand Down Expand Up @@ -1754,6 +1746,21 @@ mod tests {
);
}

#[test]
fn replay_restores_data_url_images_as_image_content() {
let part = Part::media(
Modality::Image,
"image/png",
DataRef::uri("data:image/png;base64,AQID"),
);
let chunk = user_replay_content(&part).expect("image replay content");

assert!(matches!(
chunk.content,
ContentBlock::Image(image) if image.data == "AQID"
));
}

#[test]
fn transcript_replay_preserves_order_and_skips_unrepresentable_history() {
let transcript = vec![
Expand Down
28 changes: 27 additions & 1 deletion src/protocols/acp/v2.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1724,7 +1724,8 @@ pub(crate) fn component(
mod tests {
use serde_json::json;

use agentkit_core::{MetadataMap, TurnCancellation};
use agent_client_protocol::schema::MaybeUndefined;
use agentkit_core::{DataRef, MetadataMap, Modality, TurnCancellation};
use agentkit_loop::{
Agent, ModelAdapter, ModelTurn, ModelTurnEvent, ModelTurnResult, SessionConfig,
TurnRequest, TurnResult,
Expand Down Expand Up @@ -2731,6 +2732,31 @@ mod tests {
));
}

#[test]
fn replay_preserves_data_url_user_images() {
let replay = transcript_replay(
&wire::SessionId::new("saved"),
&[Item::new(
ItemKind::User,
vec![Part::media(
Modality::Image,
"image/png",
DataRef::uri("data:image/png;base64,AQID"),
)],
)],
);
let wire::SessionUpdate::UserMessage(message) = &replay[0].update else {
panic!("expected user message");
};
let MaybeUndefined::Value(content) = &message.content else {
panic!("expected user content");
};
assert!(matches!(
content.as_slice(),
[wire::ContentBlock::Image(image)] if image.data == "AQID"
));
}

#[test]
fn v2_config_mapping_uses_v2_ids_categories_and_values() {
let current =
Expand Down
Loading