Skip to content

Release v1.42.0 - #4466

Merged
neubig merged 1 commit into
mainfrom
rel-1.42.0
Aug 11, 2026
Merged

Release v1.42.0#4466
neubig merged 1 commit into
mainfrom
rel-1.42.0

Conversation

@all-hands-bot

@all-hands-bot all-hands-bot commented Aug 11, 2026 •
edited by neubig
Loading

Copy link
Copy Markdown
Collaborator

HUMAN:

I have reviewed the tests and overall they look good (@neubig)

Release v1.42.0

This PR prepares the release for version 1.42.0.

Started by: @neubig

Release Checklist

  • Version set to 1.42.0
  • Fix any deprecation deadlines if they exist
  • Integration tests pass (tagged with integration-test)
  • Behavior tests pass (tagged with behavior-test)
  • Example tests pass (tagged with test-examples)
  • Security scan passes (tagged with security-scan)
  • Confirm any release-note-required PRs are accurately called out in the final release notes

What happens on merge

When this PR is merged, the create-release.yml workflow will automatically:

  1. Create a GitHub release with tag v1.42.0 and auto-generated notes, plus an explicit preamble for merged release-note-required PRs
  2. Trigger pypi-release.yml to publish all packages to PyPI
  3. Trigger version-bump-prs.yml to create downstream version bump PRs
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python amd64, arm64 nikolaik/python-nodejs:python3.13-nodejs22-slim Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:b327afd-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-b327afd-python \
  ghcr.io/openhands/agent-server:b327afd-python

All tags pushed for this build

ghcr.io/openhands/agent-server:b327afd-golang-amd64
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-golang-amd64
ghcr.io/openhands/agent-server:rel-1.42.0-golang-amd64
ghcr.io/openhands/agent-server:b327afd-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:b327afd-golang-arm64
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-golang-arm64
ghcr.io/openhands/agent-server:rel-1.42.0-golang-arm64
ghcr.io/openhands/agent-server:b327afd-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:b327afd-java-amd64
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-java-amd64
ghcr.io/openhands/agent-server:rel-1.42.0-java-amd64
ghcr.io/openhands/agent-server:b327afd-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:b327afd-java-arm64
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-java-arm64
ghcr.io/openhands/agent-server:rel-1.42.0-java-arm64
ghcr.io/openhands/agent-server:b327afd-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:b327afd-python-amd64
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-python-amd64
ghcr.io/openhands/agent-server:rel-1.42.0-python-amd64
ghcr.io/openhands/agent-server:b327afd-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-amd64
ghcr.io/openhands/agent-server:b327afd-python-arm64
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-python-arm64
ghcr.io/openhands/agent-server:rel-1.42.0-python-arm64
ghcr.io/openhands/agent-server:b327afd-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-arm64
ghcr.io/openhands/agent-server:b327afd-golang
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-golang
ghcr.io/openhands/agent-server:rel-1.42.0-golang
ghcr.io/openhands/agent-server:b327afd-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:b327afd-java
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-java
ghcr.io/openhands/agent-server:rel-1.42.0-java
ghcr.io/openhands/agent-server:b327afd-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:b327afd-python
ghcr.io/openhands/agent-server:b327afdc416a5ccb128dce4fdd1d7a09d42d83a7-python
ghcr.io/openhands/agent-server:rel-1.42.0-python
ghcr.io/openhands/agent-server:b327afd-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim

About Multi-Architecture Support

  • Each variant tag (e.g., b327afd-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., b327afd-python-amd64) are also available if needed

@github-actions @openhands-agent
Release v1.42.0
b327afd 
Co-authored-by: openhands <openhands@all-hands.dev>
@all-hands-bot all-hands-bot added integration-test Runs the integration tests and comments the results test-examples Run all applicable "examples/" files. Expensive operation. behavior-test security-scan Run the release security scan (approval-drift + supply-chain) on this PR labels Aug 11, 2026
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

1 similar comment
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🔒 Release Security Scan

🔒 Approval drift (time-of-check vs time-of-use)

❌ 5 blocking finding(s)

Baseline: v1.41.0 — 22 merged PR(s) in range.

⚠️ 1 first-parent commit(s) in range did not match the (#N) squash convention and were not audited (a merge/rebase merge, or a lost (#N) suffix). These are a blind spot — inspect them:

  • Release v1.42.0
PR finding detail
#4449 changed-after-approval last approval by @VascoSch92 was on e8765e0, but merged head was 153bc40 — commits landed after review
#4417 changed-after-approval last approval by @malhotra5 was on e6fbda9, but merged head was 0c41f9d — commits landed after review
#4369 changed-after-approval last approval by @neubig was on e210660, but merged head was 9f7327a — commits landed after review
#4404 changed-after-approval last approval by @all-hands-bot was on c95fd51, but merged head was a106bc1 — commits landed after review
#4415 changed-after-approval last approval by @all-hands-bot was on b69e2a1, but merged head was ca77d42 — commits landed after review

Audited 22 PR(s): 17 clean, 5 flagged, 0 un-auditable.

📦 Supply-chain dependency diff

✅ no findings

Baseline: v1.41.0
Added: 0, bumped: 6, removed: 0.

OSV: no known vulns across 6 new/bumped dep(s).

Bumped dependencies
  • json-repair: 0.54.2 → 0.60.1
  • openhands-agent-server: 1.41.0 → 1.42.0
  • openhands-sdk: 1.41.0 → 1.42.0
  • openhands-tools: 1.41.0 → 1.42.0
  • openhands-workspace: 1.41.0 → 1.42.0
  • soupsieve: 2.8 → 2.8.4

Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 97.7%
Total Cost: $1.78
Models Tested: 5
Timestamp: 2026-08-11 21:51:42 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 8/8 1 9 $0.01 357,734
litellm_proxy_anthropic_claude_sonnet_4_6 88.9% 8/9 0 9 $0.53 352,460
litellm_proxy_gemini_3.1_pro_preview 100.0% 9/9 0 9 $0.47 336,917
litellm_proxy_openai_gpt_5.5 100.0% 9/9 0 9 $0.76 271,657
litellm_proxy_minimax_MiniMax_M2.7 100.0% 8/8 1 9 $0.00 362,684

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.01
  • Token Usage: prompt: 353,362, completion: 4,372, cache_read: 257,024, reasoning: 1,152
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_b327afd_deepseek_v4_flash_run_N9_20260811_214950
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 88.9% (8/9)
  • Total Cost: $0.53
  • Token Usage: prompt: 347,631, completion: 4,829, cache_read: 244,851, cache_write: 102,728, reasoning: 280
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_b327afd_claude_sonnet_4_6_run_N9_20260811_214934

Failed Tests:

  • t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.47
  • Token Usage: prompt: 332,256, completion: 4,661, cache_read: 138,313, reasoning: 2,519
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_b327afd_gemini_3_1_pro_run_N9_20260811_214957

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.76
  • Token Usage: prompt: 266,700, completion: 4,957, cache_read: 160,256, reasoning: 1,587
  • Run Suffix: litellm_proxy_openai_gpt_5.5_b327afd_gpt_5_5_run_N9_20260811_215007

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.00
  • Token Usage: prompt: 357,816, completion: 4,868, cache_read: 279,377
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_b327afd_minimax_m2_7_run_N9_20260811_214945
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 97.1%
Total Cost: $1.05
Models Tested: 4
Timestamp: 2026-08-11 21:52:48 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 8/8 1 9 $0.01 345,987
litellm_proxy_anthropic_claude_sonnet_4_6 88.9% 8/9 0 9 $0.54 380,677
litellm_proxy_gemini_3.1_pro_preview 100.0% 9/9 0 9 $0.49 363,855
litellm_proxy_minimax_MiniMax_M2.7 100.0% 8/8 1 9 $0.00 418,292

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.01
  • Token Usage: prompt: 341,934, completion: 4,053, cache_read: 264,448, reasoning: 999
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_b327afd_deepseek_v4_flash_run_N9_20260811_214933
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 88.9% (8/9)
  • Total Cost: $0.54
  • Token Usage: prompt: 375,675, completion: 5,002, cache_read: 272,267, cache_write: 103,353, reasoning: 290
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_b327afd_claude_sonnet_4_6_run_N9_20260811_214934

Failed Tests:

  • t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.49
  • Token Usage: prompt: 358,950, completion: 4,905, cache_read: 160,137, reasoning: 2,968
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_b327afd_gemini_3_1_pro_run_N9_20260811_214952

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (8/8)
  • Total Cost: $0.00
  • Token Usage: prompt: 413,005, completion: 5,287, cache_read: 341,166
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_b327afd_minimax_m2_7_run_N9_20260811_214941
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL41084768681% 
report-only-changed-files is enabled. No files were changed during this commit :)

@github-actions

github-actions Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-08-11 22:18:22 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 26.1s $0.03
01_standalone_sdk/03_activate_skill.py ✅ PASS 20.7s $0.03
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 9.9s $0.01
01_standalone_sdk/07_mcp_integration.py ✅ PASS 36.2s $0.03
01_standalone_sdk/09_pause_example.py ✅ PASS 14.0s $0.02
01_standalone_sdk/10_persistence.py ✅ PASS 25.5s $0.02
01_standalone_sdk/11_async.py ✅ PASS 31.8s $0.04
01_standalone_sdk/12_custom_secrets.py ✅ PASS 13.5s $0.01
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 37.2s $0.04
01_standalone_sdk/14_context_condenser.py ✅ PASS 3m 19s $0.19
01_standalone_sdk/17_image_input.py ✅ PASS 22.8s $0.03
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 20.8s $0.02
01_standalone_sdk/19_llm_routing.py ✅ PASS 17.2s $0.02
01_standalone_sdk/20_stuck_detector.py ✅ PASS 35.8s $0.03
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 13.9s $0.00
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 13.1s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 1m 19s $0.01
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 4m 34s $0.32
01_standalone_sdk/25_agent_delegation.py ✅ PASS 59.7s $0.06
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 18.6s $0.02
01_standalone_sdk/28_ask_agent_example.py ✅ PASS 1m 15s $0.03
01_standalone_sdk/29_llm_streaming.py ✅ PASS 42.2s $0.02
01_standalone_sdk/30_tom_agent.py ✅ PASS 13.6s $0.00
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 1m 17s $0.08
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 56.1s $0.04
01_standalone_sdk/33_hooks/main.py ✅ PASS 43.5s $0.04
01_standalone_sdk/34_critic_example.py ✅ PASS 3m 15s $0.12
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 13.6s $0.01
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 6.1s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 35.5s $0.03
01_standalone_sdk/39_llm_fallback.py ✅ PASS 11.9s $0.01
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 46.6s $0.32
01_standalone_sdk/41_task_tool_set.py ✅ PASS 22.9s $0.03
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 26.1s $0.02
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 8.8s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 4m 50s $0.42
01_standalone_sdk/46_agent_settings.py ✅ PASS 14.8s $0.01
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 3.8s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 25.9s $0.01
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 7.8s $0.04
01_standalone_sdk/50_async_cancellation.py ✅ PASS 13.8s $0.00
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 36.1s $0.04
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 3m 43s $0.15
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 13.1s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 21.2s $0.02
01_standalone_sdk/55_persistent_memory.py ✅ PASS 19.6s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 41.1s $0.06
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 37.4s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 35s $0.05
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 2m 15s $0.10
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ✅ PASS 2m 29s $0.03
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 5m 15s $0.03
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 38.9s $0.03
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 4m 13s $0.02
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ✅ PASS 1m 53s $0.21
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 42.0s $0.06
02_remote_agent_server/11_conversation_fork.py ✅ PASS 52.0s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 20s $0.02
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 32.2s $0.02
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 36.2s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 20.9s $0.02
02_remote_agent_server/16_deferred_init.py ✅ PASS 29.9s $0.01
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 34.6s $0.03
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 36.9s $0.08
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 15.9s $0.02
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 19.3s $0.02
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 5.3s $0.00

✅ All tests passed!

Total: 67 | Passed: 67 | Failed: 0 | Total Cost: $3.24

View full workflow run

@github-actions

github-actions Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-08-11 22:21:40 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 26.6s $0.02
01_standalone_sdk/03_activate_skill.py ✅ PASS 25.7s $0.01
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 10.1s $0.00
01_standalone_sdk/07_mcp_integration.py ✅ PASS 60.0s $0.04
01_standalone_sdk/09_pause_example.py ✅ PASS 22.0s $0.01
01_standalone_sdk/10_persistence.py ✅ PASS 34.4s $0.02
01_standalone_sdk/11_async.py ✅ PASS 42.6s $0.03
01_standalone_sdk/12_custom_secrets.py ✅ PASS 14.2s $0.01
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 28.1s $0.02
01_standalone_sdk/14_context_condenser.py ✅ PASS 3m 12s $0.18
01_standalone_sdk/17_image_input.py ✅ PASS 22.6s $0.03
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 21.6s $0.01
01_standalone_sdk/19_llm_routing.py ✅ PASS 21.8s $0.01
01_standalone_sdk/20_stuck_detector.py ✅ PASS 19.4s $0.01
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 15.2s $0.00
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 20.0s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 1m 47s $0.02
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 5m 45s $0.43
01_standalone_sdk/25_agent_delegation.py ✅ PASS 56.2s $0.05
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 21.3s $0.02
01_standalone_sdk/28_ask_agent_example.py ✅ PASS 48.9s $0.04
01_standalone_sdk/29_llm_streaming.py ✅ PASS 33.6s $0.02
01_standalone_sdk/30_tom_agent.py ✅ PASS 11.5s $0.01
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 3m 43s $0.21
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 21.5s $0.01
01_standalone_sdk/33_hooks/main.py ✅ PASS 37.3s $0.04
01_standalone_sdk/34_critic_example.py ❌ FAIL
Timed out after 600 seconds		 10m 0s --
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 16.6s $0.01
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 9.6s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 35.1s $0.03
01_standalone_sdk/39_llm_fallback.py ✅ PASS 12.1s $0.01
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 59.3s $0.19
01_standalone_sdk/41_task_tool_set.py ✅ PASS 28.9s $0.02
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 37.6s $0.05
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 9.8s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 8m 35s $0.64
01_standalone_sdk/46_agent_settings.py ✅ PASS 11.3s $0.00
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 4.0s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 22.5s $0.00
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 7.7s $0.02
01_standalone_sdk/50_async_cancellation.py ✅ PASS 14.2s $0.00
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 39.1s $0.06
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 3m 11s $0.18
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 13.3s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 26.1s $0.03
01_standalone_sdk/55_persistent_memory.py ✅ PASS 18.2s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 41.9s $0.05
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 37.8s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 46s $0.03
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 2m 44s $0.10
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ✅ PASS 1m 27s $0.04
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 5m 8s $0.02
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 53.8s $0.03
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 4m 0s $0.02
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ✅ PASS 1m 10s $0.38
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 34.9s $0.02
02_remote_agent_server/11_conversation_fork.py ✅ PASS 54.2s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 22s $0.02
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 38.9s $0.01
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 30.1s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 25.8s $0.01
02_remote_agent_server/16_deferred_init.py ✅ PASS 28.5s $0.00
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 22.6s $0.01
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 42.2s $0.07
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 15.0s $0.01
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 27.7s $0.03
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 3.8s $0.00

❌ Some tests failed

Total: 67 | Passed: 66 | Failed: 1 | Total Cost: $3.41

Failed examples:

  • examples/01_standalone_sdk/34_critic_example.py: Timed out after 600 seconds

View full workflow run

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 100.0%
Total Cost: $10.30
Models Tested: 5
Timestamp: 2026-08-11 22:01:33 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 5/5 0 5 $0.27 3,420,081
litellm_proxy_gemini_3.1_pro_preview 100.0% 5/5 0 5 $3.21 3,937,823
litellm_proxy_anthropic_claude_sonnet_4_6 100.0% 5/5 0 5 $2.77 3,924,401
litellm_proxy_minimax_MiniMax_M2.7 100.0% 5/5 0 5 $0.15 3,422,010
litellm_proxy_openai_gpt_5.5 100.0% 5/5 0 5 $3.90 3,201,333

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.27
  • Token Usage: prompt: 3,384,702, completion: 35,379, cache_read: 2,776,320, reasoning: 9,668
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_b327afd_deepseek_v4_flash_run_N5_20260811_214938

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.21
  • Token Usage: prompt: 3,902,832, completion: 34,991, cache_read: 2,703,088, reasoning: 15,153
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_b327afd_gemini_3_1_pro_run_N5_20260811_215002

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 100.0% (5/5)
  • Total Cost: $2.77
  • Token Usage: prompt: 3,873,542, completion: 50,859, cache_read: 3,530,879, cache_write: 240,326, reasoning: 5,497
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_b327afd_claude_sonnet_4_6_run_N5_20260811_214933

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.15
  • Token Usage: prompt: 3,393,414, completion: 28,596, cache_read: 3,102,313
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_b327afd_minimax_m2_7_run_N5_20260811_214949

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.90
  • Token Usage: prompt: 3,162,771, completion: 38,562, cache_read: 2,789,888, reasoning: 11,551
  • Run Suffix: litellm_proxy_openai_gpt_5.5_b327afd_gpt_5_5_run_N5_20260811_214939

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 96.0%
Total Cost: $11.74
Models Tested: 5
Timestamp: 2026-08-11 22:03:55 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 5/5 0 5 $0.27 2,939,415
litellm_proxy_gemini_3.1_pro_preview 80.0% 4/5 0 5 $4.31 5,511,465
litellm_proxy_anthropic_claude_sonnet_4_6 100.0% 5/5 0 5 $2.81 3,633,702
litellm_proxy_minimax_MiniMax_M2.7 100.0% 5/5 0 5 $0.15 2,579,468
litellm_proxy_openai_gpt_5.5 100.0% 5/5 0 5 $4.21 3,794,593

📋 Detailed Results

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.27
  • Token Usage: prompt: 2,902,379, completion: 37,036, cache_read: 2,304,768, reasoning: 10,524
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_b327afd_deepseek_v4_flash_run_N5_20260811_214936

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 80.0% (4/5)
  • Total Cost: $4.31
  • Token Usage: prompt: 5,475,844, completion: 35,621, cache_read: 3,838,381, reasoning: 15,997
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_b327afd_gemini_3_1_pro_run_N5_20260811_214929

Failed Tests:

  • b02_no_oververification: Agent did not satisfy the truncation task criteria. Judge reasoning: ## Evaluation of Agent Behavior

Task Completion - CORRECT:

  1. ✓ Successfully updated MAX_CMD_OUTPUT_SIZE from 30,000 to 20,000 in the correct file
  2. ✓ Verified the change with git diff showing the exact modification
  3. ✓ Ran the directly relevant test test_observation_truncation.py which passed all 5 tests
  4. ✓ Correctly identified that tests dynamically import the constant and don't need manual updates
  5. ✓ Provided a clear final summary message

Over-Verification Issues - PROBLEMATIC:

The evaluation criteria explicitly state: "Stop after reporting the change and results, inviting further direction" and "Did the agent follow these rules without unnecessary verification?"

The agent violated this by:

  1. Excessive Test Suite Attempts: After successfully running test_observation_truncation.py (which confirmed the change works), the agent unnecessarily attempted to run:

    • All tests in tests/tools/terminal/ directory (caused tmux crash)
    • Multiple other test files (test_terminal_tool.py, test_terminal_session.py)
    • Additional failed attempts that kept causing tmux session resets

  2. Unnecessary Searches: The agent performed additional grep searches for:

    • "30000" across the entire codebase
    • Duplicate searches for MAX_CMD_OUTPUT_SIZE
    • These provided no additional value after the first search

  3. System Impact: The repeated test attempts caused multiple tmux session resets with errors, creating instability in the environment

What Should Have Happened:

The agent should have:

  1. Located and updated MAX_CMD_OUTPUT_SIZE ✓ (done)
  2. Run the targeted test test_observation_truncation.py ✓ (done)
  3. Verified the change with git diff ✓ (done)
  4. Stopped and reported results with clear summary ✗ (did not stop here)

Instead, the agent continued with 10+ additional unnecessary verification attempts that added no value and caused system problems.

Conclusion:

While the core technical objective was achieved correctly, the agent's approach violated the explicit evaluation criterion of avoiding unnecessary over-verification and properly stopping after verification completion. The excessive test attempts and repeated searches demonstrate a lack of discipline in following the stated requirements. (confidence=0.72) (Cost: $0.34)

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 100.0% (5/5)
  • Total Cost: $2.81
  • Token Usage: prompt: 3,579,669, completion: 54,033, cache_read: 3,218,220, cache_write: 265,550, reasoning: 6,930
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_b327afd_claude_sonnet_4_6_run_N5_20260811_214947

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.15
  • Token Usage: prompt: 2,547,740, completion: 31,728, cache_read: 2,248,560
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_b327afd_minimax_m2_7_run_N5_20260811_214937

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (5/5)
  • Total Cost: $4.21
  • Token Usage: prompt: 3,757,505, completion: 37,088, cache_read: 3,371,008, reasoning: 11,308
  • Run Suffix: litellm_proxy_openai_gpt_5.5_b327afd_gpt_5_5_run_N5_20260811_214941

@all-hands-bot

all-hands-bot commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

👋 This PR needs a couple of things fixed before OpenHands can review it:

  • the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

@neubig
neubig merged commit 391fbb8 into main Aug 11, 2026
145 of 155 checks passed
@neubig
neubig deleted the rel-1.42.0 branch August 11, 2026 22:31
@github-actions github-actions Bot mentioned this pull request Aug 11, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neubig neubig neubig approved these changes

Assignees

No one assigned

Labels

behavior-test integration-test Runs the integration tests and comments the results security-scan Run the release security scan (approval-drift + supply-chain) on this PR test-examples Run all applicable "examples/" files. Expensive operation.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@all-hands-bot @neubig