From de268f8e134b1042953b3afb2842f7d4f66c38fc Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:31:39 -0700 Subject: [PATCH 1/7] ops: weekly ops review 2026-08-17 --- .ops/weekly-reviews/2026-08-17.md | 136 ++++++++++++++++++ .../artifacts/2026-08-17-release-rate.svg | 20 +++ 2 files changed, 156 insertions(+) create mode 100644 .ops/weekly-reviews/2026-08-17.md create mode 100644 .ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md new file mode 100644 index 0000000000..612dcf283e --- /dev/null +++ b/.ops/weekly-reviews/2026-08-17.md @@ -0,0 +1,136 @@ +# [Kiro-CLI] Weekly Ops Review - 08/17/2026 + +**On Call:** zoelin | **Next:** dingfeli | **Previous:** shhvin + +## 1. Summary +* Pages: 8 across 8 tickets (degraded engagements fallback; per-ticket notification counts and outside-hours counts unavailable) +* [Ticket Queue](https://t.corp.amazon.com/issues/?q=extensions.tt.status%3A%28Assigned%20OR%20Researching%20OR%20%22Work%20In%20Progress%22%20OR%20Pending%29%20AND%20extensions.tt.assignedGroup%3A%22Amazon%20Q%20for%20CLI%22): 153 → 153 +* Incoming Tickets: 16 (21 raw; two QCLI-SuccessRateDown tickets collapse to one issue, four identical invalid-leg SmokeBot tickets collapse to one, and two identical inconsistent-verdict tickets collapse to one) +* Resolved Tickets: 16 (1 Sev2, 15 lower) +* Large Scale Significant Events: 0 +* Releases Shipped: 1 (v2.18.0) | Avg 3.0 releases/week (trailing 4 weeks) + +![Stable release rate per week](artifacts/2026-08-17-release-rate.svg) + +_Reliable tag data starts 2026-05-01; earlier months used chat-prefixed tagging and are not comparable._ + +## 2. Graphs +### Ticket Resolved Count By Root Cause (Previous week) + +_Split by severity, Sev2 first. Severity is the ticket's **current** severity, so tickets that paged at Sev2 and were later downgraded appear in the Non-Sev2 table._ + +**Sev2 (1)** + +| Root Cause | Count | Topic | Tickets | +|---|---|---|---| +| Working as Intended / no issue | 1 | Security review closed after validation concluded Kiro did not access the credential store | [P473397715](https://t.corp.amazon.com/P473397715) | +| Total | 1 | | | + +**Non-Sev2 (15)** + +| Root Cause | Count | Topic | Tickets | +|---|---|---|---| +| Test infrastructure failure | 4 | SmokeBot judge emitted invalid or internally inconsistent leg results without evidence of a CLI defect; ticket fields labeling two as release tickets were overridden by their descriptions | [P487800125](https://t.corp.amazon.com/P487800125), [P490847964](https://t.corp.amazon.com/P490847964), [P491197513](https://t.corp.amazon.com/P491197513), [P493245390](https://t.corp.amazon.com/P493245390) | +| Release | 3 | Release tracking tickets closed or superseded this week; release tickets closed are not the same unit as releases shipped | [P486126350](https://t.corp.amazon.com/P486126350), [P487400785](https://t.corp.amazon.com/P487400785), [P489150075](https://t.corp.amazon.com/P489150075) | +| Duplicate | 3 | Two OpenSSH 7.4 SetEnv reports consolidated into V2126340432; auto-scroll report consolidated into D480144322 | [P403116931](https://t.corp.amazon.com/P403116931), [P443070433](https://t.corp.amazon.com/P443070433), [P461034891](https://t.corp.amazon.com/P461034891) | +| Downstream dependency issue | 2 | First-token latency and success-rate alarms resolved after downstream model-serving recovery | [V2321344948](https://t.corp.amazon.com/V2321344948), [V2322113921](https://t.corp.amazon.com/V2322113921) | +| Product defect - subagent model handling | 1 | Dotted model version was rewritten during subagent dispatch, breaking the selected model | [P472601597](https://t.corp.amazon.com/P472601597) | +| Feature request / capability gap | 1 | GPT-5.6 Sol readable reasoning summaries are not currently supported over ACP | [P492031704](https://t.corp.amazon.com/P492031704) | +| Working as Intended - unsupported platform | 1 | Debian 12 on VirtualBox is outside the supported macOS, Ubuntu, and Cloud Desktop platforms | [V2184334418](https://t.corp.amazon.com/V2184334418) | +| Total | 15 | | | + +## 3. Operational Pain Level + +_Pick one (1–10) during the ops review._ + +**Suggested: 7/10** — eight paged tickets included six customer-impacting Mantle overload episodes under one reused alarm ticket, while OAuth cases and release work added sustained follow-up load. This is a suggestion for the oncall to confirm during review. + +## 4. Large Scale Significant Events +* None + +## 5. Previous Week's Action Items + +During the meeting, go over open action items here https://tiny.amazon.com/1auvbeoty/taskamazdevroom7c22task + +Candidate carried-forward items: +* **Closed** — [P473397715](https://t.corp.amazon.com/P473397715): security review resolved as no issue after validation concluded Kiro did not access the credential store. +* **Open** — [P488415895](https://t.corp.amazon.com/P488415895): fix ListAvailableProfiles credential selection and confirm whether KAS shares the 2.13.0 application-loop behavior. +* **Open** — [P491128276](https://t.corp.amazon.com/P491128276): track durable Mantle gateway/autoscaling and availability-alarm improvements after the repeated cell3 overload episodes. + +## 6. Page Log + +_One row per paged ticket. Because Builder Insights was unavailable, this uses the degraded engagements fallback: it identifies paged tickets but overwrites notification history, so every Pages value is 1 and repeated notifications cannot be recovered. A † timestamp is the latest retained alias engagement rather than a guaranteed first notification._ + +| # | Ticket | Pages | First Page (PT) | Page Sequence | Impact Summary | Root Cause | Mitigation | Action Items | Risk of Recurrence | Related Tickets | +|---|---|---|---|---|---|---|---|---|---|---| +| 1 | [P488415895](https://t.corp.amazon.com/P488415895) | 1 | 08-10 09:30† | Retained engagement | Kiro CLI 2.13.0 generated about 590K guaranteed-fail ListAvailableProfiles calls/day; no customer or service impact was observed | Version 2.13.0 sent API-key credentials to an operation that accepts only OIDC/SSO bearer tokens, then repeated the call in an application loop | Ticket was downgraded while the API-key owner and client path are investigated; the auth layer correctly rejected the invalid token type | Fix the client credential selection and confirm whether the same behavior exists in KAS | Medium - legacy 2.13.0 clients continue generating high-volume failed traffic | [V2310285730](https://t.corp.amazon.com/V2310285730) | +| 2 | [V2321344948](https://t.corp.amazon.com/V2321344948) | 1 | 08-10 20:00 | New Sev2 | QCLI first-token latency exceeded 25 seconds for 14 of 15 one-minute datapoints | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | +| 3 | [V2322113921](https://t.corp.amazon.com/V2322113921) | 1 | 08-11 10:26 | New Sev2 | QCLI success rate fell below 99% for two of three five-minute datapoints | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | +| 4 | [V2322698178](https://t.corp.amazon.com/V2322698178) | 1 | 08-11 19:54 | Retained engagement† | Six overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m; observed affected-user counts reached 1,656 in one episode | Mantle us-east-1 cell3 shared-gateway saturation produced 529/500 failures across Claude models; autoscaling was blind to the Atlas-path pressure | Mantle rebalanced four Kiro accounts from cell3 to cell2 and service recovered | Track durable gateway/autoscaling and 500-only alarm fixes in P491128276 and P491374811 | High - six episodes occurred and the engagement fallback overwrote earlier notification history | [P491128276](https://t.corp.amazon.com/P491128276), [P491374811](https://t.corp.amazon.com/P491374811), [V2317183654](https://t.corp.amazon.com/V2317183654) | +| 5 | [V2314292550](https://t.corp.amazon.com/V2314292550) | 1 | 08-12 09:29† | Upgraded | CSIRO MCP tool results silently lost structuredContent, so agents could miss structured data and substitute generic answers | Both CLI paths intentionally omit structuredContent when a non-empty content block is present in the shared tool-result conversion helper | Behavior and a server-side workaround were explained; ticket was downgraded as a product capability gap | Decide whether to preserve structuredContent alongside content in model context | Medium - every server returning both fields is affected until behavior changes | [P485942261](https://t.corp.amazon.com/P485942261) | +| 6 | [V2313145792](https://t.corp.amazon.com/V2313145792) | 1 | 08-12 09:30† | Upgraded | FINRA could not complete remote MCP OAuth from native Windows PowerShell; Kiro IDE worked with the same configuration | CLI clipboard detection did not recognize the available PowerShell 7 clipboard command and failed to surface the OAuth URL | IDE/v3 and diagnostic-log URL retrieval were provided as workarounds; 2.18.1 surfaces the URL when clipboard copy fails | Await customer confirmation on 2.18.1 and separately investigate WSL unauthorized responses if they persist | Low after 2.18.1; WSL authorization may be a separate issue | [V2311766396](https://t.corp.amazon.com/V2311766396) | +| 7 | [V2311766396](https://t.corp.amazon.com/V2311766396) | 1 | 08-13 09:29† | Upgraded | Numerix could not authorize a remote MCP server from Kiro CLI on Windows although the same configuration worked in Kiro IDE | The Windows clipboard path failed and the CLI did not visibly expose the OAuth URL | A diagnostic-log workaround was provided; version 2.18.1 surfaces the URL when clipboard handling fails | Await customer confirmation after upgrade to 2.18.1 | Low after 2.18.1 | [V2313145792](https://t.corp.amazon.com/V2313145792) | +| 8 | [V2325742869](https://t.corp.amazon.com/V2325742869) | 1 | 08-14 09:30† | Upgraded | GBST could not authenticate Kiro CLI 2.18 to Atlassian MCP authv2; consent failed and the localhost callback never arrived | The v2 engine added OAuth scopes that make Atlassian authv2's consent metadata request fail with an empty tenant; older CLI and v3 no-scope flows work | Use the v3 engine or non-authv2 endpoint; the v2 fix in PR #4076 merged and awaits release | Ship the v2 OAuth-scope fix and confirm the customer can complete consent | Low once the merged fix is released | None | +| | **Total** | **8** | | | | | | | | | + +_Engagement data retained only one row per alias. [V2322698178](https://t.corp.amazon.com/V2322698178) alone contains nine alarm cuts mapping to six overload episodes, so the headline of 8 pages is a known lower bound, not a recoverable notification total._ + +_High-severity candidates reviewed but not identified as paged by the fallback: [D434404377](https://t.corp.amazon.com/D434404377), [D479638291](https://t.corp.amazon.com/D479638291), [D499857327](https://t.corp.amazon.com/D499857327), [P459152339](https://t.corp.amazon.com/P459152339), [P473397715](https://t.corp.amazon.com/P473397715), [P476098688](https://t.corp.amazon.com/P476098688), [P479056395](https://t.corp.amazon.com/P479056395), [V2243924150](https://t.corp.amazon.com/V2243924150), [V2248275776](https://t.corp.amazon.com/V2248275776), [V2274501749](https://t.corp.amazon.com/V2274501749), [V2277483794](https://t.corp.amazon.com/V2277483794), [V2279916688](https://t.corp.amazon.com/V2279916688), [V2288565081](https://t.corp.amazon.com/V2288565081), [V2299574204](https://t.corp.amazon.com/V2299574204), [V2304775994](https://t.corp.amazon.com/V2304775994), [V2304984688](https://t.corp.amazon.com/V2304984688), [V2305734483](https://t.corp.amazon.com/V2305734483), [V2306322595](https://t.corp.amazon.com/V2306322595), and [V2317754397](https://t.corp.amazon.com/V2317754397). Most were closed, resolved, or downgraded; V2304984688 remains assigned at Sev3._ + +## 7. Open Sev2s + +| # | Ticket | Description | Next Step | ETA To Resolve | +|---|---|---|---|---| +| | None | No tickets are currently at Sev2 or higher in the Amazon Q for CLI open queue | None | N/A | + +## 8. Open Queue by Category (153 open as of 08/17) + + + +| Category | Count | Share | What it covers | +|---|---|---|---| +| Security and trust boundary | 27 | 18% | Security-org reports plus permission, trust-boundary, sandbox, symlink, prompt-injection, and supply-chain defects | +| Auth and connectivity | 19 | 12% | Login, token, OAuth, IdP, proxy, transport, entitlement, and service-identity failures | +| MCP servers and tools | 17 | 11% | MCP registration, discovery, transport, tool surfaces, tool-result handling, and governance | +| TUI and terminal | 16 | 10% | Rendering, viewport, input, key handling, cursor, tmux, and approval display | +| Context and model behaviour | 15 | 10% | Compaction, context accounting, reasoning controls, session integrity, and fabricated calls | +| Stability and crashes | 14 | 9% | Panics, hangs, signals, payload limits, media limits, and transcript invariant failures | +| Release infra and access | 13 | 8% | Release pipelines, artifacts, smoke tests, and repository or organization access | +| Platform and packaging | 11 | 7% | Installation, update, OS integration, shell integration, architecture, and runtime provisioning | +| Subagents and hooks | 10 | 7% | Subagent orchestration, hooks, inherited configuration, and MCP readiness | +| Other | 7 | 5% | IDs: [V2322698178](https://t.corp.amazon.com/V2322698178), [V2317280805](https://t.corp.amazon.com/V2317280805), [V2316809228](https://t.corp.amazon.com/V2316809228), [V2308110975](https://t.corp.amazon.com/V2308110975), [V2304984688](https://t.corp.amazon.com/V2304984688), [V2261880005](https://t.corp.amazon.com/V2261880005), [P423601484](https://t.corp.amazon.com/P423601484) | +| Compliance and measurement | 4 | 3% | Accessibility, certifications, security questionnaires, usage, and efficiency reporting | +| **Total** | **153** | **100%** | | + +_Classification was performed fresh over all 153 currently open tickets. It is a point-in-time composition snapshot and is not comparable to prior weekly classifications; the total can differ from Section 1 because that queue figure is anchored to the handoff boundary while this section reflects report-run time. This week both are 153._ + +**Observations** + +* Security and trust boundary is the largest bucket (27/153). The oldest security items include [P375836923](https://t.corp.amazon.com/P375836923), [P374306511](https://t.corp.amazon.com/P374306511), [P368143082](https://t.corp.amazon.com/P368143082), and [P267248099](https://t.corp.amazon.com/P267248099); a focused owner/ECD review would clear the aged tail. +* Smoke-test infrastructure repeats two patterns: invalid or unverified judge legs ([P492450190](https://t.corp.amazon.com/P492450190), [P491166268](https://t.corp.amazon.com/P491166268), [P490528839](https://t.corp.amazon.com/P490528839), [P487177927](https://t.corp.amazon.com/P487177927)) and artifact-matrix mismatch ([P489952897](https://t.corp.amazon.com/P489952897), [P487504735](https://t.corp.amazon.com/P487504735)); fixing the shared ticketing/judge path would clear both clusters. +* Remote MCP OAuth failures form one customer cluster across GBST, FINRA, and Numerix: [V2325742869](https://t.corp.amazon.com/V2325742869), [V2313145792](https://t.corp.amazon.com/V2313145792), and [V2311766396](https://t.corp.amazon.com/V2311766396). Confirming 2.18.1 and the v2 scope fix with all three customers would close the loop. +* [V2314292550](https://t.corp.amazon.com/V2314292550) and [P485942261](https://t.corp.amazon.com/P485942261) are a customer/internal pair for the same structuredContent behavior and should be managed under one capability decision. + +## 9. Security Risks + +_To be reviewed during the meeting._ + +* Acknowledge High/Critical risks https://policyengine.amazon.com/dashboard/zoelin +* OS Patching https://mirador.security.aws.dev/#/insights/patching?viewingAs=zoelin +* AppSec findings https://mirador.security.aws.dev/#/findings?viewingAs=zoelin +* SAS risks https://sas.corp.amazon.com/summary/all/zoelin + +## 10. Tickets Cut to Other Teams + +| # | Team - Ticket | Description | +|---|---|---| +| 1 | Mantle - [P491128276](https://t.corp.amazon.com/P491128276) | Cross-team investigation of recurring us-east-1 cell3 shared-gateway saturation across Mantle-hosted Claude models. Mantle rebalanced four Kiro accounts to cell2; follow-up alarm and autoscaling work remains tracked separately. | + +_The partner-tag search returned no additional reassigned ticket, and no cross-team CR was found in the reviewed worklogs._ + +## 11. Dashboard Review + +Add dashboard spikes related findings here [[Kiro-CLI] Weekly Dashboard Investigation Notes](https://quip-amazon.com/umwaAzDXcFo1) diff --git a/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg b/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg new file mode 100644 index 0000000000..1f188b557a --- /dev/null +++ b/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg @@ -0,0 +1,20 @@ + + + Stable release rate for the four weeks ending August 17, 2026 + Weekly stable release counts were five, three, three, and one, averaging three releases per week. + + Stable releases per oncall week + Trailing average: 3.0/week + + + + + + + 5331 + + + 07/2708/0308/1008/17 + + Strict semver tags; weeks end at Monday 09:30 PT + From 1d49c22eb3a502a2c1f0c46a97aea7bd74a87ae8 Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:35:59 -0700 Subject: [PATCH 2/7] fix: correct weekly release count --- .ops/weekly-reviews/2026-08-17.md | 4 ++-- .../artifacts/2026-08-17-release-rate.svg | 18 +++++++++--------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md index 612dcf283e..bfbff5c77a 100644 --- a/.ops/weekly-reviews/2026-08-17.md +++ b/.ops/weekly-reviews/2026-08-17.md @@ -8,11 +8,11 @@ * Incoming Tickets: 16 (21 raw; two QCLI-SuccessRateDown tickets collapse to one issue, four identical invalid-leg SmokeBot tickets collapse to one, and two identical inconsistent-verdict tickets collapse to one) * Resolved Tickets: 16 (1 Sev2, 15 lower) * Large Scale Significant Events: 0 -* Releases Shipped: 1 (v2.18.0) | Avg 3.0 releases/week (trailing 4 weeks) +* Releases Shipped: 3 (v2.17.0, v2.18.0, v2.18.1) | Avg 3.3 releases/week (trailing 4 weeks) ![Stable release rate per week](artifacts/2026-08-17-release-rate.svg) -_Reliable tag data starts 2026-05-01; earlier months used chat-prefixed tagging and are not comparable._ +_Release counts use successful public CloudFront production promotions rather than stable-tag creation times._ ## 2. Graphs ### Ticket Resolved Count By Root Cause (Previous week) diff --git a/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg b/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg index 1f188b557a..e99ee7f11e 100644 --- a/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg +++ b/.ops/weekly-reviews/artifacts/2026-08-17-release-rate.svg @@ -1,20 +1,20 @@ - + Stable release rate for the four weeks ending August 17, 2026 - Weekly stable release counts were five, three, three, and one, averaging three releases per week. + Weekly stable release counts were four, four, two, and three, averaging 3.25 releases per week. Stable releases per oncall week - Trailing average: 3.0/week + Trailing average: 3.3/week - - - - + + + + - 5331 + 4423 07/2708/0308/1008/17 - Strict semver tags; weeks end at Monday 09:30 PT + Successful public production promotions; weeks end at Monday 09:30 PT From 05b2f039112f31a3986783b7702cf78e411545a0 Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:40:26 -0700 Subject: [PATCH 3/7] fix: reconcile weekly page log --- .ops/weekly-reviews/2026-08-17.md | 38 +++++++++++++++++-------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md index bfbff5c77a..e0ddf5f004 100644 --- a/.ops/weekly-reviews/2026-08-17.md +++ b/.ops/weekly-reviews/2026-08-17.md @@ -3,7 +3,7 @@ **On Call:** zoelin | **Next:** dingfeli | **Previous:** shhvin ## 1. Summary -* Pages: 8 across 8 tickets (degraded engagements fallback; per-ticket notification counts and outside-hours counts unavailable) +* Pages: 14 across 10 tickets (oncall page-message archive) * [Ticket Queue](https://t.corp.amazon.com/issues/?q=extensions.tt.status%3A%28Assigned%20OR%20Researching%20OR%20%22Work%20In%20Progress%22%20OR%20Pending%29%20AND%20extensions.tt.assignedGroup%3A%22Amazon%20Q%20for%20CLI%22): 153 → 153 * Incoming Tickets: 16 (21 raw; two QCLI-SuccessRateDown tickets collapse to one issue, four identical invalid-leg SmokeBot tickets collapse to one, and two identical inconsistent-verdict tickets collapse to one) * Resolved Tickets: 16 (1 Sev2, 15 lower) @@ -43,7 +43,7 @@ _Split by severity, Sev2 first. Severity is the ticket's **current** severity, s _Pick one (1–10) during the ops review._ -**Suggested: 7/10** — eight paged tickets included six customer-impacting Mantle overload episodes under one reused alarm ticket, while OAuth cases and release work added sustained follow-up load. This is a suggestion for the oncall to confirm during review. +**Suggested: 7/10** — fourteen page notifications across ten tickets included five notifications for the reused Mantle alarm ticket, two cross-team pages omitted by the degraded fallback, and sustained OAuth and release follow-up. This is a suggestion for the oncall to confirm during review. ## 4. Large Scale Significant Events * None @@ -59,23 +59,25 @@ Candidate carried-forward items: ## 6. Page Log -_One row per paged ticket. Because Builder Insights was unavailable, this uses the degraded engagements fallback: it identifies paged tickets but overwrites notification history, so every Pages value is 1 and repeated notifications cannot be recovered. A † timestamp is the latest retained alias engagement rather than a guaranteed first notification._ +_One row per paged ticket. Counts and timestamps come from the archived oncall page messages for the reporting window; repeated notifications are retained and times are shown in PT._ | # | Ticket | Pages | First Page (PT) | Page Sequence | Impact Summary | Root Cause | Mitigation | Action Items | Risk of Recurrence | Related Tickets | |---|---|---|---|---|---|---|---|---|---|---| -| 1 | [P488415895](https://t.corp.amazon.com/P488415895) | 1 | 08-10 09:30† | Retained engagement | Kiro CLI 2.13.0 generated about 590K guaranteed-fail ListAvailableProfiles calls/day; no customer or service impact was observed | Version 2.13.0 sent API-key credentials to an operation that accepts only OIDC/SSO bearer tokens, then repeated the call in an application loop | Ticket was downgraded while the API-key owner and client path are investigated; the auth layer correctly rejected the invalid token type | Fix the client credential selection and confirm whether the same behavior exists in KAS | Medium - legacy 2.13.0 clients continue generating high-volume failed traffic | [V2310285730](https://t.corp.amazon.com/V2310285730) | -| 2 | [V2321344948](https://t.corp.amazon.com/V2321344948) | 1 | 08-10 20:00 | New Sev2 | QCLI first-token latency exceeded 25 seconds for 14 of 15 one-minute datapoints | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | -| 3 | [V2322113921](https://t.corp.amazon.com/V2322113921) | 1 | 08-11 10:26 | New Sev2 | QCLI success rate fell below 99% for two of three five-minute datapoints | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | -| 4 | [V2322698178](https://t.corp.amazon.com/V2322698178) | 1 | 08-11 19:54 | Retained engagement† | Six overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m; observed affected-user counts reached 1,656 in one episode | Mantle us-east-1 cell3 shared-gateway saturation produced 529/500 failures across Claude models; autoscaling was blind to the Atlas-path pressure | Mantle rebalanced four Kiro accounts from cell3 to cell2 and service recovered | Track durable gateway/autoscaling and 500-only alarm fixes in P491128276 and P491374811 | High - six episodes occurred and the engagement fallback overwrote earlier notification history | [P491128276](https://t.corp.amazon.com/P491128276), [P491374811](https://t.corp.amazon.com/P491374811), [V2317183654](https://t.corp.amazon.com/V2317183654) | -| 5 | [V2314292550](https://t.corp.amazon.com/V2314292550) | 1 | 08-12 09:29† | Upgraded | CSIRO MCP tool results silently lost structuredContent, so agents could miss structured data and substitute generic answers | Both CLI paths intentionally omit structuredContent when a non-empty content block is present in the shared tool-result conversion helper | Behavior and a server-side workaround were explained; ticket was downgraded as a product capability gap | Decide whether to preserve structuredContent alongside content in model context | Medium - every server returning both fields is affected until behavior changes | [P485942261](https://t.corp.amazon.com/P485942261) | -| 6 | [V2313145792](https://t.corp.amazon.com/V2313145792) | 1 | 08-12 09:30† | Upgraded | FINRA could not complete remote MCP OAuth from native Windows PowerShell; Kiro IDE worked with the same configuration | CLI clipboard detection did not recognize the available PowerShell 7 clipboard command and failed to surface the OAuth URL | IDE/v3 and diagnostic-log URL retrieval were provided as workarounds; 2.18.1 surfaces the URL when clipboard copy fails | Await customer confirmation on 2.18.1 and separately investigate WSL unauthorized responses if they persist | Low after 2.18.1; WSL authorization may be a separate issue | [V2311766396](https://t.corp.amazon.com/V2311766396) | -| 7 | [V2311766396](https://t.corp.amazon.com/V2311766396) | 1 | 08-13 09:29† | Upgraded | Numerix could not authorize a remote MCP server from Kiro CLI on Windows although the same configuration worked in Kiro IDE | The Windows clipboard path failed and the CLI did not visibly expose the OAuth URL | A diagnostic-log workaround was provided; version 2.18.1 surfaces the URL when clipboard handling fails | Await customer confirmation after upgrade to 2.18.1 | Low after 2.18.1 | [V2313145792](https://t.corp.amazon.com/V2313145792) | -| 8 | [V2325742869](https://t.corp.amazon.com/V2325742869) | 1 | 08-14 09:30† | Upgraded | GBST could not authenticate Kiro CLI 2.18 to Atlassian MCP authv2; consent failed and the localhost callback never arrived | The v2 engine added OAuth scopes that make Atlassian authv2's consent metadata request fail with an empty tenant; older CLI and v3 no-scope flows work | Use the v3 engine or non-authv2 endpoint; the v2 fix in PR #4076 merged and awaits release | Ship the v2 OAuth-scope fix and confirm the customer can complete consent | Low once the merged fix is released | None | -| | **Total** | **8** | | | | | | | | | - -_Engagement data retained only one row per alias. [V2322698178](https://t.corp.amazon.com/V2322698178) alone contains nine alarm cuts mapping to six overload episodes, so the headline of 8 pages is a known lower bound, not a recoverable notification total._ - -_High-severity candidates reviewed but not identified as paged by the fallback: [D434404377](https://t.corp.amazon.com/D434404377), [D479638291](https://t.corp.amazon.com/D479638291), [D499857327](https://t.corp.amazon.com/D499857327), [P459152339](https://t.corp.amazon.com/P459152339), [P473397715](https://t.corp.amazon.com/P473397715), [P476098688](https://t.corp.amazon.com/P476098688), [P479056395](https://t.corp.amazon.com/P479056395), [V2243924150](https://t.corp.amazon.com/V2243924150), [V2248275776](https://t.corp.amazon.com/V2248275776), [V2274501749](https://t.corp.amazon.com/V2274501749), [V2277483794](https://t.corp.amazon.com/V2277483794), [V2279916688](https://t.corp.amazon.com/V2279916688), [V2288565081](https://t.corp.amazon.com/V2288565081), [V2299574204](https://t.corp.amazon.com/V2299574204), [V2304775994](https://t.corp.amazon.com/V2304775994), [V2304984688](https://t.corp.amazon.com/V2304984688), [V2305734483](https://t.corp.amazon.com/V2305734483), [V2306322595](https://t.corp.amazon.com/V2306322595), and [V2317754397](https://t.corp.amazon.com/V2317754397). Most were closed, resolved, or downgraded; V2304984688 remains assigned at Sev3._ +| 1 | [V2322698178](https://t.corp.amazon.com/V2322698178) | 5 | 08-11 19:55 | New Sev2; upgraded 08-12 03:55, 08-13 11:02, 08-13 12:53, 08-14 12:30 | Six overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m; observed affected-user counts reached 1,656 in one episode | Mantle us-east-1 cell3 shared-gateway saturation produced 529/500 failures across Claude models; autoscaling was blind to the Atlas-path pressure | Mantle rebalanced four Kiro accounts from cell3 to cell2 and service recovered | Track durable gateway/autoscaling and 500-only alarm fixes in P491128276 and P491374811 | High - five page notifications occurred across four days and six overload episodes | [P491128276](https://t.corp.amazon.com/P491128276), [P491374811](https://t.corp.amazon.com/P491374811), [V2317183654](https://t.corp.amazon.com/V2317183654) | +| 2 | [D490637679](https://t.corp.amazon.com/D490637679) | 1 | 08-10 09:30 | Upgraded Sev2 | A customer reported continued Q Developer charges after attempting to cancel and requested manual confirmation and refund review | One Builder ID individual Q Developer Pro subscription was cancelled on July 8; the August invoice was the final full-month July charge, and no active claims or profiles remained | Kiro CP confirmed zero active or pending resources and documented the cancellation and billing timeline | Support/Billing should handle any refund review and confirm whether the customer saw a later charge | Low - subscription and metering are confirmed ended | None | +| 3 | [P488415895](https://t.corp.amazon.com/P488415895) | 1 | 08-10 09:30 | Upgraded Sev2 | Kiro CLI 2.13.0 generated about 590K guaranteed-fail ListAvailableProfiles calls/day; no customer or service impact was observed | Version 2.13.0 sent API-key credentials to an operation that accepts only OIDC/SSO bearer tokens, then repeated the call in an application loop | Ticket was downgraded while the API-key owner and client path are investigated; the auth layer correctly rejected the invalid token type | Fix the client credential selection and confirm whether the same behavior exists in KAS | Medium - legacy 2.13.0 clients continue generating high-volume failed traffic | [V2310285730](https://t.corp.amazon.com/V2310285730) | +| 4 | [V2321344948](https://t.corp.amazon.com/V2321344948) | 1 | 08-11 09:30 | Upgraded Sev2 | QCLI first-token latency exceeded 25 seconds for 14 of 15 one-minute datapoints | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | +| 5 | [V2322113921](https://t.corp.amazon.com/V2322113921) | 1 | 08-11 10:27 | New Sev2 | QCLI success rate fell below 99% for two of three five-minute datapoints | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | +| 6 | [P470434547](https://t.corp.amazon.com/P470434547) | 1 | 08-11 11:13 | Requested Sev2 | Partner Central's launch was blocked because its SaaS MCP server remained auth-required in Kiro CLI after browser sign-in | OAuth discovery, DCR, sign-in, and token exchange succeeded; the authenticated POST /mcp was then blocked with a CloudFront/WAF 403 at the Partner Central service edge | Trace evidence isolated the service-edge block, and the requester later confirmed the flow succeeds with Kiro CLI v3 | Partner Central should verify its WAF/CloudFront rules; resolve after requester confirmation | Medium - the affected service-edge policy can still block v2 clients | None | +| 7 | [V2314292550](https://t.corp.amazon.com/V2314292550) | 1 | 08-12 09:30 | Upgraded Sev2 | CSIRO MCP tool results silently lost structuredContent, so agents could miss structured data and substitute generic answers | Both CLI paths intentionally omit structuredContent when a non-empty content block is present in the shared tool-result conversion helper | Behavior and a server-side workaround were explained; ticket was downgraded as a product capability gap | Decide whether to preserve structuredContent alongside content in model context | Medium - every server returning both fields is affected until behavior changes | [P485942261](https://t.corp.amazon.com/P485942261) | +| 8 | [V2313145792](https://t.corp.amazon.com/V2313145792) | 1 | 08-12 09:33 | Upgraded Sev2 | FINRA could not complete remote MCP OAuth from native Windows PowerShell; Kiro IDE worked with the same configuration | CLI clipboard detection did not recognize the available PowerShell 7 clipboard command and failed to surface the OAuth URL | IDE/v3 and diagnostic-log URL retrieval were provided as workarounds; 2.18.1 surfaces the URL when clipboard copy fails | Await customer confirmation on 2.18.1 and separately investigate WSL unauthorized responses if they persist | Low after 2.18.1; WSL authorization may be a separate issue | [V2311766396](https://t.corp.amazon.com/V2311766396) | +| 9 | [V2311766396](https://t.corp.amazon.com/V2311766396) | 1 | 08-13 09:30 | Upgraded Sev2 | Numerix could not authorize a remote MCP server from Kiro CLI on Windows although the same configuration worked in Kiro IDE | The Windows clipboard path failed and the CLI did not visibly expose the OAuth URL | A diagnostic-log workaround was provided; version 2.18.1 surfaces the URL when clipboard handling fails | Await customer confirmation after upgrade to 2.18.1 | Low after 2.18.1 | [V2313145792](https://t.corp.amazon.com/V2313145792) | +| 10 | [V2325742869](https://t.corp.amazon.com/V2325742869) | 1 | 08-14 09:30 | Upgraded Sev2 | GBST could not authenticate Kiro CLI 2.18 to Atlassian MCP authv2; consent failed and the localhost callback never arrived | The v2 engine added OAuth scopes that make Atlassian authv2's consent metadata request fail with an empty tenant; older CLI and v3 no-scope flows work | Use the v3 engine or non-authv2 endpoint; the v2 fix in PR #4076 merged and awaits release | Ship the v2 OAuth-scope fix and confirm the customer can complete consent | Low once the merged fix is released | None | +| | **Total** | **14** | | | | | | | | | + +_The page-message archive retains repeated notifications: [V2322698178](https://t.corp.amazon.com/V2322698178) generated five pages. [P470434547](https://t.corp.amazon.com/P470434547) and [D490637679](https://t.corp.amazon.com/D490637679) were outside the Amazon Q for CLI resolver-group search and are included because they paged the oncall._ + +_High-severity candidates reviewed but not present in the page-message archive: [D434404377](https://t.corp.amazon.com/D434404377), [D479638291](https://t.corp.amazon.com/D479638291), [D499857327](https://t.corp.amazon.com/D499857327), [P459152339](https://t.corp.amazon.com/P459152339), [P473397715](https://t.corp.amazon.com/P473397715), [P476098688](https://t.corp.amazon.com/P476098688), [P479056395](https://t.corp.amazon.com/P479056395), [V2243924150](https://t.corp.amazon.com/V2243924150), [V2248275776](https://t.corp.amazon.com/V2248275776), [V2274501749](https://t.corp.amazon.com/V2274501749), [V2277483794](https://t.corp.amazon.com/V2277483794), [V2279916688](https://t.corp.amazon.com/V2279916688), [V2288565081](https://t.corp.amazon.com/V2288565081), [V2299574204](https://t.corp.amazon.com/V2299574204), [V2304775994](https://t.corp.amazon.com/V2304775994), [V2304984688](https://t.corp.amazon.com/V2304984688), [V2305734483](https://t.corp.amazon.com/V2305734483), [V2306322595](https://t.corp.amazon.com/V2306322595), and [V2317754397](https://t.corp.amazon.com/V2317754397). Most were closed, resolved, or downgraded; V2304984688 remains assigned at Sev3._ ## 7. Open Sev2s @@ -128,8 +130,10 @@ _To be reviewed during the meeting._ | # | Team - Ticket | Description | |---|---|---| | 1 | Mantle - [P491128276](https://t.corp.amazon.com/P491128276) | Cross-team investigation of recurring us-east-1 cell3 shared-gateway saturation across Mantle-hosted Claude models. Mantle rebalanced four Kiro accounts to cell2; follow-up alarm and autoscaling work remains tracked separately. | +| 2 | Kiro CP - [D490637679](https://t.corp.amazon.com/D490637679) | Subscription and billing investigation confirmed the Builder ID Pro subscription was cancelled July 8, metering ended July 31, and no active claims or profiles remained; Support/Billing owns any refund review. | +| 3 | AWSID Signin / Partner Central - [P470434547](https://t.corp.amazon.com/P470434547) | Cross-team OAuth investigation showed authentication completed before Partner Central's CloudFront/WAF blocked the authenticated MCP request; the requester later confirmed Kiro CLI v3 succeeds. | -_The partner-tag search returned no additional reassigned ticket, and no cross-team CR was found in the reviewed worklogs._ +_The partner-tag search returned no additional reassigned ticket; the page-message archive surfaced the two cross-team pages above that were outside the Amazon Q for CLI resolver-group search._ ## 11. Dashboard Review From caa63b0419982f90b070f2a9556cd111c2935182 Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:42:51 -0700 Subject: [PATCH 4/7] fix: count only tickets cut by oncall --- .ops/weekly-reviews/2026-08-17.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md index e0ddf5f004..9108cb036b 100644 --- a/.ops/weekly-reviews/2026-08-17.md +++ b/.ops/weekly-reviews/2026-08-17.md @@ -127,13 +127,7 @@ _To be reviewed during the meeting._ ## 10. Tickets Cut to Other Teams -| # | Team - Ticket | Description | -|---|---|---| -| 1 | Mantle - [P491128276](https://t.corp.amazon.com/P491128276) | Cross-team investigation of recurring us-east-1 cell3 shared-gateway saturation across Mantle-hosted Claude models. Mantle rebalanced four Kiro accounts to cell2; follow-up alarm and autoscaling work remains tracked separately. | -| 2 | Kiro CP - [D490637679](https://t.corp.amazon.com/D490637679) | Subscription and billing investigation confirmed the Builder ID Pro subscription was cancelled July 8, metering ended July 31, and no active claims or profiles remained; Support/Billing owns any refund review. | -| 3 | AWSID Signin / Partner Central - [P470434547](https://t.corp.amazon.com/P470434547) | Cross-team OAuth investigation showed authentication completed before Partner Central's CloudFront/WAF blocked the authenticated MCP request; the requester later confirmed Kiro CLI v3 succeeds. | - -_The partner-tag search returned no additional reassigned ticket; the page-message archive surfaced the two cross-team pages above that were outside the Amazon Q for CLI resolver-group search._ +* None ## 11. Dashboard Review From a8b29deb6673cce6727057d069d347d0536d7f9d Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:44:30 -0700 Subject: [PATCH 5/7] fix: carry forward prior action items only --- .ops/weekly-reviews/2026-08-17.md | 1 - 1 file changed, 1 deletion(-) diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md index 9108cb036b..fb0fdb3120 100644 --- a/.ops/weekly-reviews/2026-08-17.md +++ b/.ops/weekly-reviews/2026-08-17.md @@ -55,7 +55,6 @@ During the meeting, go over open action items here https://tiny.amazon.com/1auvb Candidate carried-forward items: * **Closed** — [P473397715](https://t.corp.amazon.com/P473397715): security review resolved as no issue after validation concluded Kiro did not access the credential store. * **Open** — [P488415895](https://t.corp.amazon.com/P488415895): fix ListAvailableProfiles credential selection and confirm whether KAS shares the 2.13.0 application-loop behavior. -* **Open** — [P491128276](https://t.corp.amazon.com/P491128276): track durable Mantle gateway/autoscaling and availability-alarm improvements after the repeated cell3 overload episodes. ## 6. Page Log From e347f26463a2c5731790311643169857f44899d2 Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:46:39 -0700 Subject: [PATCH 6/7] fix: clarify page log and backend instability --- .ops/weekly-reviews/2026-08-17.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md index fb0fdb3120..a0b441e998 100644 --- a/.ops/weekly-reviews/2026-08-17.md +++ b/.ops/weekly-reviews/2026-08-17.md @@ -46,7 +46,8 @@ _Pick one (1–10) during the ops review._ **Suggested: 7/10** — fourteen page notifications across ten tickets included five notifications for the reused Mantle alarm ticket, two cross-team pages omitted by the degraded fallback, and sustained OAuth and release follow-up. This is a suggestion for the oncall to confirm during review. ## 4. Large Scale Significant Events -* None +* None formally declared. +* **Operational note — recurring backend instability (not a declared LSE):** Success-rate drops and `ModelOverloadedError` spikes recurred throughout the week. The reused QCLI-SuccessRateDown ticket generated five page notifications across four days; six measured overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m, with up to 1,656 affected users observed in one episode. Investigation linked the failures to Mantle us-east-1 cell3 shared-gateway saturation across Claude models. ## 5. Previous Week's Action Items @@ -60,19 +61,19 @@ Candidate carried-forward items: _One row per paged ticket. Counts and timestamps come from the archived oncall page messages for the reporting window; repeated notifications are retained and times are shown in PT._ -| # | Ticket | Pages | First Page (PT) | Page Sequence | Impact Summary | Root Cause | Mitigation | Action Items | Risk of Recurrence | Related Tickets | -|---|---|---|---|---|---|---|---|---|---|---| -| 1 | [V2322698178](https://t.corp.amazon.com/V2322698178) | 5 | 08-11 19:55 | New Sev2; upgraded 08-12 03:55, 08-13 11:02, 08-13 12:53, 08-14 12:30 | Six overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m; observed affected-user counts reached 1,656 in one episode | Mantle us-east-1 cell3 shared-gateway saturation produced 529/500 failures across Claude models; autoscaling was blind to the Atlas-path pressure | Mantle rebalanced four Kiro accounts from cell3 to cell2 and service recovered | Track durable gateway/autoscaling and 500-only alarm fixes in P491128276 and P491374811 | High - five page notifications occurred across four days and six overload episodes | [P491128276](https://t.corp.amazon.com/P491128276), [P491374811](https://t.corp.amazon.com/P491374811), [V2317183654](https://t.corp.amazon.com/V2317183654) | -| 2 | [D490637679](https://t.corp.amazon.com/D490637679) | 1 | 08-10 09:30 | Upgraded Sev2 | A customer reported continued Q Developer charges after attempting to cancel and requested manual confirmation and refund review | One Builder ID individual Q Developer Pro subscription was cancelled on July 8; the August invoice was the final full-month July charge, and no active claims or profiles remained | Kiro CP confirmed zero active or pending resources and documented the cancellation and billing timeline | Support/Billing should handle any refund review and confirm whether the customer saw a later charge | Low - subscription and metering are confirmed ended | None | -| 3 | [P488415895](https://t.corp.amazon.com/P488415895) | 1 | 08-10 09:30 | Upgraded Sev2 | Kiro CLI 2.13.0 generated about 590K guaranteed-fail ListAvailableProfiles calls/day; no customer or service impact was observed | Version 2.13.0 sent API-key credentials to an operation that accepts only OIDC/SSO bearer tokens, then repeated the call in an application loop | Ticket was downgraded while the API-key owner and client path are investigated; the auth layer correctly rejected the invalid token type | Fix the client credential selection and confirm whether the same behavior exists in KAS | Medium - legacy 2.13.0 clients continue generating high-volume failed traffic | [V2310285730](https://t.corp.amazon.com/V2310285730) | -| 4 | [V2321344948](https://t.corp.amazon.com/V2321344948) | 1 | 08-11 09:30 | Upgraded Sev2 | QCLI first-token latency exceeded 25 seconds for 14 of 15 one-minute datapoints | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | -| 5 | [V2322113921](https://t.corp.amazon.com/V2322113921) | 1 | 08-11 10:27 | New Sev2 | QCLI success rate fell below 99% for two of three five-minute datapoints | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | -| 6 | [P470434547](https://t.corp.amazon.com/P470434547) | 1 | 08-11 11:13 | Requested Sev2 | Partner Central's launch was blocked because its SaaS MCP server remained auth-required in Kiro CLI after browser sign-in | OAuth discovery, DCR, sign-in, and token exchange succeeded; the authenticated POST /mcp was then blocked with a CloudFront/WAF 403 at the Partner Central service edge | Trace evidence isolated the service-edge block, and the requester later confirmed the flow succeeds with Kiro CLI v3 | Partner Central should verify its WAF/CloudFront rules; resolve after requester confirmation | Medium - the affected service-edge policy can still block v2 clients | None | -| 7 | [V2314292550](https://t.corp.amazon.com/V2314292550) | 1 | 08-12 09:30 | Upgraded Sev2 | CSIRO MCP tool results silently lost structuredContent, so agents could miss structured data and substitute generic answers | Both CLI paths intentionally omit structuredContent when a non-empty content block is present in the shared tool-result conversion helper | Behavior and a server-side workaround were explained; ticket was downgraded as a product capability gap | Decide whether to preserve structuredContent alongside content in model context | Medium - every server returning both fields is affected until behavior changes | [P485942261](https://t.corp.amazon.com/P485942261) | -| 8 | [V2313145792](https://t.corp.amazon.com/V2313145792) | 1 | 08-12 09:33 | Upgraded Sev2 | FINRA could not complete remote MCP OAuth from native Windows PowerShell; Kiro IDE worked with the same configuration | CLI clipboard detection did not recognize the available PowerShell 7 clipboard command and failed to surface the OAuth URL | IDE/v3 and diagnostic-log URL retrieval were provided as workarounds; 2.18.1 surfaces the URL when clipboard copy fails | Await customer confirmation on 2.18.1 and separately investigate WSL unauthorized responses if they persist | Low after 2.18.1; WSL authorization may be a separate issue | [V2311766396](https://t.corp.amazon.com/V2311766396) | -| 9 | [V2311766396](https://t.corp.amazon.com/V2311766396) | 1 | 08-13 09:30 | Upgraded Sev2 | Numerix could not authorize a remote MCP server from Kiro CLI on Windows although the same configuration worked in Kiro IDE | The Windows clipboard path failed and the CLI did not visibly expose the OAuth URL | A diagnostic-log workaround was provided; version 2.18.1 surfaces the URL when clipboard handling fails | Await customer confirmation after upgrade to 2.18.1 | Low after 2.18.1 | [V2313145792](https://t.corp.amazon.com/V2313145792) | -| 10 | [V2325742869](https://t.corp.amazon.com/V2325742869) | 1 | 08-14 09:30 | Upgraded Sev2 | GBST could not authenticate Kiro CLI 2.18 to Atlassian MCP authv2; consent failed and the localhost callback never arrived | The v2 engine added OAuth scopes that make Atlassian authv2's consent metadata request fail with an empty tenant; older CLI and v3 no-scope flows work | Use the v3 engine or non-authv2 endpoint; the v2 fix in PR #4076 merged and awaits release | Ship the v2 OAuth-scope fix and confirm the customer can complete consent | Low once the merged fix is released | None | -| | **Total** | **14** | | | | | | | | | +| # | Ticket | Title | Pages | First Page (PT) | Page Sequence | Impact Summary | Root Cause | Mitigation | Action Items | Risk of Recurrence | Related Tickets | +|---|---|---|---|---|---|---|---|---|---|---|---| +| 1 | [V2322698178](https://t.corp.amazon.com/V2322698178) | [ALARM] [us-east-1] QCLI-SuccessRateDown | 5 | 08-11 19:55 | New Sev2; upgraded 08-12 03:55, 08-13 11:02, 08-13 12:53, 08-14 12:30 | Six overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m; observed affected-user counts reached 1,656 in one episode | Mantle us-east-1 cell3 shared-gateway saturation produced 529/500 failures across Claude models; autoscaling was blind to the Atlas-path pressure | Mantle rebalanced four Kiro accounts from cell3 to cell2 and service recovered | Track durable gateway/autoscaling and 500-only alarm fixes in P491128276 and P491374811 | High - five page notifications occurred across four days and six overload episodes | [P491128276](https://t.corp.amazon.com/P491128276), [P491374811](https://t.corp.amazon.com/P491374811), [V2317183654](https://t.corp.amazon.com/V2317183654) | +| 2 | [D490637679](https://t.corp.amazon.com/D490637679) | confirmation on active resources for Q and deletion steps: Account ID: 460928920438 | 1 | 08-10 09:30 | Upgraded Sev2 | A customer reported continued Q Developer charges after attempting to cancel and requested manual confirmation and refund review | One Builder ID individual Q Developer Pro subscription was cancelled on July 8; the August invoice was the final full-month July charge, and no active claims or profiles remained | Kiro CP confirmed zero active or pending resources and documented the cancellation and billing timeline | Support/Billing should handle any refund review and confirm whether the customer saw a later charge | Low - subscription and metering are confirmed ended | None | +| 3 | [P488415895](https://t.corp.amazon.com/P488415895) | KiroCLI 2.13.0 sends API-key credential on ListAvailableProfiles → 100% InvalidTokenException (~590K failed calls/day) | 1 | 08-10 09:30 | Upgraded Sev2 | Kiro CLI 2.13.0 generated about 590K guaranteed-fail ListAvailableProfiles calls/day; no customer or service impact was observed | Version 2.13.0 sent API-key credentials to an operation that accepts only OIDC/SSO bearer tokens, then repeated the call in an application loop | Ticket was downgraded while the API-key owner and client path are investigated; the auth layer correctly rejected the invalid token type | Fix the client credential selection and confirm whether the same behavior exists in KAS | Medium - legacy 2.13.0 clients continue generating high-volume failed traffic | [V2310285730](https://t.corp.amazon.com/V2310285730) | +| 4 | [V2321344948](https://t.corp.amazon.com/V2321344948) | [ALARM] [us-east-1] QCLI-FirstTokenLatency | 1 | 08-11 09:30 | Upgraded Sev2 | QCLI first-token latency exceeded 25 seconds for 14 of 15 one-minute datapoints | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | +| 5 | [V2322113921](https://t.corp.amazon.com/V2322113921) | [ALARM] [us-east-1] QCLI-SuccessRateDown | 1 | 08-11 10:27 | New Sev2 | QCLI success rate fell below 99% for two of three five-minute datapoints | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | +| 6 | [P470434547](https://t.corp.amazon.com/P470434547) | Kiro CLI and Amazon Quick fail OAuth discovery (RFC 9728) with IAM Sign-In for SaaS MCP server | 1 | 08-11 11:13 | Requested Sev2 | Partner Central's launch was blocked because its SaaS MCP server remained auth-required in Kiro CLI after browser sign-in | OAuth discovery, DCR, sign-in, and token exchange succeeded; the authenticated POST /mcp was then blocked with a CloudFront/WAF 403 at the Partner Central service edge | Trace evidence isolated the service-edge block, and the requester later confirmed the flow succeeds with Kiro CLI v3 | Partner Central should verify its WAF/CloudFront rules; resolve after requester confirmation | Medium - the affected service-edge policy can still block v2 clients | None | +| 7 | [V2314292550](https://t.corp.amazon.com/V2314292550) | Kiro - 178591474300389 - {customer.supportlevel} - 4 - 095077079535 CSIRO Australia - Kiro CLI does not support specific MCP response components | 1 | 08-12 09:30 | Upgraded Sev2 | CSIRO MCP tool results silently lost structuredContent, so agents could miss structured data and substitute generic answers | Both CLI paths intentionally omit structuredContent when a non-empty content block is present in the shared tool-result conversion helper | Behavior and a server-side workaround were explained; ticket was downgraded as a product capability gap | Decide whether to preserve structuredContent alongside content in model context | Medium - every server returning both fields is affected until behavior changes | [P485942261](https://t.corp.amazon.com/P485942261) | +| 8 | [V2313145792](https://t.corp.amazon.com/V2313145792) | Kiro - 178552476000308 - {customer.supportlevel} - 3 - 570164370074 FINRA - MCP Oauth flow is broken in latest Kiro CLI version 2.16 on windows 11 Powershell. | 1 | 08-12 09:33 | Upgraded Sev2 | FINRA could not complete remote MCP OAuth from native Windows PowerShell; Kiro IDE worked with the same configuration | CLI clipboard detection did not recognize the available PowerShell 7 clipboard command and failed to surface the OAuth URL | IDE/v3 and diagnostic-log URL retrieval were provided as workarounds; 2.18.1 surfaces the URL when clipboard copy fails | Await customer confirmation on 2.18.1 and separately investigate WSL unauthorized responses if they persist | Low after 2.18.1; WSL authorization may be a separate issue | [V2311766396](https://t.corp.amazon.com/V2311766396) | +| 9 | [V2311766396](https://t.corp.amazon.com/V2311766396) | Kiro - 178549183500365 - {customer.supportlevel} - 2 - 108336546307 Numerix LLC - OAuth flow broken for remote MCP via Kiro CLI | 1 | 08-13 09:30 | Upgraded Sev2 | Numerix could not authorize a remote MCP server from Kiro CLI on Windows although the same configuration worked in Kiro IDE | The Windows clipboard path failed and the CLI did not visibly expose the OAuth URL | A diagnostic-log workaround was provided; version 2.18.1 surfaces the URL when clipboard handling fails | Await customer confirmation after upgrade to 2.18.1 | Low after 2.18.1 | [V2313145792](https://t.corp.amazon.com/V2313145792) | +| 10 | [V2325742869](https://t.corp.amazon.com/V2325742869) | Kiro - 178668148000875 - {customer.supportlevel} - 1 - 575510108455 GBST - Chat: Kiro CLI v2.18 Unable to Authenticate with Atlassian MCP authv2 – OAuth Callback Missing | 1 | 08-14 09:30 | Upgraded Sev2 | GBST could not authenticate Kiro CLI 2.18 to Atlassian MCP authv2; consent failed and the localhost callback never arrived | The v2 engine added OAuth scopes that make Atlassian authv2's consent metadata request fail with an empty tenant; older CLI and v3 no-scope flows work | Use the v3 engine or non-authv2 endpoint; the v2 fix in PR #4076 merged and awaits release | Ship the v2 OAuth-scope fix and confirm the customer can complete consent | Low once the merged fix is released | None | +| | **Total** | | **14** | | | | | | | | | _The page-message archive retains repeated notifications: [V2322698178](https://t.corp.amazon.com/V2322698178) generated five pages. [P470434547](https://t.corp.amazon.com/P470434547) and [D490637679](https://t.corp.amazon.com/D490637679) were outside the Amazon Q for CLI resolver-group search and are included because they paged the oncall._ From 24be626a23f483fd348d00f62d2b137f1b79feee Mon Sep 17 00:00:00 2001 From: Zoe Lin Date: Mon, 17 Aug 2026 09:50:50 -0700 Subject: [PATCH 7/7] fix: add quantified customer impact --- .ops/weekly-reviews/2026-08-17.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.ops/weekly-reviews/2026-08-17.md b/.ops/weekly-reviews/2026-08-17.md index a0b441e998..b218511469 100644 --- a/.ops/weekly-reviews/2026-08-17.md +++ b/.ops/weekly-reviews/2026-08-17.md @@ -66,8 +66,8 @@ _One row per paged ticket. Counts and timestamps come from the archived oncall p | 1 | [V2322698178](https://t.corp.amazon.com/V2322698178) | [ALARM] [us-east-1] QCLI-SuccessRateDown | 5 | 08-11 19:55 | New Sev2; upgraded 08-12 03:55, 08-13 11:02, 08-13 12:53, 08-14 12:30 | Six overload episodes caused 37,264 model-error turns across 878,083 user turns (4.24%) over 2h36m; observed affected-user counts reached 1,656 in one episode | Mantle us-east-1 cell3 shared-gateway saturation produced 529/500 failures across Claude models; autoscaling was blind to the Atlas-path pressure | Mantle rebalanced four Kiro accounts from cell3 to cell2 and service recovered | Track durable gateway/autoscaling and 500-only alarm fixes in P491128276 and P491374811 | High - five page notifications occurred across four days and six overload episodes | [P491128276](https://t.corp.amazon.com/P491128276), [P491374811](https://t.corp.amazon.com/P491374811), [V2317183654](https://t.corp.amazon.com/V2317183654) | | 2 | [D490637679](https://t.corp.amazon.com/D490637679) | confirmation on active resources for Q and deletion steps: Account ID: 460928920438 | 1 | 08-10 09:30 | Upgraded Sev2 | A customer reported continued Q Developer charges after attempting to cancel and requested manual confirmation and refund review | One Builder ID individual Q Developer Pro subscription was cancelled on July 8; the August invoice was the final full-month July charge, and no active claims or profiles remained | Kiro CP confirmed zero active or pending resources and documented the cancellation and billing timeline | Support/Billing should handle any refund review and confirm whether the customer saw a later charge | Low - subscription and metering are confirmed ended | None | | 3 | [P488415895](https://t.corp.amazon.com/P488415895) | KiroCLI 2.13.0 sends API-key credential on ListAvailableProfiles → 100% InvalidTokenException (~590K failed calls/day) | 1 | 08-10 09:30 | Upgraded Sev2 | Kiro CLI 2.13.0 generated about 590K guaranteed-fail ListAvailableProfiles calls/day; no customer or service impact was observed | Version 2.13.0 sent API-key credentials to an operation that accepts only OIDC/SSO bearer tokens, then repeated the call in an application loop | Ticket was downgraded while the API-key owner and client path are investigated; the auth layer correctly rejected the invalid token type | Fix the client credential selection and confirm whether the same behavior exists in KAS | Medium - legacy 2.13.0 clients continue generating high-volume failed traffic | [V2310285730](https://t.corp.amazon.com/V2310285730) | -| 4 | [V2321344948](https://t.corp.amazon.com/V2321344948) | [ALARM] [us-east-1] QCLI-FirstTokenLatency | 1 | 08-11 09:30 | Upgraded Sev2 | QCLI first-token latency exceeded 25 seconds for 14 of 15 one-minute datapoints | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | -| 5 | [V2322113921](https://t.corp.amazon.com/V2322113921) | [ALARM] [us-east-1] QCLI-SuccessRateDown | 1 | 08-11 10:27 | New Sev2 | QCLI success rate fell below 99% for two of three five-minute datapoints | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | +| 4 | [V2321344948](https://t.corp.amazon.com/V2321344948) | [ALARM] [us-east-1] QCLI-FirstTokenLatency | 1 | 08-11 09:30 | Upgraded Sev2 | During the exact 02:45Z–03:00Z alarm window, 5,564 of 48,678 requests (11.4%) exceeded 25 seconds across approximately 3,428 of 10,282 `clientId` installation proxies; overall p90 TTFT was 39.9 seconds, with 97% of slow requests attributed to `auto` | Downstream model-serving latency degradation; no CLI-side regression was identified | The downstream condition recovered and the alarm ticket was resolved | None recorded | Medium - model-serving latency events can recur | None | +| 5 | [V2322113921](https://t.corp.amazon.com/V2322113921) | [ALARM] [us-east-1] QCLI-SuccessRateDown | 1 | 08-11 10:27 | New Sev2 | During the exact 17:11Z–17:26Z alarm window, 2,789 customer-visible turns failed across approximately 1,667 `clientId` installation proxies out of 431,302 requests and approximately 31,201 clients; `ModelOverloadedError` caused 1,785 failures (64%) across approximately 1,002 clients | Downstream model-serving availability degradation | The service recovered and the alarm ticket was resolved | Continue backend-capacity follow-up under P491128276 | High - the same alarm produced multiple later episodes this week | [P491128276](https://t.corp.amazon.com/P491128276), [V2322698178](https://t.corp.amazon.com/V2322698178) | | 6 | [P470434547](https://t.corp.amazon.com/P470434547) | Kiro CLI and Amazon Quick fail OAuth discovery (RFC 9728) with IAM Sign-In for SaaS MCP server | 1 | 08-11 11:13 | Requested Sev2 | Partner Central's launch was blocked because its SaaS MCP server remained auth-required in Kiro CLI after browser sign-in | OAuth discovery, DCR, sign-in, and token exchange succeeded; the authenticated POST /mcp was then blocked with a CloudFront/WAF 403 at the Partner Central service edge | Trace evidence isolated the service-edge block, and the requester later confirmed the flow succeeds with Kiro CLI v3 | Partner Central should verify its WAF/CloudFront rules; resolve after requester confirmation | Medium - the affected service-edge policy can still block v2 clients | None | | 7 | [V2314292550](https://t.corp.amazon.com/V2314292550) | Kiro - 178591474300389 - {customer.supportlevel} - 4 - 095077079535 CSIRO Australia - Kiro CLI does not support specific MCP response components | 1 | 08-12 09:30 | Upgraded Sev2 | CSIRO MCP tool results silently lost structuredContent, so agents could miss structured data and substitute generic answers | Both CLI paths intentionally omit structuredContent when a non-empty content block is present in the shared tool-result conversion helper | Behavior and a server-side workaround were explained; ticket was downgraded as a product capability gap | Decide whether to preserve structuredContent alongside content in model context | Medium - every server returning both fields is affected until behavior changes | [P485942261](https://t.corp.amazon.com/P485942261) | | 8 | [V2313145792](https://t.corp.amazon.com/V2313145792) | Kiro - 178552476000308 - {customer.supportlevel} - 3 - 570164370074 FINRA - MCP Oauth flow is broken in latest Kiro CLI version 2.16 on windows 11 Powershell. | 1 | 08-12 09:33 | Upgraded Sev2 | FINRA could not complete remote MCP OAuth from native Windows PowerShell; Kiro IDE worked with the same configuration | CLI clipboard detection did not recognize the available PowerShell 7 clipboard command and failed to surface the OAuth URL | IDE/v3 and diagnostic-log URL retrieval were provided as workarounds; 2.18.1 surfaces the URL when clipboard copy fails | Await customer confirmation on 2.18.1 and separately investigate WSL unauthorized responses if they persist | Low after 2.18.1; WSL authorization may be a separate issue | [V2311766396](https://t.corp.amazon.com/V2311766396) |