diff --git a/.github/CLAUDE.md b/.github/CLAUDE.md index 2b5d19bc2..97cf95926 100644 --- a/.github/CLAUDE.md +++ b/.github/CLAUDE.md @@ -1,819 +1,185 @@ -# CLAUDE.md - CI/CD & GitHub Actions Complete Guide - -This file provides comprehensive guidance for Claude Code and human engineers working with the CI/CD infrastructure and GitHub Actions workflows in this repository. - -## Overview - -The Aignostics Python SDK uses a **sophisticated multi-stage CI/CD pipeline** built on GitHub Actions with: - -* **Multiple workflow files**, including both entry-point and reusable workflows -* **Reusable workflow architecture** for modularity and maintainability -* **Environment-based testing** (staging/production with scheduled validation) -* **Multi-category test execution** (unit, integration, e2e, long_running, very_long_running, scheduled) -* **Automated PR reviews** with Claude Code -* **Comprehensive quality gates** (lint, audit, test, CodeQL) -* **Native executable builds** for 6 platforms -* **Automated releases** with package publishing -* **External monitoring** via BetterStack heartbeats - -## Workflow Architecture +# CLAUDE.md — CI/CD & GitHub Actions + +Guidance for working with the GitHub Actions workflows in `.github/workflows/`. +The YAML files are the source of truth; this doc orients you and calls out +non-obvious behaviour. When in doubt, read the workflow. + +See root `CLAUDE.md` and `Makefile` for development commands and test markers. + +## Workflow layout + +Entry-point workflows (`+ …` names) are triggered by events; reusable +workflows (`> …`, prefixed `_`) are called via `uses:`. + +| Entry point | Trigger | Calls | +|-------------|---------|-------| +| `ci-cd.yml` | push (main, release/v*, tag v*.*.*), PR (main, release/v*), release created, workflow_dispatch | `_lint`, `_docs`, `_audit`, `_test`, `_codeql`, `_ketryx_report_and_check`, `_package-publish`, `_docker-publish` | +| `prepare-release.yml` | workflow_dispatch | — | +| `publish-release.yml` | workflow_dispatch | — | +| `merge-release.yml` | workflow_dispatch | — | +| `build-native-only.yml` | push/PR/release when msg/label has `build:native:only` | `_build-native-only` | +| `claude-code-interactive.yml` | `@claude` mentions + workflow_dispatch | `_claude-code` (interactive) | +| `claude-code-automation-pr-review.yml` | PR with `claude` label or `ready_for_review` | `_claude-code` (automation) | +| `claude-code-automation-operational-excellence-weekly.yml` | schedule + workflow_dispatch | `_claude-code` (automation) | +| `scheduled-testing-staging-hourly.yml` | cron `0 * * * *` | `_scheduled-test-hourly` (staging) | +| `scheduled-testing-staging-daily.yml` | cron `0 12 * * *` | `_scheduled-test-daily` (staging) | +| `scheduled-testing-production-hourly.yml` | cron `0 * * * *` | `_scheduled-test-hourly` (production) | +| `scheduled-testing-production-daily.yml` | cron `0 12 * * *` | `_scheduled-test-daily` (production) | +| `scheduled-audit-hourly.yml` | cron `0 * * * *` | `_scheduled-audit` | +| `codeql-scheduled.yml` | cron `22 3 * * 2` (Tue 03:22) | `_codeql` | +| `labels-sync.yml` | push to label config | — | + +Reusable: `_lint`, `_docs`, `_audit`, `_test`, `_codeql`, +`_ketryx_report_and_check`, `_package-publish`, `_docker-publish`, +`_build-native-only`, `_claude-code`, `_scheduled-audit`, +`_scheduled-test-hourly`, `_scheduled-test-daily`, `_scheduled-test-stress`. +(`stress-testing-staging.yml.paused` is currently disabled.) + +## Main pipeline (`ci-cd.yml`) + +**Triggers**: push to `main` / `release/v*` / tags `v*.*.*`; PR to `main` / +`release/v*`; `release` created; `workflow_dispatch` with a +`platform_environment` input (`staging` | `production`, default `staging`). + +**Skip conditions** (checked per job): commit message or PR label contains +`skip:ci` or `build:native:only`. + +**Jobs**: `get-commit-message` (extracts commit message + release version from +branch), `lint`, `docs`, `audit`, `test`, `codeql`, `sonarcloud`, +`ketryx_report_and_check`, `package_publish`, `docker_publish`. + +**Dependency graph**: ```text -┌─────────────────────────────────────────────────────────────────────┐ -│ ci-cd.yml (Main Orchestrator) │ -│ Triggered on: push to main/release/v*, PR, release, tag v*.*.* │ -├─────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────┐ ┌───────┐ ┌────────────────┐ ┌────────┐ │ -│ │ Lint │ │ Audit │ │ Test │ │ CodeQL │ │ -│ │ (5 min)│ │(3 min)│ │ (Multi-stage) │ │ (10m) │ │ -│ └───┬────┘ └───┬───┘ └───┬────────────┘ └───┬────┘ │ -│ │ │ │ │ │ -│ │ │ ┌─────┴──────┐ │ │ -│ │ │ │ unit (3m) │ │ │ -│ │ │ │ integ (5m) │ │ │ -│ │ │ │ e2e (7m) │ │ │ -│ │ │ │ long (opt) │ │ │ -│ │ │ │ vlong(opt) │ │ │ -│ │ │ └────────────┘ │ │ -│ │ │ │ │ │ -│ └───────────┴──────────┴────────────────────┘ │ -│ ↓ │ -│ ┌──────────────────────┐ │ -│ │ Ketryx Report Check │ │ -│ │ (Medical Compliance) │ │ -│ └──────────┬───────────┘ │ -│ ↓ │ -│ ┌───────────────┴─────────────────┐ │ -│ │ │ │ -│ ┌────────────┐ ┌────────────┐ │ -│ │ Package │ │ Docker │ │ -│ │ Publish │ │ Publish │ │ -│ │ (on tag) │ │ (on tag) │ │ -│ └────────────┘ └────────────┘ │ -└─────────────────────────────────────────────────────────────────────┘ - -┌───────────────────────────────────────────────────────────────┐ -│ Parallel Entry Points │ -├───────────────────────────────────────────────────────────────┤ -│ prepare-release.yml → Create release branch │ -│ publish-release.yml → Tag + changelog → CI/CD publish │ -│ merge-release.yml → Merge branch into main │ -│ build-native-only.yml → Native executables (6 platforms) │ -│ claude-code-*.yml → PR reviews + interactive sessions │ -│ test-scheduled-*.yml → Staging (6h) + Production (24h) │ -│ audit-scheduled.yml → Security audit (hourly) │ -│ codeql-scheduled.yml → CodeQL scan (weekly) │ -└───────────────────────────────────────────────────────────────┘ +get-commit-message ──> lint, docs, audit, test, codeql + test ──> sonarcloud +lint, audit, test, codeql, sonarcloud, docs ──> ketryx_report_and_check + ├──> package_publish (tags only) + └──> docker_publish (tags only) ``` -## All Workflows Reference - -### Entry Point Workflows (Triggered Directly) - -| Workflow | Triggers | Purpose | Calls | -|----------|----------|---------|-------| -| **ci-cd.yml** | push(main, release/v*), PR, release, tag | Main CI/CD pipeline | _lint,_audit, _test,_codeql, _ketryx,_package-publish, _docker-publish | -| **prepare-release.yml** | workflow_dispatch | Create release branch + bump version | — | -| **publish-release.yml** | workflow_dispatch | Generate changelog, tag, push → CI/CD | — | -| **merge-release.yml** | workflow_dispatch | Merge release branch into main | — | -| **build-native-only.yml** | push, PR, release (if msg contains `build:native:only`) | Native executable builds | _build-native-only | -| **claude-code-interactive.yml** | workflow_dispatch (manual) | Manual Claude sessions | _claude-code (interactive) | -| **claude-code-automation-pr-review.yml** | PR opened/sync (excludes bots) | Automated PR reviews | _claude-code (automation) | -| **test-scheduled-staging.yml** | schedule (every 6h) | Continuous staging validation | _scheduled-test (staging) | -| **test-scheduled-production.yml** | schedule (every 24h) | Daily production validation | _scheduled-test (production) | -| **audit-scheduled.yml** | schedule (hourly) | Security & license audit | _scheduled-audit | -| **codeql-scheduled.yml** | schedule (Tue 3:22 AM) | Weekly CodeQL scan | _codeql | - -### Reusable Workflows (Called by Others) - -| Workflow | Purpose | Duration | Key Outputs | -|----------|---------|----------|-------------| -| **_lint.yml** | Code quality (ruff, pyright, mypy) | ~5 min | Formatted code, type safety | -| **_docs.yml** | Documentation build (Sphinx) | ~3 min | HTML docs, validation | -| **_audit.yml** | Security + license compliance | ~3 min | SBOM (CycloneDX, SPDX), vulnerabilities, licenses | -| **_test.yml** | Multi-stage test execution | ~15 min | Coverage reports, JUnit XML | -| **_codeql.yml** | Security vulnerability scanning | ~10 min | CodeQL SARIF results | -| **_ketryx_report_and_check.yml** | Medical device compliance | ~2 min | Ketryx project report | -| **_package-publish.yml** | PyPI package publishing | ~3 min | Wheel/sdist on PyPI, GitHub release | -| **_docker-publish.yml** | Docker image publishing | ~5 min | Multi-arch Docker images | -| **_build-native-only.yml** | Native executable builds | ~10 min/platform | aignostics.7z per platform | -| **_claude-code.yml** | Claude Code execution | varies | Code changes, analysis | -| **_scheduled-audit.yml** | Scheduled audit runner | ~5 min | Audit reports + BetterStack heartbeat | -| **_scheduled-test.yml** | Scheduled test runner | ~10 min | Test reports + BetterStack heartbeat | - -## Test Execution Strategy - -### Test Categories - -The SDK has **7 test categories** with different execution strategies. - -**CRITICAL REQUIREMENT**: Every test **MUST** be marked with at least one of: `unit`, `integration`, or `e2e`. Tests without these markers will **NOT run in CI** because the pipeline explicitly filters by these markers. - -```python -# ✅ CORRECT - Has category marker -@pytest.mark.unit -def test_something(): - pass - - -# ❌ INCORRECT - No category marker, will NOT run in CI -def test_something_else(): - pass - - -# ✅ CORRECT - Multiple markers including category -@pytest.mark.e2e -@pytest.mark.long_running -def test_complex_workflow(): - pass -``` - -#### 1. Unit Tests - -**Marker**: `unit` - -**Characteristics**: - -* Fast, isolated tests with no external dependencies -* No API calls, no file I/O (except temp files) -* ~3 minutes total execution time - -**Parallelization**: `XDIST_WORKER_FACTOR=0.0` (sequential execution) - -* Fast enough that parallelization overhead reduces performance -* Single worker for predictable execution - -**CI Behavior**: Always run in all CI contexts - -**Run locally**: - -```bash -make test_unit -# Or directly: -uv run pytest -m "unit and not long_running and not very_long_running" -v -``` - -#### 2. Integration Tests - -**Marker**: `integration` - -**Characteristics**: - -* Tests with mocked external services (API responses, S3 calls) -* Some I/O but mostly CPU-bound -* ~5 minutes total execution time - -**Parallelization**: `XDIST_WORKER_FACTOR=0.2` (20% of logical CPUs) - -* Limited parallelism due to CPU-bound nature -* Example: 8 CPU machine → max(1, int(8 * 0.2)) = 1 worker - -**CI Behavior**: Always run in all CI contexts - -**Run locally**: - -```bash -make test_integration -# Or directly: -uv run pytest -m "integration and not long_running and not very_long_running" -v -``` - -#### 3. E2E Tests (Regular) - -**Marker**: `e2e` (excluding `long_running` and `very_long_running`) - -**Characteristics**: +`package_publish`/`docker_publish` run only on `refs/tags/v*`. +`ketryx_report_and_check` is skipped for `dependabot[bot]`. -* Real API calls to staging environment -* Network I/O bound -* ~7 minutes total execution time +## Test execution (`_test.yml`) -**Parallelization**: `XDIST_WORKER_FACTOR=1.0` (100% of logical CPUs) +**Matrix axis is the runner/OS**, not the Python version. Python-version +iteration happens inside `nox`, driven by the `test_*_matrix` make targets. -* Full parallelization maximizes throughput for I/O-bound tests -* Example: 8 CPU machine → 8 workers +`generate-matrix` builds the runner list: -**CI Behavior**: Always run in all CI contexts +- Always: `ubuntu-latest` (not experimental). +- Plus (unless `skip:test:matrix-runner` in commit message or PR label): + `ubuntu-24.04-arm`, `macos-latest`, `macos-15-intel`, `windows-latest` — all + `experimental: true` (job-level `continue-on-error`). -**Requirements**: `.env` file with staging credentials +Each test category is a **step** using `./.github/actions/run-tests`, all with +`continue-on-error: true`, so every suite runs and Codecov always gets a +complete report. A final **`Assert no test failures`** gate step fails the job +if any category failed. -**Run locally**: +| Step | make target | Skip marker | +|------|-------------|-------------| +| unit | `test_unit_matrix` | `skip:test:unit` | +| integration | `test_integration_matrix` | `skip:test:integration` | +| e2e (regular) | `test_e2e_matrix` | `skip:test:e2e` | +| e2e long running | `test_long_running` | `skip:test:long_running` | +| e2e very long running | `test_very_long_running` | (opt-in, see below) | -```bash -make test_e2e -# Or directly: -uv run pytest -m "e2e and not long_running and not very_long_running" -v -``` - -#### 4. Long Running Tests - -**Marker**: `long_running` - -**Characteristics**: - -* E2E tests taking >30 seconds each -* Typically involve large file operations or complex workflows -* Variable duration (5-15 minutes total) +**Very long running** runs only when enabled: `enable:test:very_long_running` +in commit message or PR label, or any push to a `release/v*` branch. -**Parallelization**: `XDIST_WORKER_FACTOR=2.0` (200% of logical CPUs) +**Parallelism** is set by `XDIST_WORKER_FACTOR` (worker count = +`max(1, int(cpu_count * factor))`). CI runs the `*_matrix` targets — unit and +integration matrix use `0.5`, e2e matrix uses `1`. The non-matrix local targets +(`test_unit`=0.0, `test_integration`=0.2, `test_e2e`=1, long/very_long=2) run a +single Python version. See the `Makefile` for the authoritative values. -* Aggressive parallelization to reduce wall-clock time -* Example: 8 CPU machine → 16 workers +### Test markers -**CI Behavior**: +Every test **must** carry at least one of `unit`, `integration`, `e2e` or it +will not run in CI (the matrix targets filter by marker). The full marker list +(incl. `long_running`, `very_long_running`, `scheduled`, `scheduled_only`, +`sequential`) lives in `pyproject.toml` `[tool.pytest.ini_options]`. -* **Draft PRs**: Always skipped -* **Non-draft PRs**: Run by default UNLESS: - * PR has label `skip:test:long_running`, OR - * Commit message contains `skip:test:long_running` -* **Main branch**: Always run -* **Releases**: Always run - -**Skip in PR**: +Skip/enable labels & commit shortcuts: `skip:ci`, `build:native:only`, +`skip:test:matrix-runner`, `skip:test:{unit,integration,e2e,long_running}`, +`enable:test:very_long_running`, `skip:codecov`. ```bash -# Add label gh pr edit --add-label "skip:test:long_running" - -# Or commit message git commit -m "fix: something skip:test:long_running" ``` -**Run locally**: +## Native builds (`build-native-only.yml`) -```bash -make test_long_running -# Or directly: -uv run pytest -m long_running -v -``` +Triggered by `build:native:only` in commit message or PR label; skips the main +pipeline and only builds native executables via `_build-native-only.yml`. +Runners: `ubuntu-latest` (stable) plus experimental `ubuntu-24.04-arm`, +`macos-latest`, `macos-15-intel`, `windows-latest` (UPX), `windows-11-arm`. +Output: `aignostics.7z` per platform. Local: `make dist_native`. -#### 5. Very Long Running Tests +## Claude Code (`_claude-code.yml`) -**Marker**: `very_long_running` +One reusable workflow, two modes via the `mode` input: -**Characteristics**: +- **interactive** — full git history (`fetch-depth: 0`). + `claude-code-interactive.yml` triggers it on `@claude` mentions + (issue/PR comments, reviews, issues, PRs) and `workflow_dispatch`. It passes + only `mode` and `track_progress` — there are **no** `prompt`/`max_turns`/ + `environment` inputs to fill in. +- **automation** — shallow history (`fetch-depth: 1`), runs a fixed `prompt`. + `claude-code-automation-pr-review.yml` triggers it when a PR carries the + `claude` label or becomes `ready_for_review`. `max_turns` is not set, so it + uses the `_claude-code.yml` default (200). Review tools include + `mcp__github_inline_comment__create_inline_comment` and posts a + `claude:review:passed` / `claude:review:failed` verdict label. -* E2E tests taking >5 minutes each -* Extremely resource-intensive operations -* 15+ minutes total execution time +`_claude-code.yml` inputs: `mode` (required), `prompt`, `max_turns` (default +`200`), `allowed_tools` (default defined in the workflow — read it there rather +than copying), `track_progress`, `use_sticky_comment`. Model is +`claude-sonnet-4-5-20250929`. -**Parallelization**: `XDIST_WORKER_FACTOR=2.0` (200% of logical CPUs) +**Setup**: installs `uv`, dev tools (`_install_dev_tools.bash`), +`uv sync --all-extras --frozen`, headless display. Only secret available is +`ANTHROPIC_API_KEY` — Claude Code workflows intentionally have **no** platform +or GCP credentials. -**CI Behavior**: +## Scheduled jobs -* **Automatically run on every push to `release/v*` branches** (no opt-in needed) -* **Otherwise never run by default** — must be explicitly enabled via: - * PR label `enable:test:very_long_running`, OR - * Commit message contains `enable:test:very_long_running` +- **Testing** — staging and production each run **hourly** (`0 * * * *`, via + `_scheduled-test-hourly.yml`) and **daily at 12:00 UTC** (`0 12 * * *`, via + `_scheduled-test-daily.yml`). Run `make test_scheduled`, send a BetterStack + heartbeat. Staging → `https://platform-staging.aignostics.com`, production → + `https://platform.aignostics.com`. +- **Audit** — hourly (`_scheduled-audit.yml`): `pip-audit`, `pip-licenses`, + Trivy SBOM scan + BetterStack heartbeat. +- **CodeQL** — weekly Tue 03:22 (`_codeql.yml`). -**Enable in PR**: +### BetterStack heartbeats -```bash -# Add label -gh pr edit --add-label "enable:test:very_long_running" +Scheduled test/audit jobs POST a metadata JSON payload to +`{HEARTBEAT_URL}/{EXIT_CODE}` (0 = success) so BetterStack alerts on failures or +missed beats. If the URL secret is unset the step logs a warning and continues. +Secrets: `BETTERSTACK_AUDIT_HEARTBEAT_URL`, +`BETTERSTACK_HEARTBEAT_URL_{STAGING,PRODUCTION}` (hourly), +`BETTERSTACK_HEARTBEAT_URL_FLOWS_{STAGING,PRODUCTION}` (daily). -# Or commit message -git commit -m "test: enable very long tests enable:test:very_long_running" -``` +## Environments & secrets -**Run locally**: +**Staging** (`platform-staging.aignostics.com`) is the default for all PR CI and +E2E. **Production** (`platform.aignostics.com`) is used only by scheduled jobs +and release validation — never in PR CI. -```bash -make test_very_long_running -# Or directly: -uv run pytest -m very_long_running -v -``` - -#### 6. Sequential Tests - -**Marker**: `sequential` - -**Characteristics**: - -* Tests that must run in specific order -* Have interdependencies or shared state -* Cannot be parallelized +GitHub secrets: `ANTHROPIC_API_KEY`, +`AIGNOSTICS_CLIENT_ID_DEVICE_{STAGING,PRODUCTION}`, +`AIGNOSTICS_REFRESH_TOKEN_{STAGING,PRODUCTION}`, +`GCP_CREDENTIALS_{STAGING,PRODUCTION}` (base64 JSON), the BetterStack URLs above, +`CODECOV_TOKEN`, `SONAR_TOKEN`, `SENTRY_DSN`/`SENTRY_AUTH_TOKEN`, +`UV_PUBLISH_TOKEN`, `DOCKER_USERNAME`/`DOCKER_PASSWORD`, +`KETRYX_PROJECT`/`KETRYX_API_KEY`, `SLACK_*_RELEASE_ANNOUNCEMENT`. -**Parallelization**: None (single worker) - -**CI Behavior**: Always run in CI (as part of test suite) - -**Run locally**: - -```bash -make test_sequential -# Or directly: -uv run pytest -m sequential -v -``` - -#### 7. Scheduled Tests - -**Markers**: `scheduled` or `scheduled_only` - -**Characteristics**: - -* Tests designed for continuous validation against live environments -* May have different behavior in staging vs production -* Validate API contract stability - -**CI Behavior**: - -* **`scheduled`**: Run in scheduled jobs AND can run in regular CI -* **`scheduled_only`**: ONLY run in scheduled jobs (never in PR CI) - -**Scheduling**: - -* **Staging**: Every 6 hours (`test-scheduled-staging.yml`) -* **Production**: Every 24 hours (`test-scheduled-production.yml`) - -**Run locally**: - -```bash -make test_scheduled -# Or directly: -uv run pytest -m "(scheduled or scheduled_only)" -v -``` - -### Test Execution Flow in CI - -**Standard PR Flow** (_test.yml): - -```text -1. Unit Tests (3 min) - ├─ Python 3.11 ─┐ - ├─ Python 3.12 ─┼─ Parallel execution - ├─ Python 3.13 ─┤ - └─ Python 3.14 ─┘ - -2. Integration Tests (5 min) - ├─ Python 3.11 ─┐ - ├─ Python 3.12 ─┼─ Parallel execution - ├─ Python 3.13 ─┤ - └─ Python 3.14 ─┘ - -3. E2E Regular (7 min) - ├─ Python 3.11 ─┐ - ├─ Python 3.12 ─┼─ Parallel execution - ├─ Python 3.13 ─┤ - └─ Python 3.14 ─┘ - -4. Long Running (if not skipped) - └─ Python 3.14 only (single version) - -5. Very Long Running (if explicitly enabled) - └─ Python 3.14 only (single version) -``` - -**Matrix Testing**: - -* Unit, Integration, E2E run on **all four Python versions** (3.11, 3.12, 3.13, 3.14) -* Long running and very long running run on **Python 3.14 only** to save CI time - -### Skip Markers System - -**PR Labels** (preferred method): - -* `skip:ci` - Skip entire CI pipeline -* `build:native:only` - Only build native executables -* `skip:test:long_running` - Skip long-running tests -* `enable:test:very_long_running` - Enable very long running tests -* `skip:test:unit` - Skip unit tests (not recommended) -* `skip:test:integration` - Skip integration tests (not recommended) -* `skip:test:e2e` - Skip e2e tests (not recommended) - -**Commit Message Shortcuts**: - -* `skip:ci` - Skip entire CI pipeline -* `build:native:only` - Only build native executables -* `skip:test:long_running` - Skip long-running tests -* `enable:test:very_long_running` - Enable very long running tests - -**Usage**: - -```bash -# Add label to PR -gh pr edit --add-label "skip:test:long_running" - -# Or in commit message -git commit -m "fix: issue skip:test:long_running" -``` - -## Main CI/CD Pipeline (ci-cd.yml) - -**Purpose**: Orchestrates the entire CI/CD pipeline for all branches, PRs, and releases. - -**Triggers**: - -* `push` to `main` branch -* `push` to `release/v*` branches (release branch CI) -* `pull_request` to `main` (opened, synchronize, reopened) -* `release` created -* `tags` matching `v*.*.*` - -**Concurrency Control**: - -```yaml -group: ${{ github.workflow }}-${{ github.ref_name }}-${{ github.event.pull_request.number || github.sha }} -cancel-in-progress: true -``` - -Cancels in-progress runs when new commits are pushed to same PR/branch. - -**Skip Conditions**: - -* Commit message contains `skip:ci` -* Commit message contains `build:native:only` -* PR has label `skip:ci` or `build:native:only` - -**Job Dependencies**: - -```text -lint ──┐ -audit ─┼──→ ketryx_report_and_check ──┬──→ package_publish (tags only) -test ──┤ └──→ docker_publish (tags only) -codeql─┘ -``` - -**Jobs**: - -1. **lint** (~5 min): Code quality checks (ruff, pyright, mypy) -2. **audit** (~3 min): Security audit (pip-audit, pip-licenses, SBOMs) -3. **test** (~15 min): Multi-stage test suite (unit, integration, e2e, long_running, very_long_running) -4. **codeql** (~10 min): CodeQL security analysis -5. **ketryx_report_and_check**: Medical device compliance reporting -6. **package_publish** (tags only): Build and publish to PyPI, create GitHub release, send Slack notification -7. **docker_publish** (tags only): Build and publish Docker images to Docker Hub - -## Native Build System - -### Purpose - -Build standalone native executables for distribution without Python runtime dependency. - -### Supported Platforms - -| Platform | Runner | Status | Notes | -|----------|--------|--------|-------| -| **Linux x86_64** | ubuntu-latest | ✅ Stable | Primary platform | -| **Linux ARM64** | ubuntu-24.04-arm | ⚠️ Experimental | continue-on-error | -| **macOS ARM (M1+)** | macos-latest | ⚠️ Experimental | Apple Silicon | -| **macOS Intel** | macos-15-intel | ⚠️ Experimental | Intel chips | -| **Windows x86_64** | windows-latest | ⚠️ Experimental | With UPX compression | -| **Windows ARM64** | windows-11-arm | ⚠️ Experimental | ARM-based Windows | - -### Build Process - -1. **Setup**: Install uv package manager -2. **Windows Only**: Install UPX compression tool via chocolatey -3. **Build**: Run `make dist_native` - * Uses PyInstaller to create standalone executable - * Bundles Python runtime and all dependencies - * Compresses with UPX (Windows only) -4. **Package**: Creates `aignostics.7z` archive -5. **Upload**: Artifacts stored for 1 day with retention - -### Triggering Native Builds - -**Automatic**: Add commit message or PR label: - -```bash -git commit -m "build:native:only: create native builds" -# Or -gh pr edit --add-label "build:native:only" -``` - -**Effect**: Skips main CI/CD pipeline, only runs native builds. - -**Local Build**: - -```bash -make dist_native -# Output: dist_native/aignostics.7z -``` - -## Claude Code Integration - -### Overview - -Claude Code is integrated into the CI/CD pipeline for: - -1. **Automated PR Reviews** - Every PR gets automatic code review -2. **Interactive Sessions** - Manual Claude assistance for development tasks - -### Workflow: _claude-code.yml - -**Two Execution Modes**: - -#### 1. Interactive Mode - -* **Use Case**: Manual Claude sessions for development assistance -* **Behavior**: Iterative conversation, Claude can ask questions -* **Git History**: Full (`fetch-depth: 0`) -* **Duration**: Variable (controlled by `max_turns`) - -**Trigger**: - -```bash -# GitHub Actions UI: Actions → Claude Code Interactive → Run workflow -# Inputs: -# - prompt: "Your task description" -# - max_turns: 200 (default) -``` - -#### 2. Automation Mode - -* **Use Case**: Single-shot automated tasks (PR reviews, automated fixes) -* **Behavior**: Non-interactive, runs predefined prompt -* **Git History**: Shallow (`fetch-depth: 1`) -* **Duration**: Typically 5-10 minutes - -**Triggered by**: `claude-code-automation-pr-review.yml` on PR events - -### Configuration - -**Inputs**: - -```yaml -mode: 'interactive' | 'automation' # Required -prompt: 'string' # For automation mode -max_turns: '200' # Default: 200 -allowed_tools: 'comma,separated,list' # Default: Read,Write,Edit,Glob,Grep,Bash(git:*),Bash(uv:*),Bash(make:*) -``` - -**Environment Setup**: - -1. Installs `uv` package manager -2. Installs dev tools (`.github/workflows/_install_dev_tools.bash`) -3. Syncs Python dependencies (`uv sync --all-extras`) -4. Sets up headless display (for GUI tests) - -**Note**: Claude Code workflows intentionally do NOT have access to Aignostics platform credentials or GCP credentials to prevent accidental credential leakage. - -**Claude Configuration**: - -```bash -claude \ - --max-turns 200 \ - --model claude-sonnet-4-5-20250929 \ - --allowed-tools "Read,Write,Edit,Glob,Grep,Bash(git:*),Bash(uv:*),Bash(make:*),Bash(gh:*),..." \ - --system-prompt "Read the CLAUDE.md file and apply guidance therein" \ - --prompt "${{ inputs.prompt }}" -``` - -**Secrets Required**: - -* `ANTHROPIC_API_KEY` - For Claude Code (only secret available to Claude Code workflows) - -### Automated PR Review (claude-code-automation-pr-review.yml) - -**Purpose**: Automated code review by Claude on every PR. - -**Triggers**: - -* `pull_request` (opened, synchronize) -* **Excludes**: dependabot, renovate PRs - -**Review Prompt**: - -```text -Review this PR thoroughly. Check code quality, test coverage, security, -and adherence to CLAUDE.md guidelines. -``` - -**Features**: - -* Posts inline comments on code -* Checks for common issues -* Validates test coverage -* Reviews documentation -* Maximum 100 turns - -**Tool Access**: - -* `mcp__github_inline_comment__create_inline_comment` - For PR comments -* File operations: `Read`, `Write`, `Edit`, `Glob`, `Grep` -* Git/GitHub: `Bash(git:*)`, `Bash(gh:*)` - -### Manual Claude Sessions (claude-code-interactive.yml) - -**Purpose**: On-demand Claude assistance for complex development tasks. - -**Trigger**: `workflow_dispatch` (manual) - -**Inputs**: - -* `prompt`: What you want Claude to work on -* `max_turns`: How many iterations (default 200) - -**Example Use Cases**: - -* "Refactor module X for better testability" -* "Add comprehensive tests for feature Y" -* "Update documentation for API changes" -* "Debug failing tests in TestClass" - -**Access**: GitHub Actions UI → Claude Code Interactive → Run workflow - -### Best Practices for Claude Code - -**DO**: - -* ✅ Use `--system-prompt` referencing CLAUDE.md -* ✅ Limit tool access (`--allowed-tools`) -* ✅ Set reasonable `--max-turns` -* ✅ Review Claude's changes before merging -* ✅ Let Claude explore workflows and test strategies - -**DON'T**: - -* ❌ Grant unrestricted tool access -* ❌ Skip CLAUDE.md system prompt -* ❌ Merge without human review -* ❌ Add platform/GCP credentials to Claude Code workflows (security risk) - -## Scheduled Jobs - -### Test Validation (Staging & Production) - -**Purpose**: Continuous validation of SDK against live environments. - -#### test-scheduled-staging.yml - -**Schedule**: Every 6 hours - -**Environment**: `https://platform-staging.aignostics.com` - -**Purpose**: - -* Early detection of API regressions -* Validate against latest staging deployment -* Fast feedback loop for breaking changes - -#### test-scheduled-production.yml - -**Schedule**: Every 24 hours - -**Environment**: `https://platform.aignostics.com` - -**Purpose**: - -* Validate SDK works with production API -* Catch discrepancies between staging and production -* Safety net for production deployments - -**Both workflows**: - -* Use `_scheduled-test.yml` reusable workflow -* Run `make test_scheduled` (tests marked `scheduled` or `scheduled_only`) -* Send BetterStack heartbeat for monitoring -* Upload test results and coverage reports - -### Audit Validation (audit-scheduled.yml) - -**Schedule**: Every hour (`0 * * * *`) - -**Purpose**: Continuous security and license compliance monitoring - -**Checks**: - -* `pip-audit`: CVE scanning for known vulnerabilities -* `pip-licenses`: License compliance verification -* Trivy: SBOM vulnerability scanning (CycloneDX + SPDX formats) - -**Workflow**: Uses `_scheduled-audit.yml` - -**Outputs**: - -* SBOM files (JSON, SPDX) -* License reports (CSV, JSON, grouped JSON) -* Vulnerability reports (JSON) -* BetterStack heartbeat - -### CodeQL Scanning (codeql-scheduled.yml) - -**Schedule**: Weekly on Tuesdays at 3:22 AM - -**Purpose**: Comprehensive security analysis with CodeQL - -**Workflow**: Uses `_codeql.yml` - -**Analysis**: Static analysis for Python security vulnerabilities - -## BetterStack Monitoring - -### Purpose - -External monitoring and alerting for scheduled jobs to detect failures outside GitHub. - -### Heartbeat System - -**Implemented in**: - -* `_scheduled-audit.yml` - Audit job monitoring -* `_scheduled-test.yml` - Test job monitoring (staging & production) - -**Functionality**: - -1. Job runs (audit or test) -2. Captures exit code (0 = success, non-zero = failure) -3. Constructs JSON payload with metadata -4. Sends POST request to BetterStack heartbeat URL with exit code appended -5. BetterStack tracks heartbeat and alerts on failures or missed beats - -**Payload Structure**: - -```json -{ - "github": { - "workflow": "Scheduled Test - Staging", - "run_url": "https://github.com/org/repo/actions/runs/12345", - "run_id": "12345", - "job": "test-scheduled", - "sha": "abc123...", - "actor": "github-actions", - "repository": "org/repo", - "ref": "refs/heads/main", - "event_name": "schedule" - }, - "job": { - "status": "success" - }, - "timestamp": "2025-10-19T14:30:00Z" -} -``` - -**URL Format**: `{HEARTBEAT_URL}/{EXIT_CODE}` - -**Required Secrets**: - -* `BETTERSTACK_AUDIT_HEARTBEAT_URL` - For audit jobs -* `BETTERSTACK_HEARTBEAT_URL_STAGING` - For staging test jobs -* `BETTERSTACK_HEARTBEAT_URL_PRODUCTION` - For production test jobs - -**Behavior**: - -* If heartbeat URL is configured: Sends heartbeat regardless of job success/failure -* If heartbeat URL is NOT configured: Logs warning and continues -* Exit code passed to URL allows BetterStack to distinguish success (0) from failures - -## Environment Configuration - -### Staging Environment - -**API Root**: `https://platform-staging.aignostics.com` - -**Secrets**: - -* `AIGNOSTICS_CLIENT_ID_DEVICE_STAGING` -* `AIGNOSTICS_REFRESH_TOKEN_STAGING` -* `GCP_CREDENTIALS_STAGING` -* `BETTERSTACK_HEARTBEAT_URL_STAGING` - -**Use Cases**: - -* PR testing (default for all PRs) -* E2E test execution -* Feature validation -* Claude Code development sessions -* Scheduled validation (every 6 hours) - -### Production Environment - -**API Root**: `https://platform.aignostics.com` - -**Secrets**: - -* `AIGNOSTICS_CLIENT_ID_DEVICE_PRODUCTION` -* `AIGNOSTICS_REFRESH_TOKEN_PRODUCTION` -* `GCP_CREDENTIALS_PRODUCTION` -* `BETTERSTACK_HEARTBEAT_URL_PRODUCTION` - -**Use Cases**: - -* Scheduled tests only (every 24 hours) -* Release validation -* Critical bug verification -* **NEVER use in PR CI** (staging only) - -## Secrets Management - -**GitHub Secrets** (Required): - -* `ANTHROPIC_API_KEY` - Claude Code -* `AIGNOSTICS_CLIENT_ID_DEVICE_{STAGING|PRODUCTION}` -* `AIGNOSTICS_REFRESH_TOKEN_{STAGING|PRODUCTION}` -* `GCP_CREDENTIALS_{STAGING|PRODUCTION}` - Base64 encoded JSON -* `BETTERSTACK_AUDIT_HEARTBEAT_URL` - Audit monitoring -* `BETTERSTACK_HEARTBEAT_URL_{STAGING|PRODUCTION}` - Test monitoring -* `CODECOV_TOKEN` - Coverage reporting to Codecov -* `SONAR_TOKEN` - Code quality reporting to SonarCloud -* `UV_PUBLISH_TOKEN` - PyPI publishing token -* `DOCKER_USERNAME`, `DOCKER_PASSWORD` - Docker Hub credentials -* `KETRYX_PROJECT`, `KETRYX_API_KEY` - Medical device compliance -* `SLACK_WEBHOOK_URL_RELEASE_ANNOUNCEMENT` - Release notifications - -**Local Secrets** (`.env` file for E2E tests): +Local `.env` for E2E: ```bash AIGNOSTICS_API_ROOT=https://platform-staging.aignostics.com @@ -821,301 +187,34 @@ AIGNOSTICS_CLIENT_ID_DEVICE=your-staging-client-id AIGNOSTICS_REFRESH_TOKEN=your-staging-refresh-token ``` -**GCP Credentials** (for bucket access): - -```bash -# In CI: base64 encoded and stored as secret -echo "$GCP_CREDENTIALS" | base64 -d > credentials.json -export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/credentials.json -``` - -## Debugging CI Failures - -### Lint Failures - -**Reproduce locally**: - -```bash -make lint -``` - -**Common Issues**: - -* Ruff formatting: Run `ruff format .` -* Ruff linting: Check `ruff check .` and fix with `--fix` -* PyRight: Type errors (basic mode, see `pyrightconfig.json`) -* MyPy: Type errors (strict mode) - -**Fix**: - -```bash -ruff format . -ruff check . --fix -``` - -### Test Failures - -**Reproduce locally**: - -```bash -# Unit tests -make test_unit - -# Integration tests -make test_integration - -# E2E tests (requires .env with credentials) -make test_e2e - -# Specific test -uv run pytest tests/path/to/test.py::test_name -vv -``` - -**Debug**: - -```bash -# Verbose output -uv run pytest tests/test_file.py -vv - -# Show print statements -uv run pytest tests/test_file.py -s - -# Drop into debugger on failure -uv run pytest tests/test_file.py --pdb +## Releasing a version -# Run single test -uv run pytest tests/test_file.py::test_function -v -``` +Four-phase workflow triggered from a developer machine so Ketryx approvals are +collected *before* the tag (and thus before PyPI publish): -**Check Coverage**: +1. `make prepare-release 1.2.3` → `prepare-release.yml`: creates + `release/vX.Y.Z` from `main`, bumps version + `uv.lock`, pushes. CI runs. +2. Point the Ketryx release at `release/vX.Y.Z`, collect approvals, ensure CI green. +3. `make publish-release` → `publish-release.yml`: generates `CHANGELOG.md`, + creates annotated `vX.Y.Z` tag → CI/CD fires on tag → Ketryx check gates PyPI. +4. `make merge-release` → `merge-release.yml`: merges `release/vX.Y.Z` into + `main` `--no-ff`, deletes the release branch. -```bash -uv run coverage report -uv run coverage html -open htmlcov/index.html -``` +`release/v*` branches should be protected so only `aignostics-release-bot[bot]` +can push. -**Minimum**: 85% coverage required +## Debugging CI failures -### Audit Failures +Reproduce locally with the same make targets CI uses: `make lint`, +`make test_unit` / `test_integration` / `test_e2e` (E2E needs `.env`), +`make dist_native`, `uv run pip-audit`. Coverage minimum is enforced via +Codecov (see `codecov.yml`). For scheduled-job failures, check the BetterStack +dashboard and the workflow run logs (usual causes: API changes in the target +environment, expired credentials). -**Security Vulnerabilities**: +Run scheduled workflows manually: ```bash -uv run pip-audit +gh workflow run scheduled-testing-staging-hourly.yml +gh workflow run scheduled-testing-production-daily.yml ``` - -**Fix**: Update vulnerable dependencies in `pyproject.toml` - -**License Violations**: - -```bash -uv run pip-licenses --allow-only="MIT;Apache-2.0;BSD-3-Clause;..." -``` - -**Fix**: Replace non-compliant dependencies or get approval for license - -### Native Build Failures - -**Platform-specific issues**: - -* Check runner compatibility -* Verify UPX installation (Windows) -* Check PyInstaller compatibility with dependencies - -**Local reproduction**: - -```bash -make dist_native -``` - -**Note**: Experimental platforms (continue-on-error) won't block CI - -### Scheduled Job Failures - -**BetterStack Alerts**: Check BetterStack dashboard for heartbeat failures - -**Investigate**: - -1. Go to GitHub Actions → Scheduled workflow -2. Check recent run logs -3. Look for API changes or credential issues - -**Common causes**: - -* API breaking changes in staging/production -* Expired credentials -* Network issues -* Dependency updates - -## Performance & Optimization - -### Parallel Testing - -**CPU-based distribution**: `-n logical` (uses all logical CPUs) - -**Work stealing**: `--dist worksteal` (dynamic load balancing) - -**XDIST_WORKER_FACTOR**: Controls parallelism (0.0-2.0) - -* 0.0 = Sequential (1 worker) -* 0.2 = 20% of CPUs -* 1.0 = 100% of CPUs -* 2.0 = 200% of CPUs (aggressive for I/O-bound) - -**Calculation**: `max(1, int(cpu_count * factor))` - -**Example** (8 CPU machine): - -* unit: 0.0 → 1 worker (sequential) -* integration: 0.2 → max(1, int(8 * 0.2)) = 1 worker -* e2e: 1.0 → 8 workers -* long_running: 2.0 → 16 workers - -### Caching - -* **uv dependencies**: Cached via `astral-sh/setup-uv` action -* **Docker layers**: Cached by Docker build action -* **Nox virtualenvs**: Reused when possible (`nox.options.reuse_existing_virtualenvs = True`) - -### Typical Run Times - -| Job | Duration | Notes | -|-----|----------|-------| -| Lint | ~5 min | Ruff, PyRight, MyPy | -| Audit | ~3 min | pip-audit, licenses, SBOMs | -| Test (per Python version) | ~5 min | Unit + Integration + E2E (no long_running) | -| Test (full matrix) | ~15 min | All 3 Python versions parallel | -| Test (with long_running) | ~25 min | Adds 10 min for long tests | -| CodeQL | ~10 min | Static analysis | -| Full CI pipeline | ~20-30 min | Depends on test configuration | -| Native builds | ~10 min/platform | 6 platforms in parallel | -| Package publish | ~3 min | Build + upload to PyPI | -| Docker publish | ~5 min | Multi-arch build | - -## Common Workflows - -### Creating a PR - -1. Create feature branch -2. Make changes -3. Run `make lint` and `make test` locally -4. Commit with conventional commit message -5. Push to GitHub -6. Create PR → Triggers: - * Lint checks - * Audit checks - * Test suite (unit, integration, e2e) - * CodeQL scan - * Claude Code automated review -7. **Important**: Add label `skip:test:long_running` to save CI time (unless you need long tests) -8. Address review feedback -9. Merge when all checks pass - -### Releasing a Version - -Releases use a four-phase workflow triggered from the developer's machine via `gh workflow run`. This lets Ketryx compliance approvals be collected *before* the tag (and thus before publishing to PyPI). - -**Phase 1 — Prepare the release branch** (triggers `prepare-release.yml`): - -```bash -make prepare-release 1.2.3 # explicit version -``` - -Creates `release/vX.Y.Z` from `main`, commits version bump + `uv.lock`, pushes. CI runs on the branch automatically. - -**Phase 2 — Collect Ketryx approvals:** - -Point the Ketryx release to `release/vX.Y.Z` and collect approvals. Ensure CI is green. - -**Phase 3 — Publish** (triggers `publish-release.yml`): - -```bash -make publish-release # auto-detects release/v* branch -make publish-release release/v1.2.3 # explicit branch -``` - -Generates `CHANGELOG.md`, creates annotated `vX.Y.Z` tag, pushes → CI/CD fires on tag → Ketryx check must pass before PyPI publish. - -**Phase 4 — Merge back to main** (triggers `merge-release.yml`): - -```bash -make merge-release # auto-detects release/v* branch -make merge-release release/v1.2.3 # explicit branch -``` - -Merges `release/vX.Y.Z` into `main` with `--no-ff`, pushes `main`, deletes the release branch. - -**Note on branch protection**: `release/v*` branches should be protected so that only the GitHub Actions bot (`aignostics-release-bot[bot]`) can push to them. This enforces the server-side workflow. Configure in GitHub Settings → Branches → Branch protection rules. - -### Manual Testing with Claude - -1. Go to: Actions → Claude Code Interactive -2. Click "Run workflow" -3. Fill in: - * **Prompt**: Describe your task - * **Max turns**: 200 (default) - * **Environment**: staging (default) -4. Click "Run workflow" -5. Monitor execution in Actions tab -6. Review changes and create PR if needed - -### Running Scheduled Tests Manually - -```bash -# Staging tests -gh workflow run test-scheduled-staging.yml - -# Production tests (use with caution) -gh workflow run test-scheduled-production.yml -``` - -### Building Native Executables - -**Via CI**: - -```bash -git commit -m "build:native:only: create native binaries" -git push -``` - -**Locally**: - -```bash -make dist_native -# Output: dist_native/aignostics.7z -``` - -## Workflow Files Summary - -| File | Type | Purpose | Duration | -|------|------|---------|----------| -| `ci-cd.yml` | Entry | Main pipeline orchestration | ~20 min | -| `prepare-release.yml` | Entry | Create release branch + bump version | ~2 min | -| `publish-release.yml` | Entry | Generate changelog, create tag, push | ~2 min | -| `merge-release.yml` | Entry | Merge release branch into main | ~1 min | -| `build-native-only.yml` | Entry | Native build trigger | ~60 min (6 platforms) | -| `claude-code-interactive.yml` | Entry | Manual Claude sessions | varies | -| `claude-code-automation-pr-review.yml` | Entry | Automated PR reviews | ~10 min | -| `test-scheduled-staging.yml` | Entry | Staging validation | ~10 min | -| `test-scheduled-production.yml` | Entry | Production validation | ~10 min | -| `audit-scheduled.yml` | Entry | Security audit | ~5 min | -| `codeql-scheduled.yml` | Entry | CodeQL scan | ~10 min | -| `_lint.yml` | Reusable | Code quality checks | ~5 min | -| `_docs.yml` | Reusable | Documentation build | ~3 min | -| `_audit.yml` | Reusable | Security & license | ~3 min | -| `_test.yml` | Reusable | Test execution | ~15 min | -| `_codeql.yml` | Reusable | Security scanning | ~10 min | -| `_ketryx_report_and_check.yml` | Reusable | Compliance reporting | ~2 min | -| `_package-publish.yml` | Reusable | PyPI publishing | ~3 min | -| `_docker-publish.yml` | Reusable | Docker publishing | ~5 min | -| `_build-native-only.yml` | Reusable | Native builds | ~10 min/platform | -| `_claude-code.yml` | Reusable | Claude Code execution | varies | -| `_scheduled-audit.yml` | Reusable | Scheduled audit runner | ~5 min | -| `_scheduled-test.yml` | Reusable | Scheduled test runner | ~10 min | - ---- - -**Built with operational excellence for medical device software development.** - -**Note**: See root `CLAUDE.md` and `Makefile` for development commands. This document focuses on CI/CD workflows and GitHub Actions. diff --git a/CLAUDE.md b/CLAUDE.md index 444b96412..d278164b5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -52,6 +52,8 @@ Every module has detailed CLAUDE.md documentation. For module-specific guidance, * [src/aignostics/system/CLAUDE.md](src/aignostics/system/CLAUDE.md) - System diagnostics * [tests/CLAUDE.md](tests/CLAUDE.md) - Test suite documentation +Modules without their own CLAUDE.md (see the source directory directly): `dicom/`, `idc/`, `marimo/`, `thumbnail/`, `tiff/`. + ## Development Commands **Primary workflow commands (use these):** @@ -74,19 +76,17 @@ make audit # Security and license compliance checks **Testing:** -* Pytest with 85% minimum coverage requirement +* Coverage thresholds live in `codecov.yml` (project/patch targets). There is no local `--fail-under` gate. * Default timeout: 10 seconds (override with `@pytest.mark.timeout(timeout=N)`) * Use `uv run pytest tests/path/to/test.py::test_function` for single tests * See **Testing Workflow** section below for complete marker documentation * Special test commands: `make test_unit`, `make test_integration`, `make test_e2e`, `make test_long_running`, `make test_very_long_running`, `make test_sequential`, `make test_scheduled` -**Type Checking (NEW in v1.0.0-beta.7 - Dual Type Checkers):** +**Type Checking (dual type checkers):** * **MyPy**: Strict mode enforced (`make lint` runs MyPy) * **PyRight**: Basic mode with selective exclusions (`pyrightconfig.json`) - * Excludes: tests, codegen, third_party modules, notebook, dataset, wsi - * Mode: `basic` (less strict than MyPy for compatibility) - * Both type checkers must pass in CI/CD +* Both type checkers must pass in CI/CD * All public APIs require type hints * Use `from __future__ import annotations` for forward references @@ -124,7 +124,7 @@ Each module follows a consistent three-layer architecture: Module/ ├── _service.py # Business logic layer (core operations) ├── _cli.py # CLI presentation layer (Typer commands) -├── _gui.py # GUI presentation layer (NiceGUI interface) +├── _gui.py # GUI presentation layer (NiceGUI interface; a package dir _gui/ when large, e.g. application/_gui/) ├── _settings.py # Configuration (Pydantic models) └── CLAUDE.md # Comprehensive documentation ``` @@ -203,13 +203,13 @@ Module/ * QuPath installation and lifecycle * Project management * Script execution -* *Dependencies*: `utils`, requires `ijson` +* *Dependencies*: `utils` (uses `ijson`, a core dep) **notebook** - Interactive analysis: * Marimo notebook server * Process management -* *Dependencies*: `utils`, requires `marimo` +* *Dependencies*: `utils` (uses `marimo`, a core dep) ### System Modules @@ -341,15 +341,16 @@ aignostics mcp list-tools ### GUI Launch -```bash -# Install with GUI support -pip install "aignostics[gui]" +The GUI (NiceGUI) ships in the core install — there is no `[gui]` extra. The +launcher is gated on `find_spec("nicegui")`, `find_spec("webview")`, and not +running in a container (see `src/aignostics/cli.py`). -# Launch desktop interface -aignostics gui +```bash +# Launch the desktop Launchpad +aignostics launchpad # Or with uvx -uvx --with "aignostics[gui]" aignostics gui +uvx aignostics launchpad ``` ## Code Standards @@ -418,30 +419,14 @@ aignostics-python-sdk/ └── CLAUDE.md # This file ``` -**Build configuration:** +**Build configuration** — all tool config is inlined in `pyproject.toml`; there +are no standalone `ruff.toml`, `cliff.toml`, or `.coveragerc` files: -* `pyproject.toml` - Package metadata and dependencies -* `noxfile.py` - **Enhanced with SDK metadata schema generation task** (NEW) -* `ruff.toml` - Linting and formatting rules +* `pyproject.toml` - Package metadata, dependencies, and `[tool.ruff]`, + `[tool.git-cliff]`, `[tool.coverage.run]`, `[tool.pytest.ini_options]` +* `noxfile.py` - Build sessions; also regenerates the versioned SDK metadata + JSON Schema into `docs/source/_static/` during doc builds * `.pre-commit-config.yaml` - Git hooks -* `cliff.toml` - Changelog generation - -**Noxfile Enhancements:** - -The `noxfile.py` now includes automated SDK metadata schema generation: - -```python -def _generate_sdk_metadata_schema(session: nox.Session) -> None: - """Generate versioned JSON Schema for SDK metadata. - - - Calls `aignostics sdk metadata-schema` CLI command - - Extracts schema version from $id field - - Outputs both versioned (v0.0.1) and latest files - - Published to docs/source/_static/ - """ -``` - -This ensures the JSON Schema is automatically regenerated during documentation builds. ## Development Guidelines @@ -493,7 +478,7 @@ def action_command(param: str): ### Testing Requirements -* Minimum 85% code coverage +* Coverage targets defined in `codecov.yml` * Unit tests for all public methods * Integration tests for CLI commands * Mock external dependencies @@ -503,11 +488,10 @@ def action_command(param: str): ### Module Loading -Some modules have conditional loading based on dependencies: - -* **qupath** requires `ijson` package -* **gui** requires `nicegui` package -* **notebook** requires `marimo` package +`nicegui`, `marimo`, and `ijson` are core dependencies, so modules using them +load unconditionally. The Launchpad CLI command is still gated at runtime on +`find_spec("nicegui")`, `find_spec("webview")`, and not running in a container +(see `src/aignostics/cli.py`). ### Platform Authentication @@ -516,205 +500,41 @@ Some modules have conditional loading based on dependencies: * 5-minute refresh buffer before expiry * OAuth 2.0 device flow -### SDK Metadata System (ENHANCED - Run v0.0.6, Item v0.0.3) - -**Automatic Run & Item Tracking**: Every application run and item submitted through the SDK automatically includes comprehensive metadata about the execution context, with support for tags and timestamps. - -**Key Features:** - -* **Automatic Attachment**: SDK metadata added to every run and item without user action -* **Environment Detection**: Automatically detects script/CLI/GUI and user/test/bridge contexts -* **CI/CD Integration**: Captures GitHub Actions workflow information and pytest test context -* **User Information**: Includes authenticated user and organization details -* **Schema Validation**: Pydantic-based validation with JSON Schema (Run: v0.0.6, Item: v0.0.3) -* **Versioned Schema**: Published JSON Schema at `docs/source/_static/sdk_{run|item}_custom_metadata_schema_*.json` -* **Tags Support** (NEW): Associate runs and items with searchable tags -* **Timestamps** (NEW): Track creation and update times (`created_at`, `updated_at`) -* **Metadata Updates** (NEW): Update custom metadata via CLI and GUI -* **Item Metadata** (NEW): Separate schema for item-level metadata including platform bucket information - -**What's Tracked (Run Level):** - -* Submission metadata (date, interface, initiator) -* Enhanced user agent with platform and CI/CD context -* User and organization information (when authenticated) -* GitHub Actions workflow details (repository, run URL, runner info) -* Pytest test context (current test, markers) -* Workflow control flag (onboard_to_portal) -* Scheduling information (due dates, deadlines) -* Optional user notes -* **Tags** (NEW): Set of tags for filtering (`set[str]`) -* **Timestamps** (NEW): `created_at`, `updated_at` - -**What's Tracked (Item Level - NEW):** - -* **Platform Bucket Metadata**: Cloud storage location (bucket name, object key, signed URL) -* **Tags**: Item-level tags (`set[str]`) -* **Timestamps**: `created_at`, `updated_at` - -**CLI Commands:** - -```bash -# Export SDK run metadata JSON Schema -aignostics sdk metadata-schema --pretty > run_schema.json - -# Update run custom metadata (including tags) -aignostics application run custom-metadata update RUN_ID \ - --custom-metadata '{"sdk": {"tags": ["experiment-1", "batch-A"]}}' - -# Dump run custom metadata as JSON -aignostics application run custom-metadata dump RUN_ID --pretty - -# Find runs by tags -aignostics application run list --tags experiment-1,batch-A -``` - -**Implementation:** - -* Module: `platform._sdk_metadata` -* **Run Functions**: `build_run_sdk_metadata()`, `validate_run_sdk_metadata()`, `get_run_sdk_metadata_json_schema()` -* **Item Functions** (NEW): `build_item_sdk_metadata()`, `validate_item_sdk_metadata()`, `get_item_sdk_metadata_json_schema()` -* Integration: Automatic in `platform.resources.runs.submit()` -* User Agent: Enhanced `utils.user_agent()` with CI/CD context -* Tests: Comprehensive test suite in `tests/aignostics/platform/sdk_metadata_test.py` -* **Schema Files**: `sdk_run_custom_metadata_schema_v0.0.6.json` and `sdk_item_custom_metadata_schema_v0.0.3.json` - -See `platform/CLAUDE.md` for detailed documentation. - -### Operation Caching & Retry System (NEW in v1.0.0-beta.7) - -**Enterprise-Grade Performance**: The SDK now implements intelligent operation caching and retry logic to ensure reliability and performance in production environments. +### SDK Metadata System -**Operation Caching (`platform/_operation_cache.py`):** +Every run and item submitted through the SDK gets custom metadata attached +automatically: submission context (script/CLI/launchpad, initiator), +CI/CD + pytest context, authenticated user/org, tags, and timestamps. Validated +against versioned Pydantic-derived JSON Schemas published under +`docs/source/_static/`. -**Key Features:** +* Implementation: `platform/_sdk_metadata.py` (`build_*`/`validate_*`/`get_*_json_schema` functions) +* Schema version constants: `SDK_METADATA_SCHEMA_VERSION` / `ITEM_SDK_METADATA_SCHEMA_VERSION` in that file — do not hardcode the numbers here +* Integration: automatic in run submit; user agent built by `utils.user_agent()` +* CLI: `aignostics sdk metadata-schema`, `aignostics application run custom-metadata update|dump` -* **Token-Aware Caching**: Per-user cache isolation prevents data leakage -* **Configurable TTLs**: 5 minutes for stable data (apps/versions), 15 seconds for dynamic data (runs) -* **Automatic Invalidation**: All caches cleared on mutations (submit/cancel/delete) -* **Memory Efficient**: Dictionary-based storage with automatic expiration +See `platform/CLAUDE.md` for the full field list and CLI details. -**Cached Operations:** +### Operation Caching & Retry System -* `Client.me()` - User information (5 min TTL) -* `Client.application()` / `application_version()` - Application metadata (5 min TTL) -* `Applications.list()` / `details()` - Application lists (5 min TTL) -* `Runs.details()` / `results()` / `list()` - Run data (15 sec TTL) +`platform/_operation_cache.py` provides per-user (token-aware) caching of +read operations with per-kind TTLs and full invalidation on mutations, plus +tenacity-based retry with exponential backoff + jitter for transient errors +(5xx, timeouts, pool/connection errors). Read calls accept `nocache=True` to +bypass the cache. TTLs, retry attempts, and timeouts are configurable via +`AIGNOSTICS_*` env vars. -**Performance Impact:** +See `platform/CLAUDE.md` for cached-operation list, env var names, and patterns. -* Cache Hit: ~0.1ms (1000x faster than API call) -* Cache Miss: Standard API latency (50-500ms) -* Typical Speedup: 100-1000x for repeated reads within TTL +### Run / Item / Artifact State Models -**Retry Logic with Exponential Backoff:** +State is enum-based: `RunState` / `ItemState` / `ArtifactState` +(PENDING → PROCESSING → TERMINATED), paired with termination-reason enums +(`RunTerminationReason`, `ItemTerminationReason`, `ArtifactTerminationReason`) +that separate "what happened" from "why". `RunItemStatistics` carries aggregate +counts; `RunOutput`/`ItemOutput`/`ArtifactOutput` bundle state + reason. -**Key Features:** - -* **Tenacity-Based**: Industry-standard retry library with exponential backoff -* **Configurable**: Per-operation retry attempts (default: 4), wait times (0.1s-60s), timeouts (30s) -* **Smart Exceptions**: Only retries transient errors (5xx, timeouts, connection issues) -* **Jitter**: Randomized wait times prevent thundering herd problem - -**Retryable Exceptions:** - -* ServiceException (5xx server errors) -* Urllib3TimeoutError -* PoolError (connection pool exhausted) -* IncompleteRead / ProtocolError / ProxyError - -**Retry Pattern:** - -``` -Attempt 1: Immediate -Attempt 2: ~100ms wait -Attempt 3: ~200-400ms wait (exponential + jitter) -Attempt 4: ~400-800ms wait (capped at 60s max) -``` - -**Configuration:** - -```bash -# Example .env configuration -AIGNOSTICS_ME_RETRY_ATTEMPTS=4 -AIGNOSTICS_ME_RETRY_WAIT_MIN=0.1 -AIGNOSTICS_ME_RETRY_WAIT_MAX=60.0 -AIGNOSTICS_ME_TIMEOUT=30.0 -AIGNOSTICS_ME_CACHE_TTL=300 - -AIGNOSTICS_RUN_RETRY_ATTEMPTS=4 -AIGNOSTICS_RUN_TIMEOUT=30.0 -AIGNOSTICS_RUN_CACHE_TTL=15 -``` - -**Cache Control:** - -```python -# Bypass cache for specific operations (useful in tests or when fresh data is required) -run = client.runs.details(run_id, nocache=True) # Force API call -applications = client.applications.list(nocache=True) # Bypass cache -``` - -**Design Decisions:** - -* ✅ **Read-Only Retries**: Only safe, idempotent read operations retry -* ✅ **Global Cache Clearing**: Simple consistency model - clear everything on writes -* ✅ **Cache Bypass** (NEW): `nocache=True` parameter forces fresh API calls -* ✅ **Logging**: Warnings logged before retry sleeps for observability -* ✅ **Re-raise**: Original exception re-raised after exhausting retries - -See `platform/CLAUDE.md` for implementation details and usage patterns. - -### API v1.0.0-beta.7 State Models (MAJOR CHANGE) - -**Breaking Change**: Complete refactoring of run, item, and artifact state management with enum-based models and termination reasons. - -**New State Enums:** - -* `RunState`: PENDING → PROCESSING → TERMINATED -* `ItemState`: PENDING → PROCESSING → TERMINATED -* `ArtifactState`: PENDING → PROCESSING → TERMINATED - -**New Termination Reason Enums:** - -* `RunTerminationReason`: ALL_ITEMS_PROCESSED, CANCELED_BY_USER, CANCELED_BY_SYSTEM -* `ItemTerminationReason`: SUCCEEDED, USER_ERROR, SYSTEM_ERROR, SKIPPED -* `ArtifactTerminationReason`: SUCCEEDED, USER_ERROR, SYSTEM_ERROR - -**New Models:** - -* `RunItemStatistics` - Aggregate counts (total, succeeded, user_error, system_error, skipped, pending, processing) -* `RunOutput`, `ItemOutput`, `ArtifactOutput` - Structured output models with state + termination_reason - -**Deleted Models (Breaking Changes):** - -* ❌ `UserPayload` → Replaced with `Auth0User` and `Auth0Organization` -* ❌ `PayloadItem` → Replaced with `ItemOutput` -* ❌ `ApplicationVersionReadResponse` → Renamed to `ApplicationVersion` - -**Benefits:** - -1. **Type Safety**: Enum-based states prevent typos -2. **Clear Semantics**: Separate "what happened" (state) from "why" (termination_reason) -3. **Granular Errors**: Distinguish user errors from system errors for better debugging -4. **Progress Tracking**: RunItemStatistics provides real-time aggregate view - -**Usage Example:** - -```python -run = client.run("run-123") -details = run.details() - -if details.output.state == RunState.TERMINATED: - if details.output.termination_reason == RunTerminationReason.ALL_ITEMS_PROCESSED: - print(f"✅ Run complete: {details.output.statistics.succeeded} items succeeded") - print( - f"❌ Failures: {details.output.statistics.user_error} user errors, " - f"{details.output.statistics.system_error} system errors" - ) -``` - -See `platform/CLAUDE.md` for complete state machine diagrams and migration guide. +See `platform/CLAUDE.md` for the state machine and model definitions. ## Testing Workflow @@ -722,92 +542,26 @@ See `platform/CLAUDE.md` for complete state machine diagrams and migration guide The SDK has a comprehensive test suite organized by test type and execution strategy. -**Pytest Configuration**: - -* Default timeout: **10 seconds** per test -* Coverage requirement: **85% minimum** -* Async mode: `auto` (detects async tests automatically) -* Parallel execution: Via pytest-xdist with work stealing - -**Test Markers** (authoritative definitions from `pyproject.toml`): - -**IMPORTANT**: Every test **MUST** have at least one of: `unit`, `integration`, or `e2e` marker, otherwise it will **NOT run in CI**. The CI pipeline explicitly runs tests with these markers only. - -**Test Categories** (Martin Fowler's Solitary vs Sociable distinction): - -* **`unit`** - Solitary unit tests - * Test a layer of a module in isolation with all dependencies mocked (except shared utils and systems module) - * Must pass **offline** (no external service calls) - * Timeout: ≤ 10s (default), must be < 5 min - * ~3 minutes total execution time - -* **`integration`** - Sociable integration tests - * Test interactions across architectural layers (CLI/GUI→Service, Service→Utils) or between modules (Application→Platform) - * Uses real SDK collaborators, real file I/O, real subprocesses, real Docker containers - * Must pass **offline** (mock external services: Aignostics Platform API, Auth0, S3/GCS, IDC) - * Timeout: ≤ 10s (default), must be < 5 min - * ~5 minutes total execution time - -* **`e2e`** - End-to-end tests - * Test complete workflows with **real external network services** (Aignostics Platform API, cloud storage, IDC, etc) - * If timeout ≥ 5 min and < 60 min, additionally mark as `long_running` - * If timeout ≥ 60 min, additionally mark as `very_long_running` - * ~7 minutes total execution time (regular tests only) - -**Test Execution Control Markers**: - -* **`long_running`** - Tests with timeout **≥ 5 min and < 60 min** - * CI/CD runs with **one Python version only** (3.14) - * Excluded by default in `make test` - use `make test_long_running` - * Can be skipped in PRs with `skip:test:long_running` label - -* **`very_long_running`** - Tests with timeout **≥ 60 min** - * CI/CD runs with **one Python version only** (3.14) - * Excluded by default in `make test` - use `make test_very_long_running` - * Only runs when explicitly enabled with `enable:test:very_long_running` label - -**Scheduling Markers**: +**Pytest Configuration**: default timeout 10s per test, async mode `auto`, +coverage via pytest-cov, parallel execution via pytest-xdist. Full config is in +`[tool.pytest.ini_options]` in `pyproject.toml`. -* **`scheduled`** - Tests to run on a schedule - * Still part of non-scheduled test executions - * Run every 6h (staging) and 24h (production) +**Test Markers**: the authoritative marker definitions (with their descriptions) +live in the `markers = [...]` list under `[tool.pytest.ini_options]` in +`pyproject.toml` — read them there rather than duplicating here. -* **`scheduled_only`** - Tests to run **on schedule only** - * Never run in regular CI/CD - * Only in scheduled test workflows +**IMPORTANT**: Every test **MUST** carry at least one of `unit`, `integration`, +or `e2e` — CI only runs tests with these category markers, so an unmarked test +silently never runs. See "Finding Unmarked Tests" below. -**Infrastructure Markers**: +The other markers gate execution: `long_running` / `very_long_running` (excluded +from `make test`; run via `make test_long_running` etc.; skippable/enable-able via +PR labels `skip:test:long_running` / `enable:test:very_long_running`), +`scheduled` / `scheduled_only`, `stress` / `stress_only`, `sequential`, +`docker`, `skip_with_act`, `no_extras`. -* **`sequential`** - Exclude from parallel test execution - * Tests that must run in specific order or have interdependencies - -* **`docker`** - Tests that require Docker - * Docker daemon must be running - -* **`skip_with_act`** - Don't run with [Act](https://github.com/nektos/act) - * For local GitHub Actions testing - -* **`no_extras`** - Tests that require no extras installed - * Test behavior without optional dependencies - -**Test Structure:** - -```text -tests/ -├── conftest.py # Global fixtures and configuration -├── aignostics/ -│ ├── platform/ # Platform module tests -│ │ ├── sdk_metadata_test.py (519 lines) -│ │ ├── authentication_test.py -│ │ ├── client_test.py -│ │ └── resources/ -│ ├── application/ # Application module tests -│ ├── wsi/ # WSI module tests -│ ├── utils/ # Utils module tests -│ │ └── user_agent_test.py (258 lines) -│ └── ... -└── CLAUDE.md # Test suite documentation -``` +Tests live under `tests/aignostics//` mirroring the source layout; see +`tests/CLAUDE.md`. ### Running Tests @@ -898,26 +652,16 @@ uv run pytest -m sequential tests/ ### Coverage Requirements -**Minimum Coverage: 85%** +Thresholds are enforced by Codecov (see `codecov.yml`), not a local +`--fail-under`. Coverage settings — source paths, `omit` list, branch/parallel +mode — live in `[tool.coverage.run]` in `pyproject.toml`. ```bash -# Check coverage -uv run coverage report - -# Generate HTML report -uv run coverage html -open htmlcov/index.html - -# Coverage enforced in CI -uv run coverage report --fail-under=85 +# Coverage is collected automatically by `make test` (pytest-cov) +uv run coverage report # text summary +uv run coverage html # HTML report → reports/coverage_html/ ``` -**Coverage Configuration (.coveragerc):** - -* Source: `src/aignostics` -* Omits: `*/tests/*`, `*/__init__.py`, `*/codegen/*` -* Reports: Terminal, XML (Codecov), HTML, Markdown - ### E2E Test Setup E2E tests require credentials to run against staging environment: @@ -949,51 +693,18 @@ uv run pytest -m "e2e and not long_running" -v ### Pytest Configuration Details -**From `pyproject.toml` `[tool.pytest.ini_options]`**: - -**Test Discovery**: - -* Test paths: `tests/` -* Python files: `*_test.py`, `test_*.py` -* Main file: `tests/main.py` - -**CLI Options** (always applied): - -```bash --p nicegui.testing.plugin # NiceGUI testing support --v # Verbose output ---strict-markers # Error on unknown markers ---log-disable=aignostics # Disable SDK logging during tests ---cov=aignostics # Coverage for src/aignostics ---cov-report=term-missing # Terminal report with missing lines ---cov-report=xml:reports/coverage.xml # XML for Codecov ---cov-report=html:reports/coverage_html # HTML report -``` - -**Timeouts**: - -* Default: **10 seconds** per test -* Override in test: `@pytest.mark.timeout(timeout=60)` -* Method: `signal` (can be configured) +Full config is `[tool.pytest.ini_options]` in `pyproject.toml`. Non-obvious +points worth knowing: -**Async Support**: - -* Mode: `auto` (automatically detects async tests) -* Default fixture loop scope: `function` - -**Coverage**: - -* Environment: `COVERAGE_FILE=.coverage`, `COVERAGE_PROCESS_START=pyproject.toml` -* Minimum: 85% (enforced in CI) -* Branch coverage: Enabled -* Parallel mode: Enabled (thread + multiprocessing concurrency) - -**Markdown Reports**: - -* Enabled: `md_report = true` -* Output: `reports/pytest.md` -* Flavor: GitHub-flavored markdown -* Exclude outcomes: `passed`, `skipped` (only show failures/errors) +* `--strict-markers` — an unregistered marker is an error, so add new markers to + the `markers` list in `pyproject.toml` first. +* `-p nicegui.testing.plugin` is always loaded (GUI test support). +* Coverage runs with subprocess collection via env vars `COVERAGE_FILE` and + `COVERAGE_PROCESS_START=pyproject.toml`; branch + parallel (thread + + multiprocessing) mode are on — so run `make test_coverage_reset` if `.coverage` + data gets stale/corrupted. +* Markdown report (`md_report`) is written to `reports/pytest.md` showing only + failures/errors. ### Test Fixtures and Patterns @@ -1267,14 +978,12 @@ uv add --group docs sphinx-rtd-theme **Optional dependency group:** ```bash -# Edit pyproject.toml -[project.optional-dependencies] -gui = ["nicegui>=1.0.0"] -qupath = ["ijson>=3.0.0"] - -# Install with extras -uv sync --extra gui -uv sync --all-extras # Install all optional groups +# Edit pyproject.toml [project.optional-dependencies] +# (current extras: pyinstaller, jupyter, marimo, qupath) + +# Install with a single extra, or all of them +uv sync --extra jupyter +uv sync --all-extras ``` ### Version Bumping and Releases @@ -1398,452 +1107,29 @@ gh pr edit --add-label "skip:test:long_running" * Set up ruff as external tool * Configure mypy plugin for type checking -## Tips and Tricks for Claude Code Efficiency - -### Quick Discovery Commands - -**Find files by pattern**: - -```bash -# Find all test files -find tests -name "*_test.py" -o -name "test_*.py" - -# Find Python files excluding tests -find src -name "*.py" | grep -v __pycache__ - -# Find configuration files -find . -maxdepth 2 -name "*.toml" -o -name "*.yml" -o -name "*.yaml" | grep -v node_modules -``` - -**Search code effectively**: - -```bash -# Find all imports of a module -grep -r "from aignostics.platform import" --include="*.py" - -# Find all test markers -grep -r "@pytest.mark." tests/ --include="*.py" | cut -d: -f2 | sort | uniq -c - -# Find all CLI commands -grep -r "@cli.*\.command" src/ --include="*.py" - -# Find TODOs and FIXMEs -grep -rn "TODO\|FIXME" src/ --include="*.py" -``` - -**Git exploration**: - -```bash -# View commit history for a specific file -git log --oneline --follow -- path/to/file.py - -# See what changed in recent commits -git log --oneline --stat -10 - -# Find who last modified a line -git blame -L 100,110 path/to/file.py - -# Check current branch and recent commits -git log --oneline --graph --decorate -20 -``` - -### Testing Shortcuts - -**Run specific test categories**: - -```bash -# Run only fast tests (unit + integration, no e2e) -uv run pytest -m "unit or integration" -v - -# Run tests for a specific module -uv run pytest tests/aignostics/platform/ -v - -# Run tests matching a pattern -uv run pytest -k "metadata" -v - -# Run last failed tests -uv run pytest --lf - -# Run tests that failed in last session, then continue with others -uv run pytest --ff -``` - -**Test discovery and validation**: - -```bash -# Collect tests without running (verify test discovery) -uv run pytest --collect-only - -# Find tests without category markers (CRITICAL - they won't run in CI!) -uv run pytest -m "not unit and not integration and not e2e" --collect-only - -# List all available markers -uv run pytest --markers - -# Dry run with verbose output -uv run pytest --collect-only -v | grep "` and `uv run pyright ` (exclusions in `pyrightconfig.json`). ### Performance Considerations -* Chunked uploads/downloads (1MB/10MB chunks) -* Streaming for large files -* Process management for subprocesses -* Memory-efficient WSI tile processing +* Chunked uploads/downloads, streaming for large files +* Memory-efficient WSI tile processing (process in tiles, not full image) ### Common Pitfalls -1. **Import errors**: Check optional dependencies -2. **Token expiry**: Force refresh with `remove_cached_token()` -3. **Large files**: Use streaming and chunking -4. **WSI memory**: Process in tiles, not full image -5. **Platform differences**: Check Windows path lengths +1. **Import errors**: run `uv sync --all-extras` +2. **Token expiry**: force refresh with `remove_cached_token()` +3. **WSI memory**: process in tiles, not full image +4. **Platform differences**: watch Windows path lengths --- diff --git a/README.md b/README.md index 72e674d8a..094718e23 100644 --- a/README.md +++ b/README.md @@ -225,7 +225,6 @@ The Aignostics Platform delivers enterprise-grade computational pathology throug - **NVIDIA® GPU clusters**: Dedicated compute provisioned per application run for maximum security and compliance ```mermaid -%%{init: {'theme':'dark', 'themeVariables': { 'fontSize':'18px', 'fontFamily':'arial', 'darkMode':'true', 'background':'#1e1e1e', 'primaryColor':'#4a4a4a', 'primaryTextColor':'#ffffff', 'primaryBorderColor':'#ffffff', 'lineColor':'#ffffff', 'secondaryColor':'#3a3a3a', 'tertiaryColor':'#2a2a2a', 'actorBkg':'#4a4a4a', 'actorBorder':'#ffffff', 'actorTextColor':'#ffffff', 'actorLineColor':'#ffffff', 'signalColor':'#ffffff', 'signalTextColor':'#ffffff', 'labelBoxBkgColor':'#3a3a3a', 'labelBoxBorderColor':'#ffffff', 'labelTextColor':'#ffffff', 'noteBkgColor':'#4a4a4a', 'noteTextColor':'#ffffff', 'noteBorderColor':'#ffffff', 'sequenceNumberColor':'#000000'}}}%% sequenceDiagram autonumber actor User as User
(Organization Member) diff --git a/docs/partials/README_platform.md b/docs/partials/README_platform.md index 18704d831..69809ec4d 100644 --- a/docs/partials/README_platform.md +++ b/docs/partials/README_platform.md @@ -118,7 +118,6 @@ The Aignostics Platform delivers enterprise-grade computational pathology throug - **NVIDIA® GPU clusters**: Dedicated compute provisioned per application run for maximum security and compliance ```mermaid -%%{init: {'theme':'dark', 'themeVariables': { 'fontSize':'18px', 'fontFamily':'arial', 'darkMode':'true', 'background':'#1e1e1e', 'primaryColor':'#4a4a4a', 'primaryTextColor':'#ffffff', 'primaryBorderColor':'#ffffff', 'lineColor':'#ffffff', 'secondaryColor':'#3a3a3a', 'tertiaryColor':'#2a2a2a', 'actorBkg':'#4a4a4a', 'actorBorder':'#ffffff', 'actorTextColor':'#ffffff', 'actorLineColor':'#ffffff', 'signalColor':'#ffffff', 'signalTextColor':'#ffffff', 'labelBoxBkgColor':'#3a3a3a', 'labelBoxBorderColor':'#ffffff', 'labelTextColor':'#ffffff', 'noteBkgColor':'#4a4a4a', 'noteTextColor':'#ffffff', 'noteBorderColor':'#ffffff', 'sequenceNumberColor':'#000000'}}}%% sequenceDiagram autonumber actor User as User
(Organization Member) diff --git a/pyproject.toml b/pyproject.toml index 3c442b4e3..9c20f8c49 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,8 +4,12 @@ version = "1.4.0" description = "🔬 Python SDK providing access to the Aignostics Platform. Includes Aignostics Launchpad (Desktop Application), Aignostics CLI (Command-Line Interface), example notebooks, and Aignostics Client Library." readme = "README.md" authors = [ - { name = "Helmut Hoffer von Ankershoffen", email = "helmut@aignostics.com" }, - { name = "Andreas Kunft", email = "andreas@aignostics.com" }, + { name = "Helmut Hoffer von Ankershoffen", email = "support@aignostics.com" }, + { name = "Andreas Kunft", email = "support@aignostics.com" }, +] +maintainers = [ + { name = "Oliver Meyer", email = "support@aignostics.com" }, + { name = "Arne Baumann", email = "support@aignostics.com" }, ] license = { file = "LICENSE" } diff --git a/src/aignostics/CLAUDE.md b/src/aignostics/CLAUDE.md index d14706c47..739c43204 100644 --- a/src/aignostics/CLAUDE.md +++ b/src/aignostics/CLAUDE.md @@ -17,6 +17,10 @@ This file provides a comprehensive overview of all modules in the Aignostics SDK | **qupath** | QuPath integration | ✅ | ✅ | ✅ | | **system** | System information | ✅ | ✅ | ✅ | +Helper packages without their own CLAUDE.md: `dicom/`, `idc/`, `marimo/`, +`thumbnail/`, `tiff/` — internal support used by the modules above (DICOM +handling, IDC access, Marimo notebook assets, thumbnailing, TIFF I/O). + ## Module Descriptions ### 🔐 platform @@ -26,11 +30,13 @@ This file provides a comprehensive overview of all modules in the Aignostics SDK - **Core Features**: - OAuth 2.0 authentication, JWT token management, API client wrapper - **SDK Metadata System**: Automatic tracking of execution context, user info, CI/CD environment - - JSON Schema validation for metadata with versioning (Run v0.0.4, Item v0.0.3) + - JSON Schema validation for metadata; schema versions live in + `platform/_sdk_metadata.py` (`SDK_METADATA_SCHEMA_VERSION` / + `ITEM_SDK_METADATA_SCHEMA_VERSION`) — not hardcoded here - Operation caching for non-mutating API calls - **CLI**: - `user login`, `user logout`, `user whoami` for authentication - - `sdk metadata-schema` for JSON Schema export + - `sdk run-metadata-schema`, `sdk item-metadata-schema` for JSON Schema export - **Dependencies**: `utils` (logging, user_agent generation) - **Used By**: All modules requiring API access; application module for automatic metadata attachment @@ -49,7 +55,7 @@ This file provides a comprehensive overview of all modules in the Aignostics SDK **Medical image file handling and processing** - **Core Features**: Format detection, thumbnail generation, metadata extraction -- **CLI**: `info`, `thumbnail` commands for WSI inspection +- **CLI**: `inspect`; `dicom inspect`, `dicom geojson_import` (see `wsi/_cli.py`) - **GUI**: Image viewer and metadata display - **Handlers**: OpenSlide (.svs, .tiff), PyDICOM (DICOM files) - **Dependencies**: `utils` (logging) @@ -92,9 +98,10 @@ This file provides a comprehensive overview of all modules in the Aignostics SDK **Desktop application launchpad** - **Core Features**: Module launcher, unified interface -- **GUI Only**: NiceGUI-based desktop interface +- **GUI Only**: NiceGUI-based desktop interface (nicegui is a core dependency; + there is no `[gui]` extra) - **Dependencies**: All modules with GUI components -- **Launch**: `uvx --with aignostics[gui] aignostics gui` +- **Launch**: `aignostics launchpad` ### 📓 notebook @@ -129,7 +136,7 @@ This file provides a comprehensive overview of all modules in the Aignostics SDK ```text ┌─────────────────────────────────────────────────────────────┐ -│ GUI Launchpad (_gui/) │ +│ GUI Launchpad (gui/) │ │ (Desktop Interface Aggregator) │ └─────────────────────────────────────────────────────────────┘ ↓ @@ -153,33 +160,8 @@ This pattern repeats for: platform, application, wsi, dataset, bucket, qupath, system (each module has CLI + Service, most have GUI) ``` -### Service Layer Dependencies - -```text - ┌──────────────────┐ - │ application │ - │ (service) │ - └──────┬───────────┘ - │ uses - ┌──────────┬───────┼────────┬──────────┐ - ↓ ↓ ↓ ↓ ↓ - ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ - │ wsi │ │ dataset │ │ bucket │ │ qupath │ - │(service)│ │(service)│ │(service)│ │(service)│ - └─────────┘ └─────────┘ └────┬────┘ └─────────┘ - │ - ↓ - ┌──────────────┐ - │ platform │ - │ (service) │ - └──────┬───────┘ - │ - ↓ - ┌──────────────┐ - │ utils │ - │(infrastructure)│ - └───────────────┘ -``` +The service dependency graph is captured textually under "Module +Communication → Direct Dependencies" below. ### Common Integration Patterns @@ -215,14 +197,31 @@ utils.locate_implementations(BaseService) ### Direct Dependencies -- **application** → `platform`, `bucket`, `wsi`, `utils`, `qupath` (optional) -- **dataset** → `platform`, `utils` -- **bucket** → `platform`, `utils` -- **wsi** → `utils` -- **gui** → All modules with GUI components -- **notebook** → `utils`, `marimo` (external) -- **qupath** → `utils` -- **system** → All modules (for health checks) +Verified from cross-module imports. `utils` and `constants` are foundation +(imported almost everywhere) and omitted from the graph for clarity. `dicom`, +`idc`, `thumbnail`, `tiff` are leaf modules that import only `utils`. + +```mermaid +graph TD + application --> platform + application --> bucket + application --> wsi + application --> qupath + application --> system + application --> gui + bucket --> platform + dataset --> platform + bucket --> gui + dataset --> gui + notebook --> gui + qupath --> gui + gui --> platform + gui --> system + system --> gui +``` + +`gui` and `system` are mutually coupled: `system` registers GUI pages through +`gui`, and the `gui` frame calls `SystemService.health_static()`. ### Shared Resources @@ -240,27 +239,17 @@ aignostics user login # List applications aignostics application list -# Submit a run -aignostics application run submit --application-id heta --files slide.svs +# Submit a run (positional: application_id, metadata CSV file) +aignostics application run submit heta metadata.csv -# Download dataset -aignostics dataset download --collection-id TCGA-LUAD --output-dir ./data +# Download an IDC dataset (positional: source, target) +aignostics dataset idc download # Get WSI info aignostics wsi inspect slide.svs -``` -## GUI Launch - -```bash -# Install with GUI support -pip install "aignostics[gui]" - -# Launch desktop interface -uvx --with "aignostics[gui]" aignostics gui - -# Or if installed locally -aignostics gui +# Launch the desktop interface +aignostics launchpad ``` ## Module-Specific Documentation @@ -280,53 +269,10 @@ For detailed information about each module, see: ## Development Guidelines -1. **New Module Checklist:** - - Inherit from `BaseService` for service discovery - - Implement `health()` and `info()` methods - - Add `_cli.py` for CLI commands using Typer - - Add `_gui.py` for GUI components using NiceGUI - - Create `CLAUDE.md` documentation - - Update this index - -2. **Service Pattern:** - - ```python - from aignostics.utils import BaseService - - - class Service(BaseService): - async def health(self) -> Health: - return Health(status=Health.Code.UP) - - async def info(self, mask_secrets=True) -> dict: - return {"version": "1.0.0"} - ``` - -3. **CLI Pattern:** - - ```python - import typer - - cli = typer.Typer(name="module", help="Module description") - - - @cli.command("action") - def action_command(): - """Action description.""" - service = Service() - service.perform_action() - ``` - -4. **GUI Pattern:** - - ```python - from nicegui import ui - - - def create_page(): - ui.label("Module Interface") - # Add components - ``` +New module checklist: inherit from `utils.BaseService` (for service discovery), +implement `health()` and `info()`, add a Typer `_cli.py`, add `_gui.py` (NiceGUI) +if it has a UI, create a `CLAUDE.md`, and update this index. Copy an existing +module (e.g. `system/`) as the pattern rather than scaffolding from scratch. --- diff --git a/src/aignostics/application/CLAUDE.md b/src/aignostics/application/CLAUDE.md index 3c214d76b..08f6c9c70 100644 --- a/src/aignostics/application/CLAUDE.md +++ b/src/aignostics/application/CLAUDE.md @@ -1,859 +1,112 @@ -# CLAUDE.md - Application Module - -This file provides comprehensive guidance to Claude Code and human engineers when working with the `application` module in this repository. - -## Module Overview - -The application module provides high-level orchestration for AI/ML applications on the Aignostics Platform, managing complex workflows for computational pathology analysis with enterprise-grade reliability and observability. - -### Core Responsibilities - -- **Workflow Orchestration**: End-to-end management of application runs from file upload to result retrieval -- **Version Management**: Semantic versioning validation using `semver` library -- **Progress Tracking**: Multi-stage progress monitoring with real-time updates and QuPath integration -- **File Processing**: WSI validation, chunked uploads, CRC32C integrity verification -- **State Management**: Complex state machines for run lifecycle with error recovery -- **SDK Metadata Integration**: Automatic attachment of SDK context metadata to all submitted runs -- **Integration Hub**: Bridges platform, WSI, bucket, and QuPath services seamlessly - -### User Interfaces - -**CLI Commands (`_cli.py`):** - -- `application list` - List available applications and versions -- `application dump-schemata` - Export input/output schemas -- `application run list` - List application runs -- `application run submit` - Submit new application run -- `application run describe` - Show run details and status -- `application run result download` - Download run results -- `application run result delete` - Delete run results - -**GUI Components (`_gui/`):** - -- `_page_index.py` - Main application listing and run submission -- `_page_application_describe.py` - Application details and version information -- `_page_application_run_describe.py` - Run monitoring with real-time progress -- QuPath integration for WSI visualization (when ijson installed) - -**Service Layer (`_service.py`):** - -Core application operations: - -- Application listing and version management (semver validation) -- Run lifecycle management (submit, monitor, complete) -- File upload with chunking (1MB chunks) and CRC32C verification -- Result download with progress tracking -- State machine for run status transitions -- QuPath project creation (when ijson available) - -## Architecture & Design Patterns - -### Health Check Gates - -The application module enforces system health checks before critical operations to prevent users from uploading data or submitting runs when the platform is unavailable. - -**CLI Health Check Enforcement (`_cli.py`):** - -The `_abort_if_system_unhealthy()` function is called before upload and submit operations: - -```python -def _abort_if_system_unhealthy() -> None: - """Check system health and abort if unhealthy.""" - health = SystemService.health_static() - if not health: - logger.error(f"Platform is not healthy: {health.reason}. Aborting.") - console.print(f"[error]Error:[/error] Platform is not healthy: {health.reason}. Aborting.") - sys.exit(1) -``` - -**Commands with Health Check Gates:** - -| Command | Health Check | Override | -|---------|--------------|----------| -| `run execute` | Yes | `--force` | -| `run upload` | Yes | `--force` | -| `run submit` | Yes | `--force` | -| `run prepare` | No | N/A | -| `run list` | No | N/A | -| `run describe` | No | N/A | -| `run result download` | No | N/A | - -**GUI Health Check Enforcement (`_gui/_page_application_describe.py`):** - -The stepper workflow checks health at the application version selection step: - -```python -# Check system health before allowing progression -system_healthy = bool(SystemService.health_static()) - -if not system_healthy: - version_next_button.disable() - ui.tooltip("System is unhealthy, you cannot prepare a run at this time.") - - # Internal users (Aignostics, pre-alpha-org, LMU, Charite) can force-skip - if is_internal_user: - ui.checkbox("Force (skip health check)", on_change=on_force_change) -``` - -**Force Option:** - -The `submit_form.force` attribute tracks whether the user has opted to skip health checks. This is only available to internal organization users. - -### Module Structure (NEW in v1.0.0-beta.7) - -The application module is organized into focused submodules: - -``` -application/ -├── _service.py # High-level orchestration and API integration -├── _models.py # Data models (DownloadProgress, DownloadProgressState) [NEW] -├── _download.py # Download helpers with progress tracking [NEW] -├── _utils.py # Shared utilities -├── _cli.py # CLI commands -└── _gui/ # GUI components -``` - -**Key Separation:** - -- **_models.py**: Pydantic models for progress tracking with computed fields -- **_download.py**: Pure download logic (URLs, artifacts, progress callbacks) -- **_service.py**: High-level business logic and module integration - -### Service Layer Architecture - -``` -┌────────────────────────────────────────────┐ -│ Application Service │ -│ (High-Level Orchestration) │ -├────────────────────────────────────────────┤ -│ Progress Tracking & State Management │ -│ (_models.py - NEW) │ -├────────────────────────────────────────────┤ -│ Integration Layer │ -│ ┌──────────┬───────────┬──────────┐ │ -│ │ Platform │ WSI │ QuPath │ │ -│ │ Service │ Service │ Service │ │ -│ └──────────┴───────────┴──────────┘ │ -├────────────────────────────────────────────┤ -│ File Processing Layer │ -│ (Upload, Download, Verification) │ -│ (_download.py - NEW) │ -└────────────────────────────────────────────┘ -``` - -### State Machine Design - -```python -RunState: - QUEUED → RUNNING → COMPLETED - ↓ - FAILED / CANCELLED - -ItemState: - PENDING → PROCESSING → COMPLETED - ↓ - FAILED -``` - -### Progress Tracking Architecture - -```python -DownloadProgress: - ├── Status (State Machine) - ├── Run Metadata - ├── Item Progress (0..1) - ├── Artifact Progress (0..1) - └── QuPath Integration Progress - ├── Add Input Progress - ├── Add Results Progress - └── Annotate Progress -``` - -## Critical Implementation Details - -### Version Management (`_service.py`) - -**Actual Semantic Version Validation:** - -```python -def application_version(self, application_id: str, version_number: str | None = None) -> ApplicationVersion: - """Validate and retrieve application version. - - Args: - application_id: The ID of the application (e.g., 'heta') - version_number: The semantic version number (e.g., '1.0.0') - If None, returns the latest version - - Returns: - ApplicationVersion with application_id and version_number attributes - """ - # Delegates to platform client which validates semver format - # Platform client uses Versions resource internally - return self.platform_client.application_version(application_id=application_id, version_number=version_number) -``` - -**Key Points:** - -- Application ID and version number are now **separate parameters** -- Version format: semantic version string without 'v' prefix (e.g., `"1.0.0"`, not `"v1.0.0"`) -- Uses `semver.Version.is_valid()` for validation in the platform layer -- Falls back to latest version if `version_number` is `None` -- Returns `ApplicationVersion` object with `application_id` and `version_number` attributes - -### File Processing Constants (Actual Values) - -```python -# From _service.py -APPLICATION_RUN_FILE_READ_CHUNK_SIZE = 1024 * 1024 * 1024 # 1GB -APPLICATION_RUN_DOWNLOAD_CHUNK_SIZE = 1024 * 1024 # 1MB -APPLICATION_RUN_UPLOAD_CHUNK_SIZE = 1024 * 1024 # 1MB -APPLICATION_RUN_DOWNLOAD_SLEEP_SECONDS = 5 # Wait between status checks -``` - -### Progress State Management - -**Actual DownloadProgress Model (`_models.py`):** - -```python -class DownloadProgress(BaseModel): - """Model for tracking download progress with computed progress metrics.""" - - # Core state - status: DownloadProgressState = DownloadProgressState.INITIALIZING - - # Run and item tracking - run: RunData | None = None - item: ItemResult | None = None - item_count: int | None = None - item_index: int | None = None - item_external_id: str | None = None - - # Artifact tracking - artifact: OutputArtifactElement | None = None - artifact_count: int | None = None - artifact_index: int | None = None - artifact_path: Path | None = None - artifact_download_url: str | None = None - artifact_size: int | None = None - artifact_downloaded_chunk_size: int = 0 # Last chunk size - artifact_downloaded_size: int = 0 # Total downloaded - - # Input slide tracking (NEW in v1.0.0-beta.7) - input_slide_path: Path | None = None - input_slide_url: str | None = None - input_slide_size: int | None = None - input_slide_downloaded_chunk_size: int = 0 - input_slide_downloaded_size: int = 0 - - # QuPath integration (conditional) - if has_qupath_extra: - qupath_add_input_progress: QuPathAddProgress | None = None - qupath_add_results_progress: QuPathAddProgress | None = None - qupath_annotate_input_with_results_progress: QuPathAnnotateProgress | None = None - - @computed_field - @property - def total_artifact_count(self) -> int | None: - """Calculate total number of artifacts across all items.""" - if self.item_count and self.artifact_count: - return self.item_count * self.artifact_count - return None - - @computed_field - @property - def total_artifact_index(self) -> int | None: - """Calculate the current artifact index across all items.""" - if self.item_count and self.artifact_count and self.item_index is not None and self.artifact_index is not None: - return self.item_index * self.artifact_count + self.artifact_index - return None - - @computed_field - @property - def item_progress_normalized(self) -> float: - """Normalized progress 0..1 across all items. - - Handles different progress states: - - DOWNLOADING_INPUT: Progress through items being downloaded - - DOWNLOADING: Progress through artifacts being downloaded - - QUPATH_*: QuPath-specific progress tracking - """ - # Implementation varies by state... - - @computed_field - @property - def artifact_progress_normalized(self) -> float: - """Normalized progress 0..1 for current artifact/input download. - - Handles different download types: - - DOWNLOADING_INPUT: Input slide download progress - - DOWNLOADING: Artifact download progress - - QUPATH_ANNOTATE: QuPath annotation progress - """ - # Implementation varies by state... -``` - -### QuPath Integration (Conditional Loading) - -**Actual Implementation:** - -```python -# At module level -has_qupath_extra = find_spec("ijson") -if has_qupath_extra: - from aignostics.qupath import ( - AddProgress as QuPathAddProgress, - AnnotateProgress as QuPathAnnotateProgress, - Service as QuPathService - ) - -# In methods -def process_with_qupath(self, ...): - if not has_qupath_extra: - logger.warning("QuPath integration not available (ijson not installed)") - return - # QuPath processing... -``` - -**Download Progress States (`_models.py`):** - -```python -class DownloadProgressState(StrEnum): - """Enum for download progress states.""" - - INITIALIZING = "Initializing ..." - DOWNLOADING_INPUT = "Downloading input slide ..." # NEW in v1.0.0-beta.7 - QUPATH_ADD_INPUT = "Adding input slides to QuPath project ..." - CHECKING = "Checking run status ..." - WAITING = "Waiting for item completing ..." - DOWNLOADING = "Downloading artifact ..." - QUPATH_ADD_RESULTS = "Adding result images to QuPath project ..." - QUPATH_ANNOTATE_INPUT_WITH_RESULTS = "Annotating input slides in QuPath project with results ..." - COMPLETED = "Completed." -``` - -### Download Module (`_download.py` - NEW in v1.0.0-beta.7) - -The download module provides reusable download helper functions with comprehensive progress tracking. - -**Key Functions:** - -```python -def extract_filename_from_url(url: str) -> str: - """Extract a filename from a URL robustly. - - Supports: - - gs:// (Google Cloud Storage) - - http:// and https:// URLs - - Handles query parameters and trailing slashes - - Sanitizes filenames for filesystem use - - Examples: - >>> extract_filename_from_url("gs://bucket/path/to/file.tiff") - 'file.tiff' - >>> extract_filename_from_url("https://example.com/slides/sample.svs?token=abc") - 'sample.svs' - """ - - -def download_url_to_file_with_progress( - progress: DownloadProgress, - url: str, - destination_path: Path, - download_progress_queue: Any | None = None, - download_progress_callable: Callable | None = None, -) -> Path: - """Download a file from a URL (gs://, http://, or https://) with progress tracking. - - Features: - - Converts gs:// URLs to signed URLs automatically - - Streams downloads with 1MB chunks (APPLICATION_RUN_DOWNLOAD_CHUNK_SIZE) - - Updates progress on every chunk - - Supports both queue and callback progress updates - - Creates parent directories automatically - - Args: - progress: Progress tracking object (updated in place) - url: URL to download (gs://, http://, https://) - destination_path: Local file path to save to - download_progress_queue: Optional queue for GUI progress updates - download_progress_callable: Optional callback for CLI progress updates - - Returns: - Path: The path to the downloaded file - - Raises: - ValueError: If URL scheme is unsupported - RuntimeError: If download fails - """ - - -def download_available_items( - progress: DownloadProgress, - application_run: Run, - destination_directory: Path, - downloaded_items: set[str], - create_subdirectory_per_item: bool = False, - download_progress_queue: Any | None = None, - download_progress_callable: Callable | None = None, -) -> None: - """Download items that are available and not yet downloaded. - - Features: - - Only downloads TERMINATED items with FULL output - - Skips already downloaded items (tracked via external_id) - - Optional subdirectory per item - - Progress tracking for each item and artifact - - Args: - progress: Progress tracking object - application_run: Run object with results - destination_directory: Directory to save files - downloaded_items: Set of already downloaded external_ids - create_subdirectory_per_item: Create item subdirectories - download_progress_queue: Optional queue for GUI updates - download_progress_callable: Optional callback for CLI updates - """ - - -def download_item_artifact( - progress: DownloadProgress, - artifact: Any, - destination_directory: Path, - prefix: str = "", - download_progress_queue: Any | None = None, - download_progress_callable: Callable | None = None, -) -> None: - """Download an artifact of a result item with progress tracking. - - Features: - - CRC32C checksum verification - - Skips download if file exists with correct checksum - - Automatic file extension detection - - Chunked downloads with progress updates - - Raises: - ValueError: If no checksum metadata found or checksum mismatch - requests.HTTPError: If download fails - """ -``` - -**Constants:** - -```python -# From _download.py -APPLICATION_RUN_FILE_READ_CHUNK_SIZE = 1024 * 1024 * 1024 # 1GB (for checksum calculation) -APPLICATION_RUN_DOWNLOAD_CHUNK_SIZE = 1024 * 1024 # 1MB (for streaming downloads) -``` - -**URL Support:** - -The download module supports three URL schemes: - -1. **gs://** - Google Cloud Storage (converted to signed URLs via `platform.generate_signed_url()`) -2. **http://** - HTTP URLs (used directly) -3. **https://** - HTTPS URLs (used directly) - -**Progress Update Pattern:** - -```python -def update_progress( - progress: DownloadProgress, - download_progress_callable: Callable | None = None, - download_progress_queue: Any | None = None, -) -> None: - """Update download progress via callback or queue. - - Dual update mechanism: - - Callback: Synchronous update (CLI, blocking) - - Queue: Asynchronous update (GUI, non-blocking) - - Both can be used simultaneously. - """ - if download_progress_callable: - download_progress_callable(progress) - if download_progress_queue: - download_progress_queue.put_nowait(progress) -``` - -## Usage Patterns & Best Practices - -### Basic Application Execution - -```python -from aignostics.application import Service - -service = Service() - -# List applications -apps = service.list_applications() - -# Get specific version (actual pattern) -try: - # Application ID and version are separate parameters - app_version = service.application_version( - application_id="heta", - version_number="2.1.0", # Semantic version without 'v' prefix - ) - # Access attributes - print(f"Application: {app_version.application_id}") - print(f"Version: {app_version.version_number}") - - # Get latest version - latest = service.application_version( - application_id="heta", - version_number=None, # Returns latest version - ) -except ValueError as e: - # Handle invalid version format - logger.error(f"Version error: {e}") -except NotFoundException as e: - # Handle missing application or version - logger.error(f"Application not found: {e}") - -# Run application (simplified - actual has more parameters) -run = service.run_application( - application_id="heta", - application_version="2.1.0", # Optional, uses latest if omitted - files=["slide1.svs", "slide2.tiff"], -) -``` - -### File Upload Pattern (Actual Implementation) - -```python -def upload_file(self, file_path: Path, signed_url: str): - """Upload file with chunking and CRC32C.""" - - with file_path.open("rb") as f: - # Calculate CRC32C - crc = crc32c.CRC32CHash() - - # Upload in chunks - while True: - chunk = f.read(APPLICATION_RUN_UPLOAD_CHUNK_SIZE) # 1MB chunks - if not chunk: - break - - crc.update(chunk) - # Upload chunk to signed URL - # (Implementation details vary) - - # Return CRC32C for verification - return base64.b64encode(crc.digest()).decode("utf-8") -``` - -### Download with Progress (Actual Pattern) - -**Basic download with progress callback:** - -```python -from aignostics.application._download import download_url_to_file_with_progress -from aignostics.application._models import DownloadProgress -from pathlib import Path - -# Create progress object -progress = DownloadProgress() - - -# Define progress callback -def on_progress(p: DownloadProgress): - if p.input_slide_size: - percent = (p.input_slide_downloaded_size / p.input_slide_size) * 100 - print(f"Downloaded: {percent:.1f}%") - - -# Download from gs://, http://, or https:// -downloaded_file = download_url_to_file_with_progress( - progress=progress, - url="gs://my-bucket/slides/sample.svs", - destination_path=Path("./downloads/sample.svs"), - download_progress_callable=on_progress, -) - -print(f"Downloaded to: {downloaded_file}") -``` - -**Download with GUI queue (non-blocking):** - -```python -from queue import Queue -from aignostics.application._download import download_url_to_file_with_progress - -# Create queue for GUI updates -progress_queue = Queue() - -# Download in background thread -download_url_to_file_with_progress( - progress=DownloadProgress(), - url="https://example.com/slide.tiff", - destination_path=Path("./slide.tiff"), - download_progress_queue=progress_queue, # Non-blocking updates -) - -# In GUI thread, poll queue -while not progress_queue.empty(): - progress = progress_queue.get() - ui.update(progress.artifact_progress_normalized) -``` - -**Download application run results:** - -```python -from aignostics.application._download import download_available_items - -# Track downloaded items to avoid re-downloading -downloaded_items = set() - -# Download all available items -download_available_items( - progress=DownloadProgress(), - application_run=run, - destination_directory=Path("./results"), - downloaded_items=downloaded_items, - create_subdirectory_per_item=True, # Create dirs per item - download_progress_callable=lambda p: print(f"Item {p.item_index}/{p.item_count}"), -) -``` - -## Testing Strategies (Actual Test Patterns) - -### Semver Validation Testing (`service_test.py`) - -```python -def test_application_version_valid_semver_formats(): - """Test valid semver formats.""" - valid_versions = [ - "1.0.0", - "1.2.3", - "10.20.30", - "1.1.2-prerelease+meta", - "1.0.0-alpha", - "1.0.0-beta", - "1.0.0-alpha.beta", - "1.0.0-rc.1+meta", - ] - - for version in valid_versions: - try: - result = service.application_version(application_id="test-app", version_number=version) - assert result.application_id == "test-app" - assert result.version_number == version - except ValueError as e: - pytest.fail(f"Valid format '{version}' rejected: {e}") - except NotFoundException: - # Application doesn't exist, but format is valid - pytest.skip(f"Application not found for test-app") - - -def test_application_version_invalid_semver_formats(): - """Test invalid formats are rejected.""" - invalid_versions = [ - "v1.0.0", # 'v' prefix not allowed - "1.0", # Incomplete version - "1.0.0-", # Trailing dash - "", # Empty string - "not-semver", # Not a valid semver - ] - - for version in invalid_versions: - with pytest.raises(ValueError, match="Invalid version format"): - service.application_version(application_id="test-app", version_number=version) -``` - -### Use Latest Fallback Test - -```python -def test_application_version_use_latest_fallback(): - """Test fallback to latest version.""" - service = ApplicationService() - - try: - # Get latest version by passing None - result = service.application_version( - application_id=HETA_APPLICATION_ID, - version_number=None, # Falls back to latest - ) - assert result is not None - assert result.application_id == HETA_APPLICATION_ID - assert result.version_number is not None - # version_number should be valid semver - assert semver.Version.is_valid(result.version_number) - except NotFoundException as e: - if "No versions found" in str(e): - # Expected if no versions exist - pytest.skip(f"No versions available for {HETA_APPLICATION_ID}") - else: - pytest.fail(f"Unexpected error: {e}") -``` - -## Operational Requirements - -### File Processing Limits - -- **Upload chunk size**: 1 MB -- **Download chunk size**: 1 MB -- **File read chunk size**: 1 GB (for large file processing) -- **Status check interval**: 5 seconds - -### Monitoring & Observability - -**Key Metrics:** - -- Run completion rate by application -- Average processing time per WSI file -- Upload/download throughput (MB/s) -- Progress callback frequency -- QuPath integration availability - -**Logging Patterns (Actual):** - -```python -logger.debug("Starting application run", extra={"application_id": app_id, "file_count": len(files)}) - -logger.warning("QuPath integration not available (ijson not installed)") - -logger.error("Application version validation failed", extra={"version_id": version_id, "error": str(e)}) -``` - -## Common Pitfalls & Solutions - -### Semver Format Issues - -**Problem:** Using incorrect version format or combining application ID with version - -**Solution:** - -```python -# Correct: Separate application_id and version_number -app_version = service.application_version( - application_id="heta", - version_number="1.2.3", # No 'v' prefix -) - -# Wrong: Old combined format -# app_version = service.application_version("heta:v1.2.3") # No longer supported - -# Wrong: Version with 'v' prefix -# version_number="v1.2.3" # Will fail validation -``` - -### QuPath Availability - -**Problem:** QuPath features not working - -**Solution:** - -```python -# Check if ijson is installed -if not has_qupath_extra: - print("QuPath features require: pip install ijson") -``` - -### Large File Processing - -**Problem:** Memory issues with large files - -**Solution:** - -```python -# Use streaming with appropriate chunk size -chunk_size = APPLICATION_RUN_FILE_READ_CHUNK_SIZE # 1GB -with open(file_path, "rb") as f: - while chunk := f.read(chunk_size): - process_chunk(chunk) -``` - -## Module Dependencies - -### Internal Module Organization (NEW in v1.0.0-beta.7) - -The application module is organized into focused submodules: - -- **`_service.py`** - High-level orchestration, API integration, run lifecycle management -- **`_models.py`** - Data models (DownloadProgress, DownloadProgressState) -- **`_download.py`** - Download helpers with progress tracking and checksum verification -- **`_utils.py`** - Shared utilities -- **`_cli.py`** - CLI commands -- **`_gui/`** - GUI components (page builders, reactive components) - -### Internal SDK Dependencies - -- `platform` - Client, API operations, **SDK metadata system** (automatic attachment to all runs), signed URLs -- `wsi` - WSI file validation -- `bucket` - Cloud storage operations -- `qupath` - Analysis integration (optional, requires ijson) -- `utils` - Logging, sanitization, and base utilities - -**SDK Metadata Integration:** - -Every run submitted through the application module automatically includes SDK metadata: - -- **Run metadata** (v0.0.4): Execution context, user info, CI/CD details, tags, timestamps -- **Item metadata** (v0.0.3): Platform bucket location, tags, timestamps -- Automatic attachment via `platform._sdk_metadata.build_run_sdk_metadata()` - -See `platform/CLAUDE.md` for detailed SDK metadata documentation and schema. - -**Custom Metadata Updates: Checksum Concurrency Control & `enrich_sdk_metadata` Toggle (NEW):** - -`Service.application_run_update_custom_metadata()` / `..._update_item_custom_metadata()` (and their -`_static` wrappers) accept two keyword-only parameters, threaded through to the platform layer: - -- `custom_metadata_checksum: str | None` — pass the checksum from a prior read - (`RunData.custom_metadata_checksum` / `ItemResultReadResponse.custom_metadata_checksum`) for - optimistic concurrency control. If the metadata changed since the checksum was read, the platform - returns HTTP 412 and the service raises `aignostics.platform.ConcurrencyConflictError` (a - `ValueError` subclass) instead of silently overwriting a concurrent change. -- `enrich_sdk_metadata: bool = True` — set to `False` to forward `custom_metadata` (including any - `sdk` field) to the platform verbatim, skipping the automatic SDK metadata merge/validation. Use - this when round-tripping a previously dumped `sdk` field unchanged. - -CLI support (`_cli.py`): +# application — application run orchestration + +High-level orchestration for AI/ML application runs on the Aignostics Platform: +listing applications/versions, submitting runs, monitoring, and downloading +results (with optional QuPath integration). Bridges `platform`, `wsi`, `bucket`, +and `qupath`. See `../CLAUDE.md` for where this sits in the whole SDK. + +## Public API (`_service.py`) + +`Service` (aliased `ApplicationService`) methods — see the file for full signatures: + +- `applications()` / `applications_static()` — list available applications. +- `application_version(application_id, application_version=None)` — retrieve one + version; `application_version=None` returns the latest. Returns an + `ApplicationVersion` whose attributes are `.application_id` and `.version_number`. + The version string is validated as semver (no `v` prefix); an invalid string + raises `ValueError` ("not compliant with semantic versioning"). +- `application_versions(application_id)` / `application_versions_static(...)`. +- `application_run_submit(...)` / `application_run_submit_from_metadata(...)` — + submit a run. +- `application_run_update_custom_metadata(...)` / `..._update_item_custom_metadata(...)` + (+ `_static` variants) — see "Custom metadata" below. + +Note the parameter name asymmetry: the request keyword is `application_version`, +while the returned object exposes it as `.version_number`. + +## Models (`_models.py`) + +`DownloadProgress` (Pydantic) tracks run/item/artifact download state plus +optional QuPath progress, with computed fields `total_artifact_count`, +`total_artifact_index`, `item_progress_normalized`, and +`artifact_progress_normalized` (0..1). `DownloadProgressState` (StrEnum) holds the +human-readable stage labels. QuPath fields are only present when `ijson` is +installed (`has_qupath_extra = find_spec("ijson")`). See `_models.py` for fields. + +## Download helpers (`_download.py`) + +- `extract_filename_from_url(url)` — filename from `gs://`/`http(s)://`. +- `download_url_to_file_with_progress(progress, url, destination_path, ...)` — + streams in 1 MB chunks; converts `gs://` to a signed URL via + `platform.generate_signed_url()`. +- `download_available_items(progress, application_run, destination_directory, + downloaded_items, ...)` — only TERMINATED items with FULL output; skips items + already in `downloaded_items` (by external_id). +- `download_item_artifact(progress, run, artifact, destination_directory, prefix="", ...)` — + resolves a fresh presigned URL from `run`, verifies CRC32C, skips re-download + when the local file already matches the checksum. + +Constants (defined here): `APPLICATION_RUN_FILE_READ_CHUNK_SIZE` = 1 GB (checksum +reads), `APPLICATION_RUN_DOWNLOAD_CHUNK_SIZE` = 1 MB (streaming). + +Progress is reported via a dual mechanism: a synchronous `download_progress_callable` +(CLI) and/or an async `download_progress_queue` (GUI); both may be set. + +## CLI (`_cli.py`) + +Top level: `list`, `dump-schemata`, `describe`. + +`application run`: `execute`, `prepare`, `upload`, `submit`, `list`, `describe`, +`cancel`, `cancel-by-filter`, `dump-metadata`, `dump-item-metadata`, +`update-metadata`, `update-item-metadata`. + +`application run result`: `download`, `delete`. +`application run share`: `status`; `share organization`: `list`/`grant`/`revoke`; +`share token`: `list`/`create`/`revoke`. +`application version`: version inspection; `version document`: `list`/`describe`/`download`. + +## Behaviour worth knowing + +**Health gating.** `execute`, `upload`, and `submit` call `_abort_if_system_unhealthy()` +first (`--force` overrides); it runs `asyncio.run(SystemService.health_static())` +and exits 1 when unhealthy. The GUI stepper disables progression on an unhealthy +system and offers a force-skip checkbox to internal orgs. Health enforcement +details are owned by `../system/CLAUDE.md`. + +**Custom metadata (optimistic concurrency).** `run dump-metadata` / +`dump-item-metadata` take `--show-checksum` (wraps output as +`{"custom_metadata": ..., "custom_metadata_checksum": ...}`). `run update-metadata` / +`update-item-metadata` take `--checksum` and `--enrich-sdk-metadata` / +`--no-enrich-sdk-metadata`. Passing a stale `--checksum` makes the platform return +HTTP 412; the service raises `platform.ConcurrencyConflictError` (a `ValueError` +subclass) and the CLI exits with code 3 (distinct from code 2 for not-found / +invalid-ID). Read-modify-write example: ```bash -# Read-modify-write loop, safe under concurrent edits: -aignostics application run custom-metadata dump-metadata RUN_ID --show-checksum --pretty -# ... edit the dumped JSON ... -aignostics application run custom-metadata update-metadata RUN_ID "" \ +aignostics application run dump-metadata RUN_ID --show-checksum --pretty +# ... edit the dumped JSON ... +aignostics application run update-metadata RUN_ID "" \ --checksum --no-enrich-sdk-metadata ``` -- `dump-metadata` / `dump-item-metadata` gained `--show-checksum`, wrapping the output as - `{"custom_metadata": ..., "custom_metadata_checksum": ...}`. -- `update-metadata` / `update-item-metadata` gained `--checksum` and - `--enrich-sdk-metadata`/`--no-enrich-sdk-metadata`. A `ConcurrencyConflictError` from the service - exits the CLI with code 3 (distinct from the existing code 2 for not-found/invalid-ID errors). - -See `platform/CLAUDE.md` for the full checksum + `enrich_sdk_metadata` semantics. - -**Signed URL Generation:** - -The `_download.py` module uses `platform.generate_signed_url()` to convert `gs://` URLs to time-limited signed URLs for downloads. - -### External Dependencies - -- `semver` - Semantic version validation (using `Version.is_valid()`) -- `crc32c` - File integrity checking (CRC32C checksums) -- `requests` - HTTP operations (streaming downloads) -- `pydantic` - Data models with validation and computed fields -- `ijson` - Required for QuPath features (optional) - -## Performance Notes - -### Current Implementation Details +`enrich_sdk_metadata=False` forwards `custom_metadata` (including any `sdk` field) +verbatim, skipping the SDK-metadata merge/validation — use it when round-tripping +a previously dumped `sdk` field. See `../platform/CLAUDE.md` for full semantics. -1. **Chunk sizes are fixed** (not adaptive) -2. **Single-threaded uploads/downloads** (no parallelization) -3. **Synchronous progress callbacks** (may impact performance) -4. **No connection pooling** configured explicitly +**SDK metadata.** Every submitted run gets SDK metadata attached automatically via +`platform._sdk_metadata.build_run_sdk_metadata()`. Schema versions and details live +in `../platform/CLAUDE.md` / `platform/_sdk_metadata.py` — not duplicated here. -### Optimization Opportunities +## Dependencies & gotchas -1. Implement adaptive chunk sizing based on bandwidth -2. Add parallel upload/download for multiple files -3. Use async progress callbacks to avoid blocking -4. Configure connection pooling for better throughput +Internal: `platform` (API, signed URLs, SDK metadata), `wsi` (validation), +`bucket` (storage), `utils` (DI/logging), and optional `qupath` (gated on `ijson`). +External deps and extras are declared in `pyproject.toml`; QuPath features require +the `qupath` extra (`ijson`). ---- +## Tests -*This documentation reflects the actual implementation verified against the codebase.* +`tests/aignostics/application/service_test.py` covers semver validation (valid/ +invalid formats, latest-version fallback) and run lifecycle. Match the existing +style there (positional args, `pytest.raises(..., match=r"not compliant with +semantic versioning")`) when adding cases. diff --git a/src/aignostics/bucket/CLAUDE.md b/src/aignostics/bucket/CLAUDE.md index af3ff2c17..ffbc0ec98 100644 --- a/src/aignostics/bucket/CLAUDE.md +++ b/src/aignostics/bucket/CLAUDE.md @@ -1,460 +1,42 @@ -# CLAUDE.md - Bucket Module - -This file provides comprehensive guidance to Claude Code and human engineers when working with the `bucket` module in this repository. - -## Module Overview - -The bucket module provides enterprise-grade cloud storage operations for the Aignostics Platform, abstracting AWS S3 and Google Cloud Storage with unified interfaces for secure file management in medical imaging workflows. - -### Core Responsibilities - -- **Cloud Storage Abstraction**: Unified interface for S3, GCS, and Azure Blob Storage -- **Secure File Transfer**: Signed URL generation with expiry and access control -- **Large File Handling**: Multipart uploads, chunked downloads, resume capability -- **Data Integrity**: CRC32C/MD5 checksums, automatic retry on corruption -- **Compliance**: HIPAA-compliant storage patterns, audit logging, encryption - -### User Interfaces - -**CLI Commands (`_cli.py`):** - -- `bucket upload` - Upload files or directories to cloud storage -- `bucket download` - Download files from cloud storage -- `bucket list` - List bucket contents -- `bucket delete` - Delete files from bucket -- `bucket sign` - Generate signed URLs - -**GUI Component (`_gui.py`):** - -- Storage browser interface -- Upload/download manager with progress -- Signed URL generator - -**Service Layer (`_service.py`):** - -Core storage operations: - -- S3/GCS client management -- Signed URL generation with security constraints -- Chunked upload/download (1MB upload, 10MB download chunks) -- ETag calculation and verification -- Progress tracking with callbacks - -## Architecture & Design Patterns - -### Service Layer Design - -``` -┌────────────────────────────────────────────┐ -│ Bucket Service │ -│ (High-Level Operations) │ -├────────────────────────────────────────────┤ -│ Storage Abstraction │ -│ ┌─────────┬─────────┬─────────┐ │ -│ │ S3 │ GCS │ Azure │ │ -│ └─────────┴─────────┴─────────┘ │ -├────────────────────────────────────────────┤ -│ Transfer Management Layer │ -│ (Multipart, Chunking, Resumption) │ -├────────────────────────────────────────────┤ -│ Security & Compliance │ -│ (Encryption, Signing, Audit) │ -└────────────────────────────────────────────┘ -``` - -### Storage Patterns - -**Hierarchical Organization:** - -``` -bucket/ -├── organizations/{org_id}/ -│ ├── applications/{app_id}/ -│ │ ├── runs/{run_id}/ -│ │ │ ├── inputs/ -│ │ │ ├── outputs/ -│ │ │ └── metadata.json -│ │ └── versions/ -│ └── datasets/ -└── temp/ # Temporary uploads with TTL -``` - -## Critical Implementation Details - -### Signed URL Generation (`_service.py`) - -**Security-First URL Generation:** - -```python -def generate_signed_url( - bucket: str, - key: str, - operation: str = "GET", - expiry_seconds: int = 3600, - content_type: str = None, - metadata: dict = None, -) -> str: - """Generate time-limited signed URL with security constraints.""" - - # Validate permissions - if not has_permission(bucket, key, operation): - raise PermissionError(f"No {operation} access to {key}") - - # Add security headers - params = { - "Bucket": bucket, - "Key": key, - "ResponseContentDisposition": f"attachment; filename={Path(key).name}", - "ResponseContentType": content_type or "application/octet-stream", - } - - # Add server-side encryption - if operation == "PUT": - params["ServerSideEncryption"] = "AES256" - params["ServerSideEncryptionCustomerAlgorithm"] = "AES256" - - # Generate presigned URL - url = s3_client.generate_presigned_url( - ClientMethod=operation.lower() + "_object", Params=params, ExpiresIn=expiry_seconds - ) - - # Audit log - audit_logger.debug( - f"Generated signed URL", - extra={"operation": operation, "bucket": bucket, "key": key, "expiry": expiry_seconds, "user": current_user.id}, - ) - - return url -``` - -### Upload and Download Management - -**Chunk Size Constants (Actual):** - -```python -UPLOAD_CHUNK_SIZE = 1024 * 1024 # 1MB upload chunks -DOWNLOAD_CHUNK_SIZE = 1024 * 1024 * 10 # 10MB download chunks -ETAG_CHUNK_SIZE = 1024 * 1024 * 100 # 100MB for ETag calculation - - -def upload_file(file_path: Path, bucket: str, key: str, progress_callback: Callable = None) -> str: - """Upload file with chunking (actual implementation pattern).""" - - # Generate signed URL for upload - url = self._get_s3_client().generate_presigned_url( - ClientMethod="put_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=3600 - ) - - # Upload with chunking - with file_path.open("rb") as f: - uploaded = 0 - while True: - chunk = f.read(UPLOAD_CHUNK_SIZE) # 1MB chunks - if not chunk: - break - - # Upload chunk logic here - uploaded += len(chunk) - if progress_callback: - progress_callback(uploaded, file_path.stat().st_size) - - return url -``` - -### Download with Stream Processing - -**Memory-Efficient Download:** - -```python -def download_file(url: str, output_path: Path, progress_callback: Callable = None) -> None: - """Download file with streaming (actual implementation pattern).""" - - response = requests.get(url, stream=True) - total_size = int(response.headers.get("Content-Length", 0)) - downloaded = 0 - - with output_path.open("wb") as f: - for chunk in response.iter_content(chunk_size=DOWNLOAD_CHUNK_SIZE): # 10MB chunks - if chunk: - f.write(chunk) - downloaded += len(chunk) - - if progress_callback: - progress = DownloadProgress(current_file_downloaded=downloaded, current_file_size=total_size) - progress_callback(progress) -``` - -## Cross-Module Integration - -### Platform Module Integration - -The bucket module integrates tightly with the [platform module](../platform/CLAUDE.md): - -- Uses platform authentication for cloud credentials -- Leverages platform's correlation IDs for tracing -- Shares error handling patterns - -### Application Module Usage - -The [application module](../application/CLAUDE.md) uses bucket for: - -- Uploading WSI files for processing -- Downloading analysis results -- Managing temporary storage during runs - -### WSI Module Coordination - -Works with [WSI module](../wsi/CLAUDE.md) for: - -- Validating image formats before upload -- Streaming large WSI files -- Thumbnail generation and caching - -## Usage Patterns & Best Practices - -### Basic Operations - -```python -from aignostics.bucket import Service - -service = Service() - - -# Upload file with progress -def progress(current, total): - print(f"Upload: {current / total:.1%}") - - -url = service.upload( - file_path=Path("slide.svs"), - bucket="aignostics-data", - key=f"runs/{run_id}/inputs/slide.svs", - progress_callback=progress, -) - -# Generate signed download URL -download_url = service.generate_signed_url( - bucket="aignostics-data", - key=f"runs/{run_id}/outputs/results.json", - operation="GET", - expiry_seconds=7200, # 2 hours -) -``` - -### Advanced Patterns - -**Batch Operations with Parallelization:** - -```python -from concurrent.futures import ThreadPoolExecutor - - -def upload_batch(files: list[Path]) -> list[str]: - """Upload multiple files in parallel.""" - - with ThreadPoolExecutor(max_workers=5) as executor: - futures = [] - for file_path in files: - future = executor.submit( - service.upload, file_path=file_path, bucket="aignostics-data", key=f"batch/{file_path.name}" - ) - futures.append(future) - - # Collect results - urls = [] - for future in futures: - try: - url = future.result(timeout=300) - urls.append(url) - except Exception as e: - logger.error(f"Upload failed: {e}") - urls.append(None) - - return urls -``` - -**Resumable Download:** - -```python -def download_with_resume(bucket: str, key: str, output_path: Path) -> None: - """Download with automatic resume on failure.""" - - # Check for partial download - if output_path.exists(): - resume_offset = output_path.stat().st_size - logger.debug(f"Resuming download from byte {resume_offset}") - else: - resume_offset = 0 - - # Download with range header - response = s3_client.get_object( - Bucket=bucket, Key=key, Range=f"bytes={resume_offset}-" if resume_offset > 0 else None - ) - - # Append to existing file or create new - mode = "ab" if resume_offset > 0 else "wb" - with output_path.open(mode) as f: - for chunk in response["Body"].iter_chunks(): - f.write(chunk) -``` - -## Testing Strategies - -### Unit Testing - -```python -@pytest.fixture -def mock_s3_client(): - """Mock boto3 S3 client.""" - with patch("boto3.client") as mock: - client = MagicMock() - client.generate_presigned_url.return_value = "https://signed.url" - mock.return_value = client - yield client - - -def test_signed_url_generation(mock_s3_client): - """Test secure URL generation.""" - service = Service() - url = service.generate_signed_url(bucket="test", key="file.txt", expiry_seconds=3600) - - assert url.startswith("https://") - mock_s3_client.generate_presigned_url.assert_called_once() -``` - -### Integration Testing - -```python -@pytest.mark.docker -def test_multipart_upload_recovery(): - """Test multipart upload with simulated failure.""" - # Use localstack for S3 simulation - # Interrupt upload midway - # Verify resume capability -``` - -## Operational Requirements - -### Monitoring & Observability - -**Key Metrics:** - -- Upload/download throughput (MB/s) -- Signed URL generation rate -- Multipart upload success rate -- Storage costs by organization -- Failed transfer recovery rate - -**Logging Standards:** - -```python -logger.debug( - "File uploaded", - extra={ - "bucket": bucket, - "key": key, - "size_mb": file_size / (1024 * 1024), - "duration_seconds": duration, - "transfer_rate_mbps": transfer_rate, - }, -) -``` - -### Security & Compliance - -**Data Protection:** - -- Server-side encryption (SSE-S3 or SSE-KMS) -- Client-side encryption for sensitive data -- Access logging for audit trails -- Versioning for data recovery -- Object lifecycle policies for retention - -**HIPAA Compliance:** - -```python -# Ensure PHI data is encrypted -if contains_phi(file_path): - encryption_config = { - "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "aws:kms", "KMSMasterKeyID": KMS_KEY_ID}}] - } -``` - -## Performance Optimization - -### Transfer Optimization - -- Adaptive chunk sizing based on bandwidth -- Parallel part uploads for multipart -- Connection pooling and reuse -- Regional endpoint selection -- Transfer acceleration for cross-region - -### Caching Strategies - -```python -# Local cache for frequently accessed files -CACHE_DIR = Path.home() / ".aignostics" / "cache" -MAX_CACHE_SIZE = 10 * 1024 * 1024 * 1024 # 10GB - - -def get_with_cache(bucket: str, key: str) -> Path: - """Get file with local caching.""" - cache_path = CACHE_DIR / bucket / key - - if cache_path.exists(): - # Verify cache validity - if is_cache_valid(cache_path, bucket, key): - return cache_path - - # Download to cache - download_to_cache(bucket, key, cache_path) - manage_cache_size() - - return cache_path -``` - -## Common Pitfalls & Solutions - -### Large File Timeouts - -**Problem:** Uploads timing out for multi-GB files - -**Solution:** Use multipart upload with smaller chunks and retry logic - -### Signed URL Expiry - -**Problem:** URLs expiring during long operations - -**Solution:** Generate URLs just-in-time or implement refresh mechanism - -### Cross-Region Latency - -**Problem:** Slow transfers across regions - -**Solution:** Use S3 Transfer Acceleration or regional replication - -## Module Dependencies - -### Internal Dependencies - -- `platform` - Authentication and API client -- `utils` - Logging and utilities - -### External Dependencies - -- `boto3` - AWS SDK for S3 operations -- `google-cloud-storage` - GCS client -- `azure-storage-blob` - Azure Blob Storage - -## Future Enhancements - -1. **Intelligent Tiering**: Automatic storage class optimization -2. **Deduplication**: Content-addressable storage -3. **Compression**: Transparent compression for efficiency -4. **Encryption**: Client-side encryption key management -5. **CDN Integration**: CloudFront/Fastly for global distribution - ---- - -*This module provides enterprise-grade cloud storage operations for medical imaging workflows.* +# bucket — cloud storage on the Aignostics Platform + +Upload/find/download/delete objects in the org's Google Cloud Storage bucket. +Used by the `application` module (input upload, result download) and via the +`bucket` CLI. See `../CLAUDE.md` for where this sits in the SDK. + +Storage is GCS only, reached through a **boto3 S3-compatible client** pointed at +`storage.googleapis.com` (`ENDPOINT_URL_DEFAULT`) with HMAC keys. There is no +Azure and no `google-cloud-storage` dependency; `BucketProtocol` (`_settings.py`) +only defines `GS`/`S3`. + +## Public API — `_service.py` +`Service()` takes no args. Credentials, bucket name, and region come from the +platform org settings (via `PlatformService.get_user_info().organization`), not +constructor args. + +- `get_bucket_name()` — the org's bucket; raises `ValueError` if unset. All ops target it. +- `create_signed_upload_url(object_key, bucket_name=None)` / `create_signed_download_url(object_key, bucket_name=None)` — presigned URLs (`put_object`/`get_object`). +- `upload(source_path, destination_prefix, callback=None)` — file or directory (recursive glob). Returns `{"success": [...], "failed": [...]}` of object keys. +- `find(what, what_is_key=False, detail=False, include_signed_urls=False)` — `what` is a list of regex patterns (default `[".*"]`) or exact keys when `what_is_key=True`. +- `download(what, destination, what_is_key=False, progress_callback=None)` → `DownloadResult`. +- `delete(what, what_is_key=False, dry_run=True)` → count deleted. `dry_run=True` is the default. +- `*_static` variants (`find_static`, `download_static`, `delete_static`) just wrap `Service().()` for one-shot calls. + +## CLI — `_cli.py` +`upload`, `find`, `download`, `delete`, `purge`. (No `list`/`sign`.) `purge` +is `delete(what=None)` over the whole bucket. `delete`/`purge` default to +`--dry-run`; pass `--no-dry-run` to actually delete. + +## Behaviour worth knowing +- `find` infers a common S3 `Prefix` from the patterns/keys (`_compute_s3_prefix`) to cut paginator pages on large buckets; invalid regex raises `ValueError` → CLI exits 2. +- Download skip is ETag-based only: if a local file's MD5 (computed in 100MB chunks, `ETAG_CHUNK_SIZE`) equals the object ETag, it is not re-downloaded. No Range/resume, no multipart. +- Upload streams via a presigned PUT in 1MB chunks (`UPLOAD_CHUNK_SIZE`); download streams in 10MB chunks. `callback(bytes, path)` on upload; `progress_callback(DownloadProgress)` on download. +- `upload` CLI `--destination-prefix` templates `{username}` and `{timestamp}`. +- No permission checks, server-side encryption, audit logging, or local cache — do not add docs for those. + +## Dependencies & gotchas +`boto3`/`botocore` (imported lazily inside methods). See `pyproject.toml`. +Requires `aignostics_bucket_hmac_access_key_id` / `_secret_access_key` / +`aignostics_bucket_name` on the org, else `ValueError`. `_settings.py` holds +`region_name` and signed-URL expirations. `PageBuilder` (`_gui.py`) is exported +only when `nicegui` is installed (see `__init__.py`). diff --git a/src/aignostics/dataset/CLAUDE.md b/src/aignostics/dataset/CLAUDE.md index 44f6d4ff5..feaac2a43 100644 --- a/src/aignostics/dataset/CLAUDE.md +++ b/src/aignostics/dataset/CLAUDE.md @@ -1,545 +1,72 @@ -# CLAUDE.md - Dataset Module - -This file provides comprehensive guidance to Claude Code and human engineers when working with the `dataset` module in this repository. - -## Module Overview - -The dataset module handles enterprise-scale medical imaging dataset operations, providing high-performance downloads from the National Cancer Institute's Imaging Data Commons (IDC) and Aignostics Platform with corporate environment support. - -### Core Responsibilities - -- **IDC Integration**: Access to 40+ TB of public cancer imaging data -- **High-Performance Downloads**: s5cmd-based parallel transfers -- **Corporate Support**: Proxy handling, custom certificates, firewall traversal -- **Process Management**: Subprocess lifecycle with graceful cleanup -- **Progress Tracking**: Real-time download monitoring with callbacks - -### User Interfaces - -**CLI Commands (`_cli.py`):** - -IDC commands: - -- `dataset idc browse` - Open browser to explore IDC portal -- `dataset idc indices` - List available indices -- `dataset idc columns` - Show columns for a given index -- `dataset idc collections` - List available collections -- `dataset idc download` - Download dataset from IDC - -Aignostics commands: - -- `dataset aignostics download` - Download proprietary sample datasets - -**GUI Component (`_gui.py`):** - -- Dataset browser interface -- Download manager with progress tracking -- Collection explorer - -**Service Layer (`_service.py`):** - -Core dataset operations: - -- IDC client with proxy support -- s5cmd subprocess management -- Download progress tracking -- Path length validation for Windows -- Process cleanup on exit - -## Architecture & Design Patterns - -### Service Architecture - -``` -┌────────────────────────────────────────────┐ -│ Dataset Service │ -│ (High-Level Operations) │ -├────────────────────────────────────────────┤ -│ IDC Client Layer │ -│ (Modified for Corporate Proxies) │ -├────────────────────────────────────────────┤ -│ Download Engine (s5cmd) │ -│ (Subprocess Management, Progress) │ -├────────────────────────────────────────────┤ -│ Path & Metadata Management │ -│ (DICOM Organization, Layout) │ -└────────────────────────────────────────────┘ -``` - -### Process Management Pattern - -```python -# Global process registry for cleanup -_active_processes: list[subprocess.Popen] = [] - -# Automatic cleanup on exit -atexit.register(_cleanup_processes) -``` - -## Critical Implementation Details - -### IDC Client Enhancement (`third_party/idc_index.py`) - -**Corporate Environment Modifications:** - -```python -class IDCClient: - """Enhanced IDC client with corporate proxy support.""" - - def __init__(self): - # Original IDC initialization - super().__init__() - - # Add proxy support - self.session = requests.Session() - self.session.proxies = {"http": os.environ.get("HTTP_PROXY"), "https": os.environ.get("HTTPS_PROXY")} - - # Custom certificate bundle - if ca_bundle := os.environ.get("REQUESTS_CA_BUNDLE"): - self.session.verify = ca_bundle - - # Retry configuration for flaky networks - retry_strategy = Retry(total=5, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504]) - adapter = HTTPAdapter(max_retries=retry_strategy) - self.session.mount("https://", adapter) -``` - -### High-Performance Download Engine (`_service.py`) - -**s5cmd Integration with Process Management:** - -```python -TARGET_LAYOUT_DEFAULT = "%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID/" - - -def download_dataset( - collection_id: str, output_dir: Path, progress_callback: Callable = None, filters: dict = None -) -> None: - """Download IDC dataset with s5cmd for maximum performance.""" - - # Query IDC for download manifest - idc_client = IDCClient() - manifest = idc_client.get_download_manifest(collection_id=collection_id, filters=filters) - - # Create s5cmd commands file - commands_file = Path(tempfile.mktemp(suffix=".txt")) - with commands_file.open("w") as f: - for item in manifest: - source = f"s3://public-datasets/{item['aws_url']}" - target = output_dir / self._get_target_path(item) - f.write(f"cp {source} {target}\n") - - # Launch s5cmd subprocess - process = subprocess.Popen( - [ - "s5cmd", - "--endpoint-url", - "https://s3.amazonaws.com", - "--concurrent", - "100", # Parallel transfers - "--retry-count", - "3", - "run", - str(commands_file), - ], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - text=True, - creationflags=SUBPROCESS_CREATION_FLAGS, - ) - - # Track process for cleanup - _active_processes.append(process) - - try: - # Monitor progress - self._monitor_download(process, progress_callback) - finally: - # Ensure cleanup - _active_processes.remove(process) - commands_file.unlink(missing_ok=True) -``` - -### Process Lifecycle Management - -**Graceful Cleanup (from tests):** - -```python -def _terminate_process(process: subprocess.Popen) -> None: - """Terminate subprocess with escalation strategy.""" - try: - logger.warning(f"Terminating subprocess {process.pid}") - process.terminate() - - # Give process time to cleanup - for _ in range(5): - if process.poll() is not None: - return - time.sleep(0.1) - - # Escalate to kill if needed - if process.poll() is None: - logger.warning(f"Force killing subprocess {process.pid}") - process.kill() - process.wait(timeout=5) - - except Exception as e: - logger.exception(f"Error terminating process {process.pid}: {e}") - - -def _cleanup_processes() -> None: - """Clean up all active processes on exit.""" - for process in _active_processes[:]: - if process.poll() is None: # Still running - _terminate_process(process) -``` - -### Progress Tracking Implementation - -**Multi-Threaded Progress Monitoring:** - -```python -def _monitor_download(process: subprocess.Popen, progress_callback: Callable) -> None: - """Monitor s5cmd output for progress updates.""" - - total_files = 0 - completed_files = 0 - failed_files = 0 - current_file = None - - # Parse s5cmd output - for line in process.stdout: - if "ERROR" in line: - failed_files += 1 - logger.error(f"Download error: {line}") - - elif "cp s3://" in line: - # Extract file being processed - match = re.search(r"cp s3://[^\s]+ ([^\s]+)", line) - if match: - current_file = match.group(1) - - elif "OK" in line: - completed_files += 1 - - if progress_callback: - progress = DownloadProgress( - total_files=total_files, - completed_files=completed_files, - failed_files=failed_files, - current_file=current_file, - throughput_mbps=self._calculate_throughput(), - ) - progress_callback(progress) - - # Check final status - return_code = process.wait() - if return_code != 0: - raise RuntimeError(f"s5cmd failed with code {return_code}") -``` - -## Cross-Module Integration - -### Platform Module Coordination - -The dataset module works with [platform module](../platform/CLAUDE.md): - -- Uses platform authentication for private datasets -- Shares correlation IDs for tracing downloads -- Leverages platform's error handling patterns - -### Application Module Usage - -The [application module](../application/CLAUDE.md) uses datasets for: - -- Bulk WSI file acquisition for batch processing -- Training data preparation for ML models -- Reference dataset management - -### WSI Module Integration - -Coordinates with [WSI module](../wsi/CLAUDE.md) for: - -- Validating downloaded DICOM files -- Converting between imaging formats -- Generating dataset previews - -## Usage Patterns & Best Practices - -### Basic Dataset Download - -```python -from aignostics.dataset import Service, IDCClient - -service = Service() -idc = IDCClient() - -# Browse available collections -collections = idc.get_collections() -for collection in collections: - print(f"{collection.id}: {collection.subjects} subjects") - - -# Download specific collection -def progress_callback(progress): - print(f"Downloaded: {progress.completed_files}/{progress.total_files}") - print(f"Throughput: {progress.throughput_mbps:.1f} MB/s") - - -service.download_dataset( - collection_id="TCGA-LUAD", - output_dir=Path("./datasets/lung"), - progress_callback=progress_callback, - filters={ - "Modality": "SM", # Slide Microscopy only - "BodyPartExamined": "LUNG", - }, -) -``` - -### Advanced Patterns - -**Incremental Download with Resume:** - -```python -def download_with_resume(collection_id: str, output_dir: Path): - """Download dataset with resume capability.""" - - # Check existing files - existing_files = set(output_dir.rglob("*.dcm")) - - # Get manifest - idc = IDCClient() - manifest = idc.get_download_manifest(collection_id) - - # Filter already downloaded - to_download = [item for item in manifest if Path(item["target_path"]) not in existing_files] - - logger.debug(f"Resuming download: {len(to_download)} files remaining") - - # Download remaining files - service.download_filtered(to_download, output_dir) -``` - -**Parallel Collection Downloads:** - -```python -from concurrent.futures import ProcessPoolExecutor - - -def download_multiple_collections(collections: list[str]): - """Download multiple collections in parallel.""" - - with ProcessPoolExecutor(max_workers=4) as executor: - futures = {} - - for collection_id in collections: - output_dir = Path(f"./datasets/{collection_id}") - future = executor.submit(service.download_dataset, collection_id=collection_id, output_dir=output_dir) - futures[future] = collection_id - - # Monitor completion - for future in concurrent.futures.as_completed(futures): - collection_id = futures[future] - try: - future.result() - logger.debug(f"Completed: {collection_id}") - except Exception as e: - logger.error(f"Failed {collection_id}: {e}") -``` - -## Testing Strategies - -### Process Management Testing (from `service_test.py`) - -```python -def test_cleanup_processes_terminates_running(): - """Verify subprocess cleanup on exit.""" - mock_process = MagicMock(spec=subprocess.Popen) - mock_process.poll.return_value = None # Still running - - _active_processes.append(mock_process) - _cleanup_processes() - - # Verify termination sequence - mock_process.terminate.assert_called_once() - if mock_process.poll.return_value is None: - mock_process.kill.assert_called_once() - - -def test_graceful_termination(): - """Test graceful process shutdown.""" - mock_process = MagicMock(spec=subprocess.Popen) - mock_process.poll.side_effect = [None, 0] # Terminates gracefully - - _terminate_process(mock_process) - - mock_process.terminate.assert_called_once() - mock_process.kill.assert_not_called() -``` - -### Integration Testing - -```python -@pytest.mark.docker -def test_idc_download_with_proxy(): - """Test IDC download through corporate proxy.""" - # Use squid proxy container - # Verify proxy authentication - # Check certificate handling -``` - -## Operational Requirements - -### Monitoring & Observability - -**Key Metrics:** - -- Download throughput (MB/s, GB/hour) -- Success/failure rates by collection -- s5cmd subprocess health -- Disk space utilization -- Network bandwidth usage - -**Logging Standards:** - -```python -logger.debug( - "Dataset download started", - extra={ - "collection_id": collection_id, - "estimated_size_gb": size_gb, - "output_directory": str(output_dir), - "filters": filters, - }, -) - -logger.error( - "Download failed", - extra={ - "collection_id": collection_id, - "error": str(e), - "completed_files": completed, - "failed_files": failed, - "duration_seconds": duration, - }, -) -``` - -### Performance Optimization - -**Network Optimization:** - -- Concurrent transfers (default: 100) -- Adaptive concurrency based on bandwidth -- Regional endpoint selection -- Connection pooling and reuse - -**Storage Optimization:** - -```python -# Path length validation for Windows -PATH_LENGTH_MAX = 260 - - -def validate_path_length(path: Path) -> bool: - """Ensure path compatibility with Windows.""" - if len(str(path)) > PATH_LENGTH_MAX: - # Shorten path using hash - return shorten_path(path) - return path -``` - -### Error Recovery - -**Automatic Retry with Exponential Backoff:** - -```python -def download_with_retry(manifest_item: dict, max_retries: int = 3): - """Download single file with retry logic.""" - - for attempt in range(max_retries): - try: - download_file(manifest_item) - return - except Exception as e: - wait_time = 2**attempt # Exponential backoff - logger.warning(f"Retry {attempt + 1}/{max_retries} after {wait_time}s") - time.sleep(wait_time) - - raise RuntimeError(f"Failed after {max_retries} attempts") -``` - -## Common Pitfalls & Solutions - -### Corporate Proxy Issues - -**Problem:** SSL verification failures behind proxy - -**Solution:** - -```bash -export HTTPS_PROXY=http://proxy:8080 -export REQUESTS_CA_BUNDLE=/path/to/ca-bundle.crt -export AWS_CA_BUNDLE=/path/to/ca-bundle.crt -``` - -### Large Dataset Storage - -**Problem:** Running out of disk space mid-download - -**Solution:** - -```python -def check_disk_space(required_gb: float, path: Path): - """Pre-flight disk space check.""" - free_gb = shutil.disk_usage(path).free / (1024**3) - if free_gb < required_gb * 1.1: # 10% buffer - raise IOError(f"Insufficient space: {free_gb:.1f}GB < {required_gb:.1f}GB") -``` - -### Windows Path Length - -**Problem:** Path too long errors on Windows - -**Solution:** Use layout with shorter paths or enable long path support - -## Module Dependencies - -### Internal Dependencies - -- `platform` - Authentication and API client -- `utils` - Logging and process utilities -- `wsi` - Image validation - -### External Dependencies - -- `s5cmd` - High-performance S3 client -- `idc-index-data` - IDC metadata -- `pydicom` - DICOM file handling -- `duckdb` - Local dataset indexing - -### Modified Third-Party - -- `third_party/idc_index.py` - Enhanced IDC client with proxy support - -## Future Enhancements - -1. **Streaming Downloads**: Process files during download -2. **Smart Caching**: Local dataset cache management -3. **Bandwidth Throttling**: Rate limiting for shared networks -4. **Compression**: On-the-fly compression for storage efficiency -5. **P2P Transfer**: Distributed dataset sharing within organization - -## Performance Considerations - -**Optimization Targets:** - -- Parallel downloads using s5cmd with configurable concurrency -- Chunked transfers for large files -- Automatic retry on failures -- Efficient memory usage through streaming - ---- - -*This module enables high-throughput downloads of cancer imaging datasets from public repositories.* +# dataset — download imaging datasets from IDC and Aignostics + +Downloads public cancer imaging data from the NCI Imaging Data Commons (IDC) +and proprietary sample datasets from Aignostics buckets. CLI + GUI wrap a +static-method `Service`. See `../CLAUDE.md` for where this module sits in the SDK. + +## Public API (`_service.py`) +`Service` (subclass of `utils.BaseService`) exposes static methods: +- `download_idc(source, target, target_layout=TARGET_LAYOUT_DEFAULT, dry_run=False) -> int` — + synchronous IDC download; returns count of matched identifier types. +- `download_with_queue(queue, source, target, target_layout, dry_run)` — same as + above but runs the actual download in a subprocess and pushes float progress + (0.0–1.0) onto a `multiprocessing.Queue`; used by the GUI. +- `download_aignostics(source_url, destination_directory, download_progress_callable=None) -> Path` — + streams one bucket object to a folder via a platform signed URL (`requests`, + 8192-byte chunks); callback gets `(len(chunk), total_size, filename)`. + +`__init__` also re-exports `IDCClient` (from `aignostics.third_party.idc_index`) +and, when `nicegui` is importable, `PageBuilder`. + +## CLI (`_cli.py`) +Root `dataset` with two sub-Typers: +- `dataset idc browse` — open the IDC portal in a browser. +- `dataset idc indices` — list `client.indices_overview` keys. +- `dataset idc columns [--index sm_instance_index]` — list columns of an index. +- `dataset idc query [SQL] [--indices ...]` — run a SQL query via `client.sql_query`. +- `dataset idc download SOURCE [TARGET] [--target-layout] [--dry-run]` — calls + `Service.download_idc`. +- `dataset aignostics download SOURCE_URL [DEST]` — calls `Service.download_aignostics` + with a rich progress bar. + +There is no `collections` subcommand. + +## Behaviour worth knowing +- **`source` matching**: `source` is a comma-separated list of IDs. Each ID list + is matched, in order, against `collection_id`, `PatientID`, `StudyInstanceUID`, + `SeriesInstanceUID`, then `SOPInstanceUID` (last uses `sm_instance_index`). + A `download_from_selection` call fires per matched column. Zero matches across + all columns raises `ValueError`. +- **How IDC bytes actually move**: `download_idc` calls + `IDCClient.download_from_selection(..., use_s5cmd_sync=True)` directly. + `download_with_queue` instead builds a small `python -c` script and runs it in a + `subprocess.Popen` (or `sys.executable --exec-script` under PyInstaller) so + progress can be scraped. s5cmd is invoked by idc-index internally — this module + never calls the `s5cmd` binary itself. +- **Progress scraping**: `Service._capture_progress_output(process, queue, base_progress=0.5)` + reads subprocess stderr one char at a time and matches `r"Downloading data:\s+(\d+)%"`, + scaling the percentage into `[base_progress, 0.99]` and pushing `1.0` on completion. + No throughput/ETA calculation. +- **Subprocess cleanup**: module-level `_active_processes` list + `atexit`-registered + `_cleanup_processes()`; `_terminate_process()` does terminate → 0.5s grace → kill. +- **Exit codes**: `ValueError` (bad input) → exit 2; any other exception → exit 1. +- **Target dir must already exist** — `download_idc`/`download_with_queue` raise + `ValueError` if it does not (CLI also enforces `exists=True`). `download_aignostics` + creates the destination directory. +- `TARGET_LAYOUT_DEFAULT` and `PATH_LENGTH_MAX = 260` are defined in both + `_service.py` and `_cli.py`. `PATH_LENGTH_MAX` is currently unused — there is no + path-shortening helper. + +## `IDCClient` (`../third_party/idc_index.py`) +Vendored + patched copy of the `idc-index` client. Use the +`IDCClient.client()` classmethod (cached per-class instance), not `IDCClient()`. +Patched to run behind corporate proxies and to retry transient HTTP failures via +`tenacity` (`_requests_get_with_retry`, 4 attempts, exponential jitter, retries +connection/timeout errors and HTTP 5xx). Requires the `s5cmd` binary on PATH (or +bundled with the package) — `__init__` raises `FileNotFoundError` otherwise. + +## Dependencies & gotchas +Deps (`idc-index`, `s5cmd`, `pandas`, `requests`, …) and extras: see +`pyproject.toml`. `Service.info`/`health` are stubs (return `{}` / `UP`). GUI code +in `_gui.py` is imported only when `find_spec("nicegui")` succeeds (`__init__.py`). +`download_aignostics` signs the source URL via `platform.generate_signed_url`. diff --git a/src/aignostics/gui/CLAUDE.md b/src/aignostics/gui/CLAUDE.md index 336fd6c7d..39cf5b531 100644 --- a/src/aignostics/gui/CLAUDE.md +++ b/src/aignostics/gui/CLAUDE.md @@ -17,7 +17,7 @@ The gui module provides common GUI framework components and theming for the Aign - `_theme.py` - Application theme and styling (`PageBuilder`, `theme`) - `_frame.py` - Common layout components and health updates -- `_error.py` - Error page handling (`ErrorPageBuilder`) +- `_error.py` - Error page handling (in-file class `PageBuilder`, imported as `ErrorPageBuilder`) **Usage Pattern:** @@ -51,20 +51,10 @@ The GUI enforces health checks before allowing critical operations: **Health State Management (`_frame.py`):** -```python -launchpad_healthy: bool | None = None # None = loading, True = healthy, False = unhealthy - - -async def _health_load_and_render() -> None: - nonlocal launchpad_healthy - with contextlib.suppress(Exception): - launchpad_healthy = bool(await run.cpu_bound(SystemService.health_static)) - health_icon.refresh() - health_link.refresh() - - -ui.timer(interval=HEALTH_UPDATE_INTERVAL, callback=_update_health, immediate=True) -``` +A `ui.timer` fires every `HEALTH_UPDATE_INTERVAL` seconds; the callback sets a +`launchpad_healthy: bool | None` state (`None` while loading) from +`await SystemService.health_static()` and refreshes the display components. +Health-enforcement semantics are owned by `system/CLAUDE.md` — see it for details. **Health Display Components:** diff --git a/src/aignostics/notebook/CLAUDE.md b/src/aignostics/notebook/CLAUDE.md index 9af048f6f..41aa9b411 100644 --- a/src/aignostics/notebook/CLAUDE.md +++ b/src/aignostics/notebook/CLAUDE.md @@ -1,339 +1,37 @@ -# CLAUDE.md - Notebook Module - -This file provides comprehensive guidance to Claude Code and human engineers when working with the `notebook` module in this repository. - -## Module Overview - -The notebook module provides interactive Marimo notebook functionality for the Aignostics Platform, enabling data exploration and analysis in a reactive notebook environment. - -### Core Responsibilities - -- **Marimo Integration**: Embedded Marimo notebook server management -- **Process Management**: Lifecycle control of notebook server subprocess -- **Health Monitoring**: Server health checks and status reporting -- **Interactive Analysis**: Reactive notebook environment for data exploration - -### User Interfaces - -**Service Layer (`_service.py`):** - -The service manages the Marimo notebook server: - -- Server lifecycle (start/stop/restart) -- Health monitoring -- Process management with graceful shutdown - -**GUI Component (`_gui.py`):** - -- Embedded notebook interface within the main GUI -- Server status display -- Launch controls - -**No CLI**: This module is GUI-only for interactive use - -## Architecture & Design Patterns - -### Process Management Pattern - -```python -class _Runner: - """Manages Marimo server subprocess.""" - - _marimo_server: Popen[str] | None = None - _monitor_thread: Thread | None = None - _server_ready: Event = Event() - - def __init__(self): - atexit.register(self.stop) # Cleanup on exit -``` - -### Server Lifecycle - -``` -Start → Monitor Output → Extract URL → Ready - ↓ - Health Check - ↓ - Stop → Cleanup -``` - -## Critical Implementation Details - -### Marimo Server Management - -**Server Startup:** - -```python -MARIMO_SERVER_STARTUP_TIMEOUT = 60 # seconds - - -def start(self, notebook_path: Path, host: str, port: int): - """Start Marimo server subprocess.""" - - cmd = [sys.executable, "-m", "marimo", "run", str(notebook_path), "--host", host, "--port", str(port), "--headless"] - - self._marimo_server = Popen(cmd, stdout=PIPE, stderr=STDOUT, text=True, creationflags=SUBPROCESS_CREATION_FLAGS) - - # Monitor thread watches for server URL - self._monitor_thread = Thread(target=self._monitor_output, daemon=True) -``` - -**URL Extraction Pattern:** - -```python -def _monitor_output(self): - """Monitor Marimo output for server URL.""" - - url_pattern = r"http://\S+:\d+" - - for line in self._marimo_server.stdout: - self._output += line - - if match := re.search(url_pattern, line): - self._server_url = match.group() - self._server_ready.set() -``` - -### Health Monitoring - -```python -def health(self) -> Health: - """Check server and monitor thread health.""" - - components = { - "marimo_server": Health(status=Health.Code.UP if self.is_marimo_server_running() else Health.Code.DOWN), - "monitor_thread": Health(status=Health.Code.UP if self.is_monitor_thread_alive() else Health.Code.DOWN), - } - - return Health(status=Health.Code.UP, components=components) -``` - -## Usage Patterns - -### Starting a Notebook Session - -```python -from aignostics.notebook import Service -from pathlib import Path - -service = Service() - -# Start Marimo server -notebook = Path("analysis.marimo.py") -server_url = service.start_notebook(notebook_path=notebook, host="127.0.0.1", port=8080) - -# Server URL available after startup -print(f"Notebook running at: {server_url}") - -# Check health -health = service.health() -print(f"Server status: {health.status}") - -# Stop when done -service.stop_notebook() -``` - -### Integration with GUI - -The notebook module integrates with the main GUI launchpad: - -```python -# In GUI context -from aignostics.notebook._gui import create_notebook_interface - - -def setup_notebook_tab(ui): - """Add notebook tab to GUI.""" - - with ui.tab("Notebook"): - create_notebook_interface() -``` - -## Dependencies & Integration - -### Internal Dependencies - -- `utils` - Base service, logging, subprocess flags -- `constants` - Default notebook configuration - -### External Dependencies - -- `marimo` - Reactive notebook framework (required) - -### Integration Points - -- **GUI Module**: Embedded in main launchpad interface -- **Application Module**: Can launch notebooks for result analysis -- **Dataset Module**: Interactive data exploration after downloads - -## Operational Requirements - -### Process Management - -- **Startup timeout**: 60 seconds -- **Graceful shutdown**: Registered with `atexit` -- **Output monitoring**: Separate daemon thread -- **Resource cleanup**: Automatic on exit - -### Monitoring & Observability - -**Key Metrics:** - -- Server startup time -- Active notebook sessions -- Memory usage per session -- CPU utilization - -**Logging Patterns:** - -```python -logger.debug("Starting Marimo server", extra={"notebook": str(notebook_path), "host": host, "port": port}) - -logger.warning("Server startup timeout", extra={"timeout": MARIMO_SERVER_STARTUP_TIMEOUT, "output": self._output}) -``` - -## Common Pitfalls & Solutions - -### Port Conflicts - -**Problem:** Port already in use - -**Solution:** - -```python -def find_free_port(start=8080, end=9000): - """Find available port.""" - import socket - - for port in range(start, end): - with socket.socket() as s: - if s.connect_ex(("127.0.0.1", port)) != 0: - return port - raise RuntimeError("No free ports") -``` - -### Server Startup Failures - -**Problem:** Marimo server fails to start - -**Solution:** - -```python -# Check Marimo installation -if not find_spec("marimo"): - raise ImportError("Marimo not installed: pip install marimo") - -# Verify notebook file exists -if not notebook_path.exists(): - raise FileNotFoundError(f"Notebook not found: {notebook_path}") -``` - -### Zombie Processes - -**Problem:** Server process not cleaned up - -**Solution:** - -```python -def cleanup_zombie_processes(): - """Kill any lingering Marimo processes.""" - import psutil - - for proc in psutil.process_iter(["pid", "name", "cmdline"]): - if "marimo" in proc.info["cmdline"]: - proc.terminate() -``` - -## Development Guidelines - -### Adding New Notebook Templates - -1. Create template in `notebooks/` directory -2. Use `.marimo.py` extension -3. Include metadata header: - - ```python - # /// script - # requires-python = ">=3.10" - # dependencies = ["aignostics", "marimo", "pandas"] - # /// - ``` - -### Extending Server Management - -```python -class ExtendedRunner(_Runner): - """Extended runner with additional features.""" - - def restart(self): - """Restart server.""" - self.stop() - self.start(self._last_notebook, self._last_host, self._last_port) - - def get_notebooks(self) -> list[Path]: - """List available notebooks.""" - notebook_dir = get_user_data_directory() / "notebooks" - return list(notebook_dir.glob("*.marimo.py")) -``` - -## Testing Strategies - -### Unit Testing - -```python -@pytest.fixture -def mock_marimo_server(): - """Mock Marimo subprocess.""" - with patch("subprocess.Popen") as mock: - process = MagicMock() - process.stdout = iter(["Starting server...", "http://localhost:8080"]) - process.poll.return_value = None - mock.return_value = process - yield mock - - -def test_server_startup(mock_marimo_server): - """Test server starts and extracts URL.""" - service = Service() - url = service.start_notebook(Path("test.marimo.py")) - assert url == "http://localhost:8080" -``` - -### Integration Testing - -```python -@pytest.mark.integration -def test_real_marimo_server(): - """Test with actual Marimo server.""" - service = Service() - - # Create test notebook - notebook = Path("test.marimo.py") - notebook.write_text("import marimo as mo\nmo.md('Test')") - - try: - url = service.start_notebook(notebook) - assert requests.get(url).status_code == 200 - finally: - service.stop_notebook() - notebook.unlink() -``` - -## Performance Notes - -### Resource Usage - -- **Memory**: ~100-500MB per notebook session -- **CPU**: Minimal when idle, spikes during cell execution -- **Network**: Local server only (default 127.0.0.1) - -### Optimization Opportunities - -1. Connection pooling for multiple notebooks -2. Shared kernel for resource efficiency -3. Notebook preloading for faster startup -4. Output caching for repeated computations - ---- - -*This module enables interactive data analysis through reactive Marimo notebooks integrated with the Aignostics Platform.* +# notebook — embedded Marimo notebook server + +Launches a local [Marimo](https://marimo.io/) server running a single fixed +notebook, embedded in the GUI for exploring application results. GUI-only, no +CLI. See `../CLAUDE.md` for placement in the SDK. + +The whole module is gated in `__init__.py` on +`find_spec("marimo") and find_spec("nicegui")` — if either is missing it exports +nothing and is silently skipped (no error). + +## Public API — `_service.py` +- Module singleton `runner: _Runner | None` + `_get_runner()` (lazy init). `_Runner` owns the subprocess, monitor thread, and URL state. +- `Service(BaseService)` is a thin facade over the singleton: + - `start()` — no args; starts the server (if not already running) and returns its URL. + - `stop()`, `health()`, `is_marimo_server_running()`, `is_monitor_thread_alive()`. + +## GUI — `_gui.py` +`PageBuilder(BasePageBuilder)` registers two pages: +- `/notebook` — landing card with a launch link. +- `/notebook/{run_id}` — calls `Service().start()` and embeds `server_url` in an + `