From c42cdb7a45ae18af30415eb6ec69d6898961f7ef Mon Sep 17 00:00:00 2001 From: Eric Wait Date: Wed, 8 Jul 2026 20:30:53 -0500 Subject: [PATCH] docs: document the feature -> develop -> PR -> main branching model CLAUDE.md: current build/test/CI reality (scikit-build-core pip path, conda-forge distribution, the Python TIFF accuracy suite, ci.yml + release.yml instead of the removed conda-build.yml). CONTRIBUTING.md: branching model and PRs target develop; release flow goes through a develop -> main PR. ci.yml: drop the retired conda-forge branch from the push triggers. ci-readme.md described a workflow that no longer exists; removed. Co-Authored-By: Claude Fable 5 --- .github/workflows/ci.yml | 2 +- CLAUDE.md | 22 ++++- CONTRIBUTING.md | 23 ++++- ci-readme.md | 184 --------------------------------------- 4 files changed, 38 insertions(+), 193 deletions(-) delete mode 100644 ci-readme.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a8f32f2..16e2b27 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -8,7 +8,7 @@ name: CI on: push: - branches: [main, develop, conda-forge] + branches: [main, develop] pull_request: branches: [main, develop] workflow_dispatch: diff --git a/CLAUDE.md b/CLAUDE.md index f327ee0..8d488fc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,11 @@ An NVIDIA GPU + CUDA Toolkit (12.x in CI) are required to build; tests skip grac ## Build Commands ```bash -# Configure + build (Linux/macOS-style; CUDA toolkit must be on PATH) +# Canonical Python build (scikit-build-core drives CMake + nvcc and installs +# the package; run from the repository root in an env with cmake/ninja/numpy) +pip install . --no-build-isolation -v + +# Raw CMake targets (Linux/macOS-style; CUDA toolkit must be on PATH) cmake -S . -B build cmake --build build --config Release --target HydraPy # Python extension cmake --build build --config Release --target HydraMex # MATLAB MEX (needs MATLAB) @@ -23,9 +27,10 @@ cmake --build --preset VS64-debug ``` - `-DHYDRA_MODULE_NAME=HIP` renames the output module (default `Hydra`; CI uses `HIP` for MATLAB builds). -- The `HydraPy` target writes `Hydra.pyd`/`Hydra.so` directly into `src/Python/hydra_image_processor/`; then install the Python package with `pip install .` from `src/Python/`. +- The `HydraPy` target writes `Hydra.pyd`/`Hydra.so` directly into `src/Python/hydra_image_processor/`. - Building `HydraMex` triggers a post-build MATLAB step (`src/c/Mex/autoBuildMex.cmake`) that regenerates the `.m` wrapper files — never hand-edit generated wrappers in `src/MATLAB/+Hydra/@Cuda/`. -- Conda package: `conda build recipe --python 3.12` (recipe in `recipe/`; `bld.bat` for Windows, `build.sh` for Linux/macOS). +- `-DCMAKE_CUDA_ARCHITECTURES` defaults to `all-major`; use `86`/`native` for fast local iteration. +- End-user distribution is via **conda-forge** (`conda install -c conda-forge hydra-image-processor`); the recipe lives in the separate `hydra-image-processor-feedstock` repository and must stay a patch-free `pip install .` wrapper. - `-DUSE_PROCESS_MUTEX=ON` enables a cross-process GPU mutex (boost interprocess, vendored in `src/c/external/`). ## Tests @@ -39,6 +44,9 @@ ctest --test-dir build # runs CppAccuracyTest # Python smoke tests (plain scripts, no pytest) cd src/Python && python test_package.py +# Python TIFF accuracy suite (needs a GPU + tifffile; THE release gate, see TESTING.md) +python src/Python/Test/test_accuracy.py + # MATLAB accuracy tests (matlab.unittest; compares against test_data/*.tif) matlab -batch "results = runtests('src/MATLAB/+Test/AccuracyTest.m')" ``` @@ -80,6 +88,12 @@ Python and MATLAB compile the same `HydraCudaStatic` backend and command framewo Type dispatch, Python method registration, and MATLAB wrappers are all generated from that. Only the pretty-named Python wrappers in `src/Python/hydra_image_processor/` (`cuda/core.py`, `core.py`) are added by hand. +## Branching & releases + +- `main` is protected (PR + green CI required); `develop` is the integration branch. +- Work on `feature/` branches cut from `develop`, merge them into `develop`, and open a PR `develop` → `main` when a release-worthy state is reached. +- Releases are tag-driven: bump `pyproject.toml` + `CHANGELOG.md`, work through TESTING.md (the GPU accuracy suites cannot run in CI), merge to `main`, then `git tag vX.Y.Z && git push origin vX.Y.Z`. `release.yml` enforces tag == version and publishes the GitHub Release; conda-forge's autotick bot then opens a feedstock version-bump PR. + ## CI -`.github/workflows/conda-build.yml` builds the conda package on Windows for Python 3.10–3.12 (CUDA 12.4.1). `.github/workflows/matlab-multibuild.yml` builds `HIP` MEX binaries on Windows + Linux and packages the MATLAB toolbox from `src/MATLAB/HydraImageProcessor.prj`. +`.github/workflows/ci.yml` is the build-gate: on Windows + Linux it installs CUDA/compilers from conda-forge, builds via the same `pip install .` path the feedstock uses, and runs an import smoke test (runners have no GPU — accuracy suites run locally, see TESTING.md). `.github/workflows/release.yml` runs on `v*` tags. `.github/workflows/matlab-multibuild.yml` builds `HIP` MEX binaries on Windows + Linux and packages the MATLAB toolbox from `src/MATLAB/HydraImageProcessor.prj`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 346d2df..44b206d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,20 @@ Hydra Image Processor (Hydra) has been written to expedite the addition of functionality. All of the machinery needed to distribute data and iterate over neighborhoods is included. Adding a new function is as easy as copying a template file and replacing the requisite lines of code. This paradigm is intended to encourage research and development of operations that would otherwise be intractable without hardware acceleration. # How to contribute -Contribution is as easy as a pull request on GitHub. Following the guidelines will ensure requests are not rejected for minor issues. Once your code conforms to the guidelines, please submit your requests [here](https://github.com/ericwait/hydra-image-processor/pulls). +Contribution is as easy as a pull request on GitHub. Following the guidelines will ensure requests are not rejected for minor issues. Once your code conforms to the guidelines, please submit your requests [here](https://github.com/ericwait/hydra-image-processor/pulls). + +## Branching model + +```text +feature/ ──► develop ──► (PR) ──► main ──► tag vX.Y.Z ──► release +``` + +- **`main`** is protected and always releasable: changes land only through a pull + request with a green CI gate, and every release is a tag on `main`. +- **`develop`** is the integration branch. Cut `feature/` (or `fix/`) + branches from it and target your pull requests at **`develop`**, not `main`. +- When `develop` reaches a release-worthy state, a maintainer opens the + `develop` → `main` PR and follows the release checklist below. Alternatively, you can contribute by detailing your needs on this [forum](https://www.hydraimageprocessor.com/forum/request-functionality). Hydra was also designed to be a practical library that meets the needs of microscopist. I really enjoy when theory and application meet. By explaining in detail (examples are always helpful as well) what your needs are, the community will be able to find novel ways accomplish your goals. @@ -47,9 +60,11 @@ GPU box before tagging a release.** Releases are tag-driven. conda-forge's autotick bot watches for new GitHub releases and opens a feedstock version-bump PR automatically. -1. Run the full accuracy suite locally on a GPU and confirm it passes. -2. Bump `[project].version` in `pyproject.toml` and add a section to `CHANGELOG.md`. -3. Merge to `main` with a green CI gate. +1. Run the full accuracy suite locally on a GPU and confirm it passes (work through + [TESTING.md](TESTING.md) top to bottom). +2. Bump `[project].version` in `pyproject.toml` and add a section to `CHANGELOG.md` + (on `develop`). +3. Open a PR from `develop` to `main` and merge it with a green CI gate. 4. Tag and push: `git tag vX.Y.Z && git push origin vX.Y.Z` (the tag **must** match the `pyproject.toml` version — the release workflow enforces this). 5. The `Release` workflow creates the GitHub Release. Merge the resulting feedstock autotick diff --git a/ci-readme.md b/ci-readme.md deleted file mode 100644 index 783dc7f..0000000 --- a/ci-readme.md +++ /dev/null @@ -1,184 +0,0 @@ -# CI/CD Setup for Hydra Image Processor - -This document describes how to build and distribute the Hydra Python extensions using GitHub Actions. - -## Files to Add to Your Repository - -1. **`.github/workflows/build-python-artifacts.yml`** - Main CI workflow that builds the Python extensions -2. **`scripts/download_artifacts.py`** - Helper script to download artifacts from GitHub Actions -3. **`test_hydra_module.py`** - Test script to verify the built module works correctly -4. **Update `CMakeLists.txt`** - Change `HYDRA_MODULE_NAME` from "HIP" to "Hydra" - -## Build Process - -### Step 1: Update Your CMakeLists.txt - -Change the module name in your root `CMakeLists.txt`: -```cmake -set(HYDRA_MODULE_NAME "Hydra") # Changed from "HIP" -``` - -### Step 2: Commit and Push the Workflow - -```bash -# Create the workflow directory -mkdir -p .github/workflows - -# Copy the workflow file -cp build-python-artifacts.yml .github/workflows/ - -# Create scripts directory -mkdir -p scripts -cp download_artifacts.py scripts/ - -# Commit and push -git add .github/workflows/build-python-artifacts.yml -git add scripts/download_artifacts.py -git add test_hydra_module.py -git commit -m "Add CI/CD for Python extensions with Hydra module name" -git push -``` - -### Step 3: Monitor the Build - -1. Go to your GitHub repository -2. Click on the "Actions" tab -3. Watch the workflow run -4. Builds will create artifacts for: - - Linux (`.so` files) with CUDA 12.2 and 12.3 - - Windows (`.pyd` files) with CUDA 12.2 and 12.3 - - Python versions 3.9, 3.10, and 3.11 - -## Downloading and Testing Artifacts - -### Method 1: Via GitHub Web Interface - -1. Go to Actions tab -2. Click on a completed workflow run -3. Scroll down to "Artifacts" -4. Download the artifacts you need - -### Method 2: Using the Download Script - -```bash -# Find your run ID from the GitHub Actions page -python scripts/download_artifacts.py --run-id - -# The script will download all artifacts and create a test script -cd downloaded_artifacts -python test_import.py -``` - -### Method 3: Using GitHub CLI - -```bash -# List recent workflow runs -gh run list --workflow=build-python-artifacts.yml - -# Download all artifacts from a specific run -gh run download -``` - -## Testing the Module - -### Quick Test -```python -import sys -sys.path.insert(0, '/path/to/artifact') -import Hydra -print(Hydra.Info()) -``` - -### Comprehensive Test -```bash -python test_hydra_module.py -``` - -## Using in Your Project - -### For [[home-media-ai]] or Other Projects - -1. Download the appropriate artifact for your platform: - - Linux: `Hydra-linux-cuda12.2-py3.10/Hydra.so` - - Windows: `Hydra-windows-cuda1220-py3.10/Hydra.pyd` - -2. Copy to your project: -```bash -# Linux -cp downloaded_artifacts/Hydra-linux-cuda12.2-py3.10/Hydra.so ~/home-media-ai/ - -# Windows -copy downloaded_artifacts\Hydra-windows-cuda1220-py3.10\Hydra.pyd C:\projects\home-media-ai\ -``` - -3. Import in Python: -```python -import Hydra - -# Check available functions -info = Hydra.Info() -for cmd in info: - print(cmd.command) - -# Use the module -import numpy as np -image = np.random.rand(512, 512).astype(np.float32) -filtered = Hydra.Gaussian(image, sigma=[2.0, 2.0]) -``` - -## Artifact Naming Convention - -Artifacts are named with the pattern: -``` -Hydra-{platform}-cuda{version}-py{python_version} -``` - -Examples: -- `Hydra-linux-cuda12.2-py3.10` -- `Hydra-windows-cuda1220-py3.11` - -## Troubleshooting - -### Import Errors - -If you get import errors, check: -1. **CUDA Runtime**: Ensure CUDA 12.x is installed -2. **Python Version**: Match the Python version of the artifact -3. **Dependencies**: Install numpy: `pip install numpy` -4. **Library Path**: On Linux, you may need to set `LD_LIBRARY_PATH` - -### CUDA Not Found - -The module will import but operations may fail if CUDA is not available. -This is normal for testing on systems without GPUs. - -### Windows Specific - -On Windows, you may need: -- Visual C++ Redistributables -- CUDA Toolkit 12.x -- Ensure `.pyd` file is in a directory on your Python path - -## Next Steps - -Once testing is successful: - -1. **Create Wheels**: Package as `.whl` files for pip installation -2. **Conda Package**: Create conda recipe for conda-forge submission -3. **Release Automation**: Auto-publish on GitHub releases - -## Conda-forge Submission (Future) - -After successful testing, we'll: -1. Fork conda-forge/staged-recipes -2. Add recipe in `recipes/hydra-image-processor/` -3. Submit PR for review -4. Once accepted, get a feedstock repository -5. Automatic builds on each release - -## Support - -For issues or questions: -- Check the [Actions logs](https://github.com/ericwait/hydra-image-processor/actions) -- Open an issue on the repository -- Check the [Hydra website](https://www.hydraimageprocessor.com) \ No newline at end of file