Dev to main

.github/workflows/tag_and_publish.yml

@@ -24,7 +24,7 @@ jobs:
       with:
         python-version: '3.10'
     - name: Install dependencies
-      run: uv sync
+      run: uv sync --extra full
     - name: Get Python version and Update README.md
       run: |
         python_version=$(grep "requires-python" pyproject.toml | grep -o ">=[^\"]*")

.github/workflows/test_and_lint.yml

@@ -4,22 +4,25 @@ on:
   pull_request:
     branches:
       - main
+      - dev
 
 jobs:
   ci:
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
+      fail-fast: true
       matrix:
-        python-version: [ '3.10', '3.11', '3.12' ]
+        os: [ ubuntu-latest, macos-latest, windows-latest ]
+        python-version: [ '3.12' ]
     steps:
     - uses: actions/checkout@v6
     - name: Set up Python ${{ matrix.python-version }}
       uses: astral-sh/setup-uv@v7
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
-      run: uv sync
+      run: uv sync --extra full
     - name: Run linter checks
-      run: uv run flake8 . && uv run interrogate --verbose .
+      run: uv run ruff check . && uv run ruff format --check . && uv run interrogate --verbose .
     - name: Run tests and coverage
-      run: uv run coverage run -m unittest discover && uv run coverage report
+      run: uv run coverage run -m pytest && uv run coverage report

CONTRIBUTING.md

@@ -2,33 +2,43 @@
 
 ### Linters and testing
 
-There are several libraries used to run linters, check documentation, and run tests.
+This project uses [uv](https://docs.astral.sh/uv/) to manage its environment and
+[ruff](https://docs.astral.sh/ruff/) for linting and formatting.
 
-- Please test your changes using the **coverage** library, which will run the tests and log a coverage report:
+- Install dependencies (including the `dev` group):
 
 ```bash
-coverage run -m unittest discover && coverage report
+uv sync
+```
+
+- Run tests with **pytest** under **coverage**:
+
+```bash
+uv run coverage run -m pytest && uv run coverage report
 ```
 
 - Use **interrogate** to check that modules, methods, etc. have been documented thoroughly:
 
 ```bash
-interrogate .
+uv run interrogate .
 ```
 
-- Use **flake8** to check that code is up to standards (no unused imports, etc.):
+- Use **ruff** to lint the code (catches unused imports, style issues, sorts imports, etc.):
+
 ```bash
-flake8 .
+uv run ruff check .
 ```
 
-- Use **black** to automatically format the code into PEP standards:
+- Use **ruff** to automatically format the code:
+
 ```bash
-black .
+uv run ruff format .
 ```
 
-- Use **isort** to automatically sort import statements:
+- To apply ruff's autofixes in one shot:
+
 ```bash
-isort .
+uv run ruff check --fix . && uv run ruff format .
 ```
 
 ### Pull requests

README.md

@@ -5,44 +5,179 @@
 [![semantic-release: angular](https://img.shields.io/badge/semantic--release-angular-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)
 ![Interrogate](https://img.shields.io/badge/interrogate-100.0%25-brightgreen)
 ![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)
-![Python](https://img.shields.io/badge/python->=3.10-blue?logo=python)
-
-## Usage
- - To use this template, click the green `Use this template` button and `Create new repository`.
- - After github initially creates the new repository, please wait an extra minute for the initialization scripts to finish organizing the repo.
- - To enable the automatic semantic version increments: in the repository go to `Settings` and `Collaborators and teams`. Click the green `Add people` button. Add `svc-aindscicomp` as a collaborator with "write" role. Modify the file in `.github/workflows/tag_and_publish.yml` by removing the if statement in line 67. The semantic version will now be incremented every time a code is committed into the main branch.
- - To publish to PyPI, first submit a request to the [Scientific Computing issue tracker](https://github.com/AllenNeuralDynamics/aind-scientific-computing/issues)
- to have the repository added as a Trusted Publisher for PyPI. Please specify the repository name (and desired PyPI package name if different).
- After the repository is added as a Trusted Publisher, enable semantic versioning and remove the if statement in line 73 in `.github/workflows/tag_and_publish.yml`. The code will now be published to PyPI every time the code is committed into the main branch.
- - The `.github/workflows/test_and_lint.yml` file will run automated tests and style checks every time a Pull Request is opened. If the checks are undesired, the `test_and_lint.yml` can be deleted. The strictness of the code coverage level, etc., can be modified by altering the configurations in the `pyproject.toml` file and the `.flake8` file.
- - Please make any necessary updates to the README.md and CITATION.cff files
+![Python](https://img.shields.io/badge/python->=3.12-blue?logo=python)
+
+Tools for processing raw [dynamic foraging](https://github.com/AllenNeuralDynamics/dynamic-foraging-task)
+acquisition data into derived containers for NWB. The package loads raw acquisition
+streams through the
+[aind-behavior-dynamic-foraging](https://github.com/AllenNeuralDynamics/Aind.Behavior.DynamicForaging)
+data contract and assembles higher-level tables (currently the NWB `trials` table)
+from those streams.
+
+> **Note:** Data must be acquired in the `aind-behavior-dynamic-foraging` data
+> contract format to be compatible with these tools.
+
+## Inputs and outputs
+**Input** — a raw acquisition directory in the `aind-behavior-dynamic-foraging`
+data contract format (Harp device registers, software events, and the task-logic /
+rig / session input schemas), loaded via `RawDataLoader`.
+
+**Outputs** — the NWB-ready containers assembled from those streams:
+
+- **Acquisition.** Built by `AcquisitionBuilder` and written to the NWB acquisition
+  module:
+  - 4 derived event series:
+    - `left_reward_delivery_time` / `right_reward_delivery_time` — reward delivery
+      times per lick port, with each event annotated as `earned`, `manual`, or
+      `automatic`.
+    - `left_lick_time` / `right_lick_time` — detected lick times per lick port.
+  - All raw contract streams — every stream returned by the loader
+    (`get_all_raw_data()`) is additionally packaged verbatim as a `DynamicTable`,
+    so the full raw dataset travels alongside the derived series.
+- **Trials table.** Built by `TrialTableBuilder`, one row per trial written to the
+  NWB `trials` table (see [`docs/trials_table_mapping.md`](docs/trials_table_mapping.md)).
 
 ## Level of Support
-Please indicate a level of support:
- - [ ] Supported: We are releasing this code to the public as a tool we expect others to use. Issues are welcomed, and we expect to address them promptly; pull requests will be vetted by our staff before inclusion.
+ - [x] Supported: We are releasing this code to the public as a tool we expect others to use. Issues are welcomed, and we expect to address them promptly; pull requests will be vetted by our staff before inclusion.
  - [ ] Occasional updates: We are planning on occasional updating this tool with no fixed schedule. Community involvement is encouraged through both issues and pull requests.
  - [ ] Unsupported: We are not currently supporting this code, but simply releasing it to the community AS IS but are not able to provide any guarantees of support. The community is welcome to submit issues, but you should not expect an active response.
 
-## Release Status
-GitHub's tags and Release features can be used to indicate a Release status.
+## Modules
+The package (`src/dynamic_foraging_processing`) is organized into the following
+modules:
 
- - Stable: v1.0.0 and above. Ready for production.
- - Beta:  v0.x.x or indicated in the tag. Ready for beta testers and early adopters.
- - Alpha: v0.x.x or indicated in the tag. Still in early development.
+### `raw_data_loader`
+Loads raw acquisition data via the data contract.
+ - **`RawDataLoader`** — wraps a `contraqctor` `Dataset` built from an acquisition
+   directory. Individual streams are accessed lazily through the tree-like contract
+   (`loader.dataset.at("Behavior").at("SoftwareEvents").at("TrialOutcome")`), or all
+   at once:
+   - `get_all_raw_data()` → `dict` mapping `parent.stream` to its loaded data
+     (`pandas.DataFrame` or `dict`).
+   - `raw_data_stream_descriptions` → `dict` mapping each stream to its description
+     (used when building NWB).
+
+### `nwb`
+Builds the NWB acquisition entries (requires the `nwb` extra).
+ - **`AcquisitionBuilder`** — assembles the acquisition module from the raw
+   streams: the four derived event series (`left`/`right_reward_delivery_time`,
+   `left`/`right_lick_time`) plus every raw contract stream packaged as a
+   `DynamicTable`. Returns `AcquisitionSeries` / `AcquisitionTable` models that a
+   writer translates into `pynwb` objects.
+
+### `processing`
+Builds derived NWB containers from the raw streams.
+ - **`TrialTableBuilder`** — assembles the NWB `trials` table from a `Dataset`.
+   `build()` returns one `pandas.DataFrame` row per trial, aligning the per-trial
+   software events, task-logic configuration, and Harp hardware streams. Pass
+   `raise_on_error=True` to fail loudly on missing/misaligned streams instead of
+   warning and continuing.
+ - **`TrialConfig`** (`processing.models.trial_config`) — a Pydantic model
+   describing one row of the trials table. Each field's `description` is the source
+   of truth for the corresponding NWB trial-column description; use
+   `TrialConfig.column_descriptions()` to retrieve them. The column-by-column source
+   mapping is documented in [`docs/trials_table_mapping.md`](docs/trials_table_mapping.md).
+
+### `qc`
+Assembles an `aind-data-schema` `QualityControl` object (requires the `qc` extra).
+ - **`RawQC`** / **`ProcessedQC`** — the raw (contract QA) and processed (behavior
+   metrics) QC stages. `build_quality_control(metrics)` collects their metrics into
+   a single `QualityControl`.
+
+### `pipeline`
+Ties the building blocks into two Code Ocean capsule entry points (requires the
+`full` extra).
+ - **`Pipeline`** — over a `RawDataLoader`:
+   - `run_nwb(output_path)` — assembles the NWB file (base metadata + acquisition
+     entries + trials table), writes it to `output_path / "behavior.nwb.zarr"`, and
+     writes a `processing.json` alongside it.
+   - `run_qc(output_path)` — runs the QC stages and writes `quality_control.json`.
+
+   Both entry points build and return their objects in memory; `output_path` is
+   optional and only written to when provided.
+
+## Repository structure
+```
+src/dynamic_foraging_processing/
+├── raw_data_loader/        # RawDataLoader: load raw streams via the data contract
+│   └── loader.py
+├── nwb/                    # NWB acquisition builder + models
+│   ├── acquisition/        # AcquisitionBuilder, AcquisitionSeries/Table
+│   └── utils.py            # clean_for_nwb: make frames safe to write to NWB
+├── processing/             # Derived NWB containers
+│   ├── _trial_table.py     # TrialTableBuilder
+│   └── models/
+│       └── trial_config.py # TrialConfig (one trials-table row + NWB descriptions)
+├── qc/                     # QualityControl assembly (raw + processed stages)
+└── pipeline/               # Pipeline: run_nwb / run_qc capsule entry points
+
+docs/                       # Source mapping and NWB notes (trials_table_mapping.md, ...)
+examples/                   # Runnable notebooks (raw_data_loader_example, trial_table_example)
+tests/                      # Unit tests
+```
+
+## Quickstart
+```python
+from pathlib import Path
+
+from dynamic_foraging_processing.raw_data_loader import RawDataLoader
+from dynamic_foraging_processing.pipeline import Pipeline
+
+# Point at the root of an acquisition directory.
+loader = RawDataLoader(path=Path("/path/to/acquisition"))
+pipeline = Pipeline(loader)
+
+# Write the NWB file (+ processing.json) and the QC (+ quality_control.json).
+pipeline.run_nwb("/path/to/output")
+pipeline.run_qc("/path/to/output")
+
+# Or use the building blocks directly.
+trials = pipeline.build_trials()   # NWB trials table, one row per trial
+raw = loader.get_all_raw_data()    # inspect raw streams
+```
+See [`examples/`](examples) for end-to-end notebooks.
 
 ## Installation
-To use the software, in the root directory, run
+This project uses [uv](https://docs.astral.sh/uv/). From the root directory, run
 ```bash
-pip install -e .
+uv sync
 ```
 
+> [!IMPORTANT]
+> The base install includes **only** the raw data loader. The optional modules
+> pull in their own dependencies, so importing them will fail on a base install:
+> - **`qc`** — `aind-data-schema`, `matplotlib` (for `dynamic_foraging_processing.qc`).
+> - **`nwb`** — `aind-nwb-utils`, `hdmf-zarr`, `pynwb` (for the `nwb` builder).
+> - **`full`** — everything above (`qc` + `nwb`); required by the `pipeline` module.
+>
+> Install the extra you need, e.g.:
+> ```bash
+> pip install -e ".[full]"
+> ```
+
 To develop the code, run
+to create the environment and install the package. To include the development
+dependencies (linting, tests, docs), run
 ```bash
-pip install -e . --group dev
+uv sync --group dev
 ```
-Note: --group flag is available only in pip versions >=25.1
 
-Alternatively, if using [uv](https://docs.astral.sh/uv/), run
+## Development
+Tests and style checks run on every pull request via
+`.github/workflows/test_and_lint.yml`. To run them locally with `uv`:
 ```bash
-uv sync
+uv run ruff check . && uv run ruff format --check .   # lint + format
+uv run interrogate -v .                               # docstring coverage
+uv run coverage run -m pytest && uv run coverage report
 ```
+To include the optional-module dependencies (qc + nwb) with uv, run
+```bash
+uv sync --extra full
+```
+
+## Release Status
+GitHub's tags and Release features can be used to indicate a Release status.
+
+ - Stable: v1.0.0 and above. Ready for production.
+ - Beta:  v0.x.x or indicated in the tag. Ready for beta testers and early adopters.
+ - Alpha: v0.x.x or indicated in the tag. Still in early development.

docs/nwb_contents.md

@@ -0,0 +1,55 @@
+# Final NWB Contents
+
+This document describes the contents of the NWB file produced by this
+repository. It is a companion to issue
+[#12](https://github.com/AllenNeuralDynamics/dynamic-foraging-processing/issues/12),
+which serves as the authoritative changelog for content decisions.
+
+When the NWB contents change, update both the [Changelog](#changelog) below
+and the relevant section in this document. Each entry should record at minimum
+the date, what changed, and why.
+
+## Acquisition
+
+The `acquisition` container holds the HARP streams from the rig (e.g.
+VR Foraging) along with four behavior-derived series carried over from the
+NWB produced by the combined dynamic foraging + FIP pipeline:
+
+- `left_lick_time`
+- `right_lick_time`
+- `left_reward_delivery_time`
+- `right_reward_delivery_time`
+
+Each series stores both timestamps and a parallel `data` array. For the
+reward delivery series, `data` annotates each reward as `earned`, `manual`,
+or `automatic`.
+
+See [`trials_table_mapping.md`](trials_table_mapping.md#acquisition-container)
+for the raw sources backing each of these four series.
+
+## Events
+
+The `events` container follows the conventions in
+[aind-physio-arch#1072](https://github.com/AllenNeuralDynamics/aind-physio-arch/issues/1072).
+
+The events sidecar will be version-controlled in this repository for now so that
+changes can be tracked alongside the code.
+
+Events are on pause pending validation by the HED team. See the
+[Changelog](#changelog) for details.
+
+## Trials
+
+The `trials` table is built from the raw acquisition streams. The full
+column-by-column mapping is documented in
+[`trials_table_mapping.md`](trials_table_mapping.md), and the source-of-truth
+discussion lives in issue
+[#5](https://github.com/AllenNeuralDynamics/dynamic-foraging-processing/issues/5).
+
+## Changelog
+
+| Date | Section | Change | Reason |
+| --- | --- | --- | --- |
+| 2026-06-03 | acquisition / trials | Initial scope confirmed: HARP streams + `{left,right}_lick_time` and `{left,right}_reward_delivery_time` in `acquisition`; trials mapping per issue #5. | Meeting with Alex. |
+| 2026-06-05 | events | Events on pause. | Pending validation by the HED team. |
+| 2026-06-08 | acquisition | Documented `data` arrays alongside timestamps; reward delivery series annotate each reward as `earned`, `manual`, or `automatic`. | Clarify acquisition contents. |