Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@ All notable changes to this project are documented in this file.

## Unreleased

### Breaking Changes
- **`source.url` is now a list of URLs (`url: list[HttpUrl]`).** A table-config section may declare one or more remote source URLs, all recorded as provenance (emitted in the edge `source_record_urls` list and the RIG). The legacy scalar form `url: https://example.com/x.tsv` is no longer accepted — wrap it in a list. Update existing configs from `url: https://...` to a sequence:

```yaml
source:
url:
- https://example.com/data.tsv
```

### Added
- **`tablassert build-fullmap --aria2c` / `-a`** opt-in downloader acceleration. When requested, the BABEL download stage uses the installed `aria2c` executable with segmented HTTP downloads plus resume/retry flags (`--continue=true`, `--max-tries`, `--retry-wait`) while keeping the existing Python downloader as the default. Missing or failing `aria2c` fails loud instead of silently falling back, and aria2 `.aria2` control files are preserved so interrupted downloads can resume on rerun.

Expand Down
9 changes: 6 additions & 3 deletions docs/configuration/advanced-example.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ template:
source:
kind: excel
local: ./DATALAKE/ALAM.XLSX
url: https://pmc.ncbi.nlm.nih.gov/articles/instance/11708054/bin/mbio.01679-24-s0006.xlsx
url:
- https://pmc.ncbi.nlm.nih.gov/articles/instance/11708054/bin/mbio.01679-24-s0006.xlsx
row_slice:
- 2 # Skip the first two rows (title + header)
- auto # Read to end
Expand Down Expand Up @@ -174,7 +175,8 @@ correlation tables linking two biological entities per row (e.g., metabolite ↔
template:
source:
kind: excel
url: https://pmc.ncbi.nlm.nih.gov/articles/instance/example/bin/data.xlsx
url:
- https://pmc.ncbi.nlm.nih.gov/articles/instance/example/bin/data.xlsx
local: ./DATALAKE/AVUTHU1.xlsx
sheet: signif_metab_microb_corre
row_slice: [2, auto]
Expand Down Expand Up @@ -221,7 +223,8 @@ the `object` (and optionally `row_slice`) per section — one section entry per
template:
source:
kind: excel
url: https://pmc.ncbi.nlm.nih.gov/articles/instance/example/bin/data.xlsx
url:
- https://pmc.ncbi.nlm.nih.gov/articles/instance/example/bin/data.xlsx
local: ./DATALAKE/BLANTON1.xlsx
sheet: Sheet1
row_slice: [2, auto]
Expand Down
10 changes: 6 additions & 4 deletions docs/configuration/table.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Defines the data file location and format.
|-------|------|----------|-------------|
| `kind` | String | No | Source kind. Model default is `"excel"`, but specify it explicitly in configs. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here — Tablassert does not download it. |
| `url` | URL | Yes | Source URL recorded as provenance (emitted as the edge `source_record_urls` column and in the RIG). Format-validated only; not fetched. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the edge `source_record_urls` list and in the RIG). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. |
| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
Expand All @@ -101,7 +101,8 @@ Defines the data file location and format.
source:
kind: excel
local: ./data/mydata.xlsx
url: https://example.com/data.xlsx
url:
- https://example.com/data.xlsx
sheet: "Sheet1"
row_slice: [1, auto] # Start at the second physical row, read to end
```
Expand All @@ -114,7 +115,7 @@ source:
|-------|------|----------|-------------|
| `kind` | String | No | Source kind. Model default is `"text"`, but specify it explicitly in configs. |
| `local` | Path | Yes | Local file path the source is read from. The file must already exist here — Tablassert does not download it. |
| `url` | URL | Yes | Source URL recorded as provenance (emitted as the edge `source_record_urls` column and in the RIG). Format-validated only; not fetched. |
| `url` | List[URL] | Yes | One or more source URLs recorded as provenance (emitted as the edge `source_record_urls` list and in the RIG). At least one URL is required; supply multiple to back a single section with several links. Format-validated only; not fetched. |
| `delimiter` | String | No | Field delimiter. Defaults to `","`. |
| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
Expand All @@ -125,7 +126,8 @@ source:
source:
kind: text
local: ./data/mydata.tsv
url: https://example.com/data.tsv
url:
- https://example.com/data.tsv
delimiter: "\t"
row_slice: [1, auto]
```
Expand Down
18 changes: 12 additions & 6 deletions docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ template:
source:
kind: text
local: ./gene-disease.csv
url: https://example.com/gene-disease.csv
url:
- https://example.com/gene-disease.csv
row_slice: [1, auto]
delimiter: ","
statement:
Expand Down Expand Up @@ -57,7 +58,8 @@ template:
source:
kind: text
local: ./drug-targets.tsv
url: https://example.com/drug-targets.tsv
url:
- https://example.com/drug-targets.tsv
row_slice: [1, auto]
delimiter: "\t"
statement:
Expand Down Expand Up @@ -102,7 +104,8 @@ template:
source:
kind: excel
local: ./microbiome-correlations.xlsx
url: https://example.com/microbiome-data.xlsx
url:
- https://example.com/microbiome-data.xlsx
sheet: correlations
row_slice: [2, auto]
statement:
Expand Down Expand Up @@ -161,7 +164,8 @@ template:
source:
kind: text
local: ./gene-pathways.csv
url: https://example.com/gene-pathways.csv
url:
- https://example.com/gene-pathways.csv
row_slice: [1, auto]
delimiter: ","
statement:
Expand Down Expand Up @@ -220,7 +224,8 @@ template:
source:
kind: text
local: ./significant-associations.csv
url: https://example.com/associations.csv
url:
- https://example.com/associations.csv
row_slice: [1, auto]
delimiter: ","
reindex:
Expand Down Expand Up @@ -268,7 +273,8 @@ template:
source:
kind: text
local: ./hierarchical-data.csv
url: https://example.com/hierarchical.csv
url:
- https://example.com/hierarchical.csv
row_slice: [1, auto]
delimiter: ","
statement:
Expand Down
3 changes: 2 additions & 1 deletion docs/examples/tutorial-table.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ template:
source:
kind: text
local: ./docs/examples/tutorial-data.csv
url: https://example.com/data.csv
url:
- https://example.com/data.csv
row_slice:
- 1
- auto
Expand Down
3 changes: 2 additions & 1 deletion docs/tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ template:
source:
kind: text
local: ./docs/examples/tutorial-data.csv
url: https://example.com/data.csv
url:
- https://example.com/data.csv
row_slice:
- 1
- auto
Expand Down
50 changes: 30 additions & 20 deletions examples/agent/QC_REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,16 @@ State dir: `.tablassert/qc-assay` · PMCs assayed: 10
- **object:** method=column encoding=D prioritize=['Gene', 'Protein']
- **source:** kind=excel sheet='Supp.Table 2A_cluster-1' local=<state-dir>/downloads/PMC8017771/PMC8017771.1/NIHMS1644812-supplement-1644812_Supp_Tab2.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC8017771', 'knowledge_level': 'statistical_association', 'agent_type': 'data_analysis_pipeline'}
- **config sha256:** `fb96e79edf16`
- **config sha256:** `676109938e71`

### Derived config (`configs/PMC8017771.yaml`)

```yaml
source:
kind: excel
local: <state-dir>/downloads/PMC8017771/PMC8017771.1/NIHMS1644812-supplement-1644812_Supp_Tab2.xlsx
url: https://pmc-oa-opendata.s3.amazonaws.com/PMC8017771.1/NIHMS1644812-supplement-1644812_Supp_Tab2.xlsx
url:
- https://pmc-oa-opendata.s3.amazonaws.com/PMC8017771.1/NIHMS1644812-supplement-1644812_Supp_Tab2.xlsx
Comment thread
coderabbitai[bot] marked this conversation as resolved.
sheet: "Supp.Table 2A_cluster-1"
row_slice: [2, "auto"]
reindex:
Expand Down Expand Up @@ -91,7 +92,7 @@ annotations:
- **object:** method=value encoding=MONDO:0007739 prioritize=None
- **source:** kind=excel sheet='Cap Score - Ion Level' local=<state-dir>/downloads/PMC13161869/PMC13161869.1/ACN3-13-911-s001.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC13161869'}
- **config sha256:** `1aa5a8bf5ded`
- **config sha256:** `a316ead5d5ab`

### Derived config (`configs/PMC13161869.yaml`)

Expand All @@ -104,7 +105,8 @@ sections:
- source:
kind: excel
local: "<state-dir>/downloads/PMC13161869/PMC13161869.1/ACN3-13-911-s001.xlsx"
url: "https://pmc-oa-opendata.s3.amazonaws.com/PMC13161869.1/ACN3-13-911-s001.xlsx"
url:
- "https://pmc-oa-opendata.s3.amazonaws.com/PMC13161869.1/ACN3-13-911-s001.xlsx"
sheet: "Cap Score - Ion Level"
row_slice: [1, "auto"]
reindex:
Expand Down Expand Up @@ -192,7 +194,7 @@ sections:
- **object:** method=column encoding=B prioritize=['Cell']
- **source:** kind=excel sheet='Supp. Table 7' local=<state-dir>/downloads/PMC12900646/PMC12900646.1/41588_2025_2486_MOESM4_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC12900646'}
- **config sha256:** `560d11724293`
- **config sha256:** `c8745021082f`

### Derived config (`configs/PMC12900646.yaml`)

Expand All @@ -205,7 +207,8 @@ sections:
- source:
kind: excel
local: <state-dir>/downloads/PMC12900646/PMC12900646.1/41588_2025_2486_MOESM4_ESM.xlsx
url: "https://pmc-oa-opendata.s3.amazonaws.com/PMC12900646.1/41588_2025_2486_MOESM4_ESM.xlsx"
url:
- "https://pmc-oa-opendata.s3.amazonaws.com/PMC12900646.1/41588_2025_2486_MOESM4_ESM.xlsx"
sheet: "Supp. Table 7"
row_slice: [7, "auto"]
reindex:
Expand Down Expand Up @@ -286,15 +289,16 @@ sections:
- **object:** method=value encoding=MONDO:0004988 prioritize=None
- **source:** kind=excel sheet='Percentiles - 16p11.2' local=<state-dir>/downloads/PMC9187732/PMC9187732.1/41467_2022_30968_MOESM16_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC9187732'}
- **config sha256:** `e8cb8eacc72d`
- **config sha256:** `468029c6b43d`

### Derived config (`configs/PMC9187732.yaml`)

```yaml
source:
kind: excel
local: <state-dir>/downloads/PMC9187732/PMC9187732.1/41467_2022_30968_MOESM16_ESM.xlsx
url: https://pmc-oa-opendata.s3.amazonaws.com/PMC9187732.1/41467_2022_30968_MOESM16_ESM.xlsx
url:
- https://pmc-oa-opendata.s3.amazonaws.com/PMC9187732.1/41467_2022_30968_MOESM16_ESM.xlsx
sheet: "Percentiles - 16p11.2"
reindex:
- column: A
Expand Down Expand Up @@ -342,7 +346,7 @@ provenance:
- **object:** method=column encoding=A prioritize=['BiologicalProcess']
- **source:** kind=excel sheet='Supplementary Table 7' local=<state-dir>/downloads/PMC13099431/PMC13099431.1/41591_2026_4228_MOESM2_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC13099431'}
- **config sha256:** `076cdc335924`
- **config sha256:** `90366dcd5469`

### Derived config (`configs/PMC13099431.yaml`)

Expand All @@ -355,7 +359,8 @@ sections:
- source:
kind: excel
local: <state-dir>/downloads/PMC13099431/PMC13099431.1/41591_2026_4228_MOESM2_ESM.xlsx
url: "https://pmc-oa-opendata.s3.amazonaws.com/PMC13099431.1/41591_2026_4228_MOESM2_ESM.xlsx"
url:
- "https://pmc-oa-opendata.s3.amazonaws.com/PMC13099431.1/41591_2026_4228_MOESM2_ESM.xlsx"
sheet: "Supplementary Table 7"
row_slice: [2, "auto"]
statement:
Expand Down Expand Up @@ -400,15 +405,16 @@ sections:
- **object:** method=value encoding=GO:0008380 prioritize=None
- **source:** kind=excel sheet='vU1-8 KO v WT' local=<state-dir>/downloads/PMC13172311/PMC13172311.1/41467_2026_73121_MOESM5_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC13172311'}
- **config sha256:** `af8d05069765`
- **config sha256:** `e1d3542f1fda`

### Derived config (`configs/PMC13172311.yaml`)

```yaml
source:
kind: excel
local: <state-dir>/downloads/PMC13172311/PMC13172311.1/41467_2026_73121_MOESM5_ESM.xlsx
url: https://pmc-oa-opendata.s3.amazonaws.com/PMC13172311.1/41467_2026_73121_MOESM5_ESM.xlsx
url:
- https://pmc-oa-opendata.s3.amazonaws.com/PMC13172311.1/41467_2026_73121_MOESM5_ESM.xlsx
sheet: vU1-8 KO v WT
statement:
subject:
Expand Down Expand Up @@ -453,7 +459,7 @@ provenance:
- **object:** method=value encoding=NCBITaxon:4530 prioritize=None
- **source:** kind=excel sheet='Map' local=<state-dir>/downloads/PMC12906585/PMC12906585.1/122_2026_5178_MOESM1_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC12906585'}
- **config sha256:** `d80ab3fe0ca0`
- **config sha256:** `6d5d4b88c16b`

### Derived config (`configs/PMC12906585.yaml`)

Expand All @@ -466,7 +472,8 @@ sections:
- source:
kind: excel
local: <state-dir>/downloads/PMC12906585/PMC12906585.1/122_2026_5178_MOESM1_ESM.xlsx
url: "https://pmc-oa-opendata.s3.amazonaws.com/PMC12906585.1/122_2026_5178_MOESM1_ESM.xlsx"
url:
- "https://pmc-oa-opendata.s3.amazonaws.com/PMC12906585.1/122_2026_5178_MOESM1_ESM.xlsx"
sheet: Map
statement:
subject:
Expand Down Expand Up @@ -516,7 +523,7 @@ sections:
- **object:** method=column encoding=F prioritize=['Pathway', 'BiologicalProcess']
- **source:** kind=excel sheet='SD15' local=<state-dir>/downloads/PMC13172025/PMC13172025.1/42003_2026_10045_MOESM3_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC13172025'}
- **config sha256:** `86448dd76087`
- **config sha256:** `ac9f8d96c81a`

### Derived config (`configs/PMC13172025.yaml`)

Expand All @@ -529,7 +536,8 @@ sections:
- source:
kind: excel
local: <state-dir>/downloads/PMC13172025/PMC13172025.1/42003_2026_10045_MOESM3_ESM.xlsx
url: "https://pmc-oa-opendata.s3.amazonaws.com/PMC13172025.1/42003_2026_10045_MOESM3_ESM.xlsx"
url:
- "https://pmc-oa-opendata.s3.amazonaws.com/PMC13172025.1/42003_2026_10045_MOESM3_ESM.xlsx"
sheet: "SD15"
row_slice: [2, "auto"]
statement:
Expand Down Expand Up @@ -574,7 +582,7 @@ sections:
- **object:** method=column encoding=A prioritize=['AnatomicalEntity', 'GrossAnatomicalStructure']
- **source:** kind=excel sheet='v68.lvedv.twas.alltissues' local=<state-dir>/downloads/PMC7206184/PMC7206184.1/41467_2020_15823_MOESM9_ESM.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC7206184', 'knowledge_level': 'statistical_association', 'agent_type': 'data_analysis_pipeline'}
- **config sha256:** `d6785bc724be`
- **config sha256:** `cedf7b1f7445`

### Derived config (`configs/PMC7206184.yaml`)

Expand All @@ -589,7 +597,8 @@ sections:
- source:
kind: excel
local: <state-dir>/downloads/PMC7206184/PMC7206184.1/41467_2020_15823_MOESM9_ESM.xlsx
url: "https://pmc-oa-opendata.s3.amazonaws.com/PMC7206184.1/41467_2020_15823_MOESM9_ESM.xlsx"
url:
- "https://pmc-oa-opendata.s3.amazonaws.com/PMC7206184.1/41467_2020_15823_MOESM9_ESM.xlsx"
sheet: "v68.lvedv.twas.alltissues"
row_slice: [1, "auto"]
statement:
Expand Down Expand Up @@ -635,15 +644,16 @@ sections:
- **object:** method=value encoding=MONDO:0004992 prioritize=None
- **source:** kind=excel sheet='Table_S7' local=<state-dir>/downloads/PMC11947420/PMC11947420.1/mmc2.xlsx
- **provenance:** {'repo': 'PMC', 'publication': 'PMC11947420'}
- **config sha256:** `8a1c74d4efc6`
- **config sha256:** `90413a33f8a9`

### Derived config (`configs/PMC11947420.yaml`)

```yaml
source:
kind: excel
local: <state-dir>/downloads/PMC11947420/PMC11947420.1/mmc2.xlsx
url: https://pmc-oa-opendata.s3.amazonaws.com/PMC11947420.1/mmc2.xlsx
url:
- https://pmc-oa-opendata.s3.amazonaws.com/PMC11947420.1/mmc2.xlsx
sheet: Table_S7
reindex:
- column: A
Expand Down
Loading
Loading