Feat/python replication

[NKeleher]

Pull Request Summary 🚀

What does this PR do? 📝

Adds Parquet exports, a data-dict.yaml data dictionary, and a full Python/Polars
replication pipeline to the replication package export, alongside a Markdown
README.md and Parquet support for local file import.

The exported zip now includes:

• Parquet copies of the raw, prepped, and corrected datasets
  (3_data/{1_raw,2_intermediate,3_final}/*.parquet), so the package is usable
  without Stata.
• 1_docs/2_codebooks/data-dict.yaml, a data dictionary compliant with the
  data-dict.yaml spec (column types, ranges,
  enums, and a primary_key constraint on the key column).
• Python replication scripts (2_scripts/{0_main,3_prepare_data,4_corrections}.py)
  that reproduce the prep/correction pipeline using Polars — no Stata required.
  Each script carries a PEP 723 inline-metadata header, so it's independently
  runnable via uv run <script>.py, which provisions its dependencies
  automatically (no separate install-packages step needed).
• README.md (was 0_README.txt) — the package README is now real Markdown
  (headings, fenced code blocks, tables) instead of a plain-text banner report.
• Parquet import support in the local-file connector
  (src/datasure/connectors/local.py), alongside the existing csv/xlsx/xls/json/dta
  types.

Why is this change needed? 🤔

The replication package previously only supported Stata: raw data as CSV, and
do-files that require Stata to run. Researchers without a Stata license had no
way to reproduce a project's corrected dataset, and there was no machine-readable
data dictionary for downstream tooling. This PR makes the package
language-agnostic (Stata and Python) and adds a standard data dictionary, all
without changing the existing Stata workflow.

How was this implemented? 🛠️

• replication/data_dict.py (new) generates the data dictionary from the
  corrected dataset, mirroring codebook.py's dtype classification.
• replication/py_script_generators.py and replication/py_prep_script_generator.py
  (new) translate the prep log and correction log into Polars code, mirroring
  the real semantics in processing/prep.py / processing/corrections.py
  rather than approximating them the way the Stata do-file translation does —
  in some cases this makes the Python output more complete than Stata's (e.g.
  get dummies and uuid translate directly instead of falling back to a
  "could not translate" note).
• No Python import step: earlier drafts included a 2_import_data.py that
  re-derived column types from the raw CSV via pl.read_csv(...). Since
  package_builder.py already bundles an authoritative _raw.parquet written
  straight from the real DuckDB-stored data, that re-inference was both
  redundant and a correctness risk (it could disagree with the true schema).
  It was removed; 3_prepare_data.py reads the bundled Parquet directly. The
  CSV round-trip is kept only on the Stata side, since Stata can't read Parquet.
• readme.py and replication_view.py's "What's inside the zip?" preview were
  updated to describe the new file layout accurately.
• Addressed two Copilot review comments: _to_parquet_bytes no longer writes an
  invalid 0-byte Parquet file for a columnless DataFrame, and the Python master
  script invokes each step via uv run (not sys.executable) so a step's own
  PEP 723 dependencies are honored.
• Merged ux-review into this branch (no conflicts) to pick up the shared UI
  helpers it introduced.
• Filed #246 for an
  unrelated, pre-existing Stata do-file bug found while porting the row-removal
  logic (equal_to/not_equal_to conditions may invert direction vs. the real
  app) — flagged for a separate fix, not touched here.

How to test or reproduce? 🧪

uv sync
uv run python -m pytest tests/replication/ tests/connectors/test_local.py tests/views/test_replication_view.py -q
just lint-py && just fmt-python

Manually: uv run datasure → Replication Package Export → select a page → expand
"What's inside the zip?" to see the updated contents → build the package →
unzip it and confirm data-dict.yaml, the .parquet files, and README.md are
present, then run uv run 0_main.py from 2_scripts/ to reproduce the prepped
and corrected datasets without Stata.

Screenshots (if applicable) 📷

Exports now look like:

Checklist ✅

• I have run and tested my changes locally
• I have limit this PR to less than 1000 lines of code change (if not, explain why)
  • Exceeds 1000 lines: two new generator modules mirror the existing Stata
    generators function-for-function, plus a data dictionary generator, each
    with matching test coverage — the line count is mostly parallel structure
    and tests, not one dense change.
• I have updated/added tests to cover my changes (if applicable)
• I have updated/added requirements to cover my changes (if applicable)
  • Added pyyaml for data-dict.yaml generation.
• I have run linting and formatting on any code changes (if applicable)
• I have updated the documentation (README, etc.) accordingly
• I have reviewed and resolved any merge conflict

Reviewer Emoji Legend

	:code:	Meaning
😃👍💯	:smiley: :+1: :100:	I like this... ...and I want the author to know it! This is a way to highlight positive parts of a code review.
⭐⭐⭐	:star: :star: :star:	Important to fix before PR can be approved... And I am providing reasons why it needs to be addressed as well as suggested improvements.
⭐⭐	:star: :star:	Important to fix but non-blocking for PR approval... And I am providing suggestions where it could be improved either in this PR or later.
⭐	:star:	Give this some thought but non-blocking for PR approval... ...and consider this a suggestion, not a requirement.
❓	:question:	I have a question. This should be a fully formed question with sufficient information and context that requires a response.
📝	:memo:	This is an explanatory note, fun fact, or relevant commentary that does not require any action.
⛏	:pick:	This is a nitpick. This does not require any changes and is often better left unsaid. This may include stylistic, formatting, or organization suggestions and should likely be prevented/enforced by linting if they really matter
♻️	:recycle:	Suggestion for refactoring. Should include enough context to be actionable and not be considered a nitpick.

[Copilot]

Pull request overview

Adds Python-based replication outputs to DataSure’s replication package generation, alongside Parquet dataset exports and a data-dict.yaml data dictionary, to improve reproducibility without requiring Stata.

Changes:

• Generate Python/Polars replication scripts (0_main.py, import, prep, corrections) with PEP 723 inline metadata for uv run.
• Add Parquet exports (raw/prepped/corrected) and a data-dict.yaml file into the replication zip.
• Extend/adjust replication README content and add/expand tests covering the new artifacts.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
uv.lock	Locks new dependency (pyyaml) and updates resolution markers from re-lock.
pyproject.toml	Adds pyyaml runtime dependency for YAML generation/parsing.
CLAUDE.md	Updates repository module map to include new replication modules.
src/datasure/replication/package_builder.py	Adds Parquet + data-dict.yaml outputs and writes Python replication scripts into the zip.
src/datasure/replication/readme.py	Updates replication README to describe Python workflow, Parquet artifacts, and data dictionary.
src/datasure/replication/data_dict.py	New generator for data-dict.yaml from a Polars DataFrame.
src/datasure/replication/py_script_generators.py	New generators for Python master/import/corrections replication scripts (PEP 723).
src/datasure/replication/py_prep_script_generator.py	New generator translating prep logs into a Python/Polars prepare script.
tests/replication/test_readme.py	Adds assertions for new README references (Python scripts, Parquet, data-dict.yaml).
tests/replication/test_package_builder.py	Adds zip-content tests for data-dict.yaml, Parquet, and Python scripts.
tests/replication/test_data_dict.py	New tests for data-dict.yaml structure and column typing/constraints.
tests/replication/test_py_script_generators.py	New tests validating generated Python scripts parse and contain expected content.
tests/replication/test_py_prep_script_generator.py	New tests for prep-log-to-Python translation logic and script validity.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[NKeleher]

For review after #244.