docs: add Senior Test Analyst candidate review

SENIOR_TEST_ANALYST_REVIEW.md

@@ -0,0 +1,224 @@
+# Senior Test Analyst — Candidate Review
+
+**Candidate:** Amarnath Rasineni (GitHub: `amar-python`)
+**Reviewed artefact:** `amar-python/PostgreDataMigrationApp` — *T&E Database Framework*
+**Reviewer:** Hiring panel technical reviewer
+**Date:** 2026-06-29
+
+> Scope note: this review is grounded in the concrete code, tests, and
+> documentation Amarnath has authored in this repository. It does not
+> substitute for a formal CV, reference checks, or right-to-work / security
+> clearance verification — see "Next steps" at the end.
+
+---
+
+## 1. Technical skills & experience review
+
+The repository is a self-built **Test & Evaluation (T&E) database framework**
+for Australian Defence-style acquisition programs. It is not a contrived demo
+— it carries a Verification Cross Reference Matrix, an ISM-aligned
+classification model, ASDEFCON-style vocabulary, and a working three-layer
+test architecture. Reviewing the diff and structure surfaces the following
+strengths against a Senior Test Analyst skill rubric.
+
+### 1.1 Test design & strategy
+
+| Capability | Evidence in the repository |
+|---|---|
+| Test architecture across layers | `ARCHITECTURE.md` defines a strict three-layer split: `build/` (ships), `tests/` (correctness coverage), `evals/` (data-driven black-box). BR-19 enforces this segregation. |
+| Test condition catalogue | `TEST_CONDITIONS.md` — every assertion enumerated in one place: 11 Python unit tests, ~140 SQL assertions across 5 suites, 23 Tier P / 1 Tier I / 1 Tier S eval scenarios, plus load-time verification queries. |
+| Risk-based tiering | Evals are explicitly tiered: **P** (offline, fast), **I** (idempotency, requires PG), **S** (full SQL suite integration) — runs degrade gracefully when an environment is unavailable (BR-17). |
+| Assertion library design | `tests/framework/test_framework.sql` defines `assert_equals`, `assert_row_count`, `assert_true`, `assert_raises`, etc., with results persisted to `test_run_results` — i.e. a purpose-built xUnit-style framework written in pure PL/pgSQL. |
+
+### 1.2 Requirements traceability (the Senior Test Analyst core skill)
+
+This is the strongest signal in the repository.
+
+- **`VCRM.md`** maps **22 business requirements (BR-01..BR-22)** to test
+  conditions across six verification layers (Python unit, SQL suite, Tier P/I/S
+  evals, load-time verification), with explicit verification methods (T / A /
+  I / D) per IEEE 1012 / MIL-STD conventions.
+- **`VCRM_GAPS.md`** records partial and deferred coverage candidly —
+  including BR-21 (cross-engine equivalence) and BR-22 (performance at scale)
+  flagged as deliberately out-of-scope rather than silently dropped.
+- The repo applies its own product (a VCRM for T&E programs) to itself —
+  a level of meta-discipline rarely seen in candidate portfolios.
+
+For a Senior Test Analyst working on Defence or regulated programs, this is
+the exact artefact reviewers expect to see produced.
+
+### 1.3 Test automation & engineering
+
+| Layer | Evidence |
+|---|---|
+| Python (`unittest` + `pytest`) | `tests/test_csv_validator.py`, `tests/test_evals_runner.py`, `tests/test_security.py`, `tests/test_parity.py`, `tests/test_regression.py`, `tests/test_snapshot.py`, `tests/test_e2e_pipeline.py`, `tests/test_csv_loader_arbitrary_shapes.py`, `tests/test_csv_utilise.py` — pytest markers (`@pytest.mark.unit`, `security`) and module-level docstrings. |
+| SQL test suites | 5 suites under `tests/suites/`, all using the project's assertion library, totalling 85+ logical assertions with a 100 % pass-rate gate (BR-20). |
+| Data-driven evals | `evals/runner.py` discovers scenarios under `evals/datasets/tier_*/` and diffs against `evals/expected/`, producing a machine-readable JSON report per run (BR-18). |
+| CI gating | `.github/workflows/python-validator-tests.yml` runs the validator suite on every push and PR on `windows-latest` — cross-platform discipline, not just Linux-happy-path. |
+| Local runners | `scripts/test.sh`, `scripts/test.ps1`, `scripts/run_qa.ps1`, `tests/run_tests.sh`, `tests/run_python_tests.ps1` — bash + PowerShell parity, recognising real-world Defence-tenant Windows environments. |
+
+### 1.4 Domain knowledge — T&E, Defence, regulated data
+
+| Domain element | Where it appears |
+|---|---|
+| ASDEFCON T&E vocabulary | `VCRM.md` opening note explicitly calls out ASDEFCON practice |
+| TEMP versioning & DT&E / AT&E / OT&E / IOT&E / LFT&E phases | BR-06, BR-14; enforced in `tests/suites/test_02_programs_phases.sql` |
+| VCRM 100 % coverage rule | BR-05; enforced in `tests/suites/test_03_requirements_vcrm.sql` |
+| Verdict & severity controlled vocabularies | BR-07, BR-08 (`pass/fail/blocked/not_run/inconclusive`; `critical/major/minor/observation`) |
+| Australian PSPF / ISM classification & clearance | BR-12 (`baseline/NV1/NV2/PV`), BR-13 (`UNCLASSIFIED/PROTECTED/SECRET/TOP SECRET`) |
+| Verification methods (T/A/I/D) | IEEE 1012 / MIL-STD convention used throughout the VCRM |
+
+### 1.5 Data quality & security
+
+- **`tests/test_security.py`** — static scans for hard-coded credentials and
+  unsafe f-string SQL across all Python + shell sources. Catches the very
+  defects a Senior Test Analyst is expected to find before they ship.
+- **Pre-deployment verification section** in `README.md` walks reviewers
+  through tool checks, secret-tracking checks, and connectivity checks.
+- **`build/csv/validator.py`** rejects malformed input, separates valid rows
+  from skipped rows with a `_skip_reason` per row (BR-10, BR-11) —
+  i.e. data quality as a designed-in capability, not a post-hoc audit.
+
+### 1.6 Communication & documentation
+
+`README.md`, `ARCHITECTURE.md`, `QUICKSTART.md`, `AZURE_DEPLOY.md`,
+`PROD_DEPLOY.md`, `RECONSTRUCT.md`, `VCRM.md`, `VCRM_GAPS.md`,
+`TEST_CONDITIONS.md`, and per-area READMEs in `scripts/` and `evals/`.
+Markdown is lint-gated (`.markdownlint.json`). The writing is structured for
+multiple audiences (T&E analysts, DBAs, DevOps, students) — explicitly
+called out in the README's "Who is this for?" table.
+
+### 1.7 Tooling & breadth
+
+| Area | Evidence |
+|---|---|
+| Languages | SQL (PL/pgSQL), Python 3.10+, Bash, PowerShell |
+| Databases | PostgreSQL primary; adapters for MariaDB, MySQL, SQLite, InfluxDB, Redis, Teradata under `build/adapters/` and `build/schema/` |
+| Cloud / IaC | Terraform (`build/terraform-github-repos/`), Azure deployment guide |
+| CI/CD | GitHub Actions, lint scripts (`scripts/lint.sh`, `lint_diff.sh`), preflight scripts (`preflight.sh` / `.ps1`) |
+| Test types covered | unit, integration (Tier I), end-to-end (`test_e2e_pipeline.py`), regression, snapshot, parity, security, data-driven evals |
+
+---
+
+## 2. Alignment with Senior Test Analyst role & contract expectations
+
+A typical Senior Test Analyst position description — particularly in
+Australian Defence / Government / regulated industry — asks for the
+capabilities below. Each row is rated against the repository evidence above.
+
+| Capability the role expects | Evidence rating | Notes |
+|---|---|---|
+| Develop test strategies and test plans | **Strong** | Three-layer architecture; explicit risk tiers; documented in `ARCHITECTURE.md` and `evals/PLAN.md`. |
+| Author and execute test cases | **Strong** | 85+ SQL assertions + 11 Python tests; runnable from a single command (BR-16). |
+| Requirements traceability (VCRM / RTM) | **Strong** | Full VCRM with explicit verification methods (T/A/I/D) and per-requirement layer coverage. Few candidates produce this unprompted. |
+| Defect lifecycle understanding | **Strong** | DR lifecycle modelled (BR-08) — severity, `resolved_at`, link to failing result — and asserted in `test_04_execution_defects.sql`. |
+| Test data management | **Strong** | `te_seed_data.sql`, `scripts/insert_random_test_data.sql`, CSV validator with split-output design, eval datasets versioned alongside expected outputs. |
+| Automation in CI/CD | **Strong** | GitHub Actions wired; cross-platform; deterministic pass/fail gateable. |
+| SQL proficiency | **Strong** | Non-trivial PL/pgSQL — assertion framework, parameterised schemas, constraint-violation testing via `assert_raises`. |
+| Defence / regulated domain literacy | **Strong** | ISM classifications, PSPF clearance levels, ASDEFCON vocabulary, IEEE 1012 methods. |
+| Gap analysis & honest reporting | **Strong** | `VCRM_GAPS.md` and `evals/FAILURE_MODES.md` candidly document partial coverage and deferred items rather than overstating completion. |
+| Stakeholder communication | **Strong (artefact)** / **Unverified (interpersonal)** | Documentation is clearly written for diverse audiences. Verbal stakeholder skill needs interview confirmation. |
+| Performance / load testing | **Partial** | BR-22 (performance at scale) is explicitly out of scope. If the role requires this, probe at interview. |
+| Test management tooling (JIRA, Xray, qTest, Azure DevOps Test Plans) | **Not evidenced** | No tool-specific artefacts visible. Likely transferable but should be confirmed. |
+| Formal certifications (ISTQB Foundation / Advanced) | **Not evidenced** | Not visible from this repo; ask for proof in pre-interview pack. |
+| Security clearance | **Indicated, not verified** | Model uses Australian clearance vocabulary (`baseline/NV1/NV2/PV`), suggesting familiarity. Actual clearance status must be confirmed with the candidate. |
+
+### Contract-mode considerations
+
+If the engagement is a fixed-term contract or labour-hire arrangement
+(common for Defence Senior Test Analyst roles):
+
+- **Self-direction.** The repo is sustained solo across many feature branches
+  (`csv-loading-postgres-migration-qvkl4l`, this review branch, etc.) with
+  merged PRs and squashed history — consistent with a contractor who can
+  operate without daily supervision.
+- **Tooling autonomy.** Provides bash + PowerShell, Linux + Windows runners,
+  Terraform-managed infra — comfortable owning the build chain end-to-end.
+- **Day-1 productivity.** A candidate who has already built a VCRM-based
+  test pipeline can be put on a real VCRM workstream from day one with
+  minimal ramp-up.
+
+### Risks / things to probe
+
+1. **Team scale.** The portfolio shows solo delivery. Confirm experience
+   leading testing in a multi-tester squad and coaching juniors.
+2. **Live programme exposure.** Verify recent paid engagements on an active
+   T&E / acquisition programme, not just personal projects.
+3. **Test management tool familiarity.** No JIRA/Xray/qTest/Azure DevOps
+   Test Plans evidence — confirm.
+4. **Performance & non-functional testing depth.** BR-22 is deferred; ask
+   about load, soak, and stress testing experience.
+5. **Clearance status.** Establish current clearance level and sponsorship
+   needs early — gating for most Defence roles.
+
+---
+
+## 3. Next steps in the recruitment process
+
+Suggested sequence for the hiring manager / recruiter:
+
+1. **Pre-screen pack (within 2 business days).** Request from the candidate:
+   - Current CV with role dates and clearance status (current level, sponsor,
+     expiry).
+   - Evidence of formal certifications (ISTQB, Defence inductions).
+   - 2–3 referees from recent paid engagements.
+   - Confirmation of right-to-work / contracting entity (PTY / sole trader /
+     payroll preference) and target rate.
+
+2. **Technical screen (45–60 min, video).** Walk the candidate through this
+   repository as the live artefact. Suggested prompts:
+   - "Talk us through the VCRM in `VCRM.md`. How would you adapt this to
+     trace a real customer's acceptance criteria?"
+   - "Pick one gap from `VCRM_GAPS.md` — how would you close it and what
+     would the test condition look like?"
+   - "Your SQL suite asserts 85 conditions. What did you choose *not* to
+     assert, and why?"
+   - "Walk through how Tier P / I / S evals degrade when PostgreSQL isn't
+     available, and how that decision was made."
+
+3. **Practical exercise (timeboxed, ≤ 2 hours).** Provide a short,
+   anonymised real requirement and ask the candidate to:
+   - Draft a test plan section.
+   - Identify two test conditions and write them as `assert_*` calls.
+   - Map them in a one-line VCRM extension.
+   This validates that the artefacts in the repo reflect the candidate's
+   own day-to-day approach.
+
+4. **Panel interview (60–75 min).** Cover:
+   - Stakeholder management (BA, dev lead, programme manager).
+   - Coaching/mentoring of junior testers.
+   - Handling defect triage disputes.
+   - Behaviour under shifting acceptance criteria and compressed timelines.
+   - Live programme exposure beyond personal projects (see risks above).
+
+5. **Reference checks (2 referees minimum).** Specifically probe:
+   - Quality of test artefacts shipped under deadline pressure.
+   - Independence vs. team integration.
+   - Communication with non-technical stakeholders (programme leadership).
+
+6. **Clearance & compliance gate.** Before offer:
+   - Confirm clearance level and currency against the role's classification.
+   - Confirm sponsorship transferability if applicable.
+   - Confirm any conflicts (current contracts, exclusivity clauses).
+
+7. **Offer & contract.** On a positive panel outcome:
+   - Issue contract specifying scope (test strategy, automation, VCRM
+     ownership), reporting line, and deliverables cadence.
+   - Schedule a 30 / 60 / 90-day plan keyed to the first VCRM iteration so
+     value is visible early.
+
+### Indicative recommendation
+
+Based on the repository evidence alone, the candidate presents as a
+**strong technical match for a Senior Test Analyst role in a T&E,
+Defence, or regulated-data context**. The portfolio shows test
+architecture maturity, requirements-traceability discipline, and
+domain-correct vocabulary well above the level of a mid-level analyst.
+
+Proceed to **technical screen** as the next concrete action; complete the
+clearance and references gates in parallel.
+
+---
+
+*This review is an internal recruitment artefact. Distribute according to
+your organisation's candidate-privacy policy.*