feat: add detector benchmark script

[sanmaxdev]

Summary

• Add a detector latency, throughput, and peak-memory benchmark script for reusable local measurements.
• Support built-in synthetic context-length sweeps (~512/2k/8k tokens), JSONL benchmark cases, warmup runs, repeated measurements, and JSON output.
• Record device, aggregate metrics, per-case metrics, context lengths, and CPU/GPU peak-memory source in the result record.
• Cover benchmark timing, context-length generation, memory reporting, device validation, and input validation in dedicated benchmark tests.

Testing

• python3 -m pytest tests/test_inference_pytest.py tests/test_benchmark_detectors_pytest.py -v -k "not TestAnswerStartToken"

• python3 -m ruff format --check --diff lettucedetect/ tests/ scripts/benchmark_detectors.py

• python3 -m ruff check lettucedetect/ tests/ scripts/benchmark_detectors.py --extend-exclude lettucedetect/integrations/

• python3 scripts/benchmark_detectors.py --help

• I certify that I have the right to submit this code and that it may be distributed under the repository's MIT license

Closes #58

[adaamko]

Thanks — the harness itself is sound (warmup, perf_counter, mean/median/p95, JSON output, no new deps), but as submitted it would auto-close #58 while delivering only part of it:

1. No peak-memory reporting — explicitly in the issue's task list. torch.cuda.max_memory_allocated() on GPU (reset before the measured loop) and resource.getrusage(RUSAGE_SELF).ru_maxrss on CPU, added to the result record, would cover it.
2. No context-length dimension — the built-in case is one short sentence, and there's no built-in way to produce the ~512/2k/8k sweep the issue asks for. Generating synthetic cases at target token lengths (or shipping case files) would keep it reproducible without hand-built JSONL.
3. No device in the output — results aren't interpretable across the CPU-vs-GPU axis without recording which device ran.

If you'd rather land this as a first step, change "Closes #58" to "Partially addresses #58" and I'm open to that too — but then the issue stays open.

Nits, non-blocking: with the default --repeats 5, p95 is just the max — consider a higher default or a note in --help; and the script-loading tests might live better in their own tests/test_benchmark_detectors.py.

[adaamko]

Heads-up: the repo now requires a contribution-rights attestation on PRs (new required CI check). When you push updates, please add this line to your PR description and tick it:

- [x] I certify that I have the right to submit this code and that it may be distributed under the repository's MIT license

[sanmaxdev]

Thanks, addressed the requested benchmark coverage:

• added peak memory reporting for CUDA and CPU
• added device to the benchmark result
• added built-in synthetic context sweeps for ~512/2k/8k tokens
• added per-case metrics so context length remains visible in JSON output
• moved benchmark tests into tests/test_benchmark_detectors_pytest.py
• rebased on latest main to clear the dirty merge state

Verified locally with:

• python3 -m pytest tests/test_inference_pytest.py tests/test_benchmark_detectors_pytest.py -v -k "not TestAnswerStartToken"
• python3 -m ruff format --check --diff lettucedetect/ tests/ scripts/benchmark_detectors.py
• python3 -m ruff check lettucedetect/ tests/ scripts/benchmark_detectors.py --extend-exclude lettucedetect/integrations/
• python3 scripts/benchmark_detectors.py --help

[adaamko]

Thanks for the thorough second round — all three blockers are properly fixed: the CUDA peak-memory reset/read placement is correct, the Darwin/Linux ru_maxrss unit difference is handled, device is resolved and recorded, and the synthetic 512/2k/8k sweep with per-case stats covers the issue.

One small ask before merge: the plain (non---json) output prints only the aggregate summary line, but the per-case latency-vs-context-length numbers are the story #58 is after — please print one line per case too (the "small table" from the issue).

Optional nit, non-blocking: a note in --help that context lengths are counted in whitespace tokens, so the 8192 case exceeds ModernBERT's 4k subword window and partly measures truncation. With the per-case output in, this is good to go.

[sanmaxdev]

Added the requested per-case table to the plain-text output and documented that synthetic lengths use whitespace tokens and may measure model truncation.

Verified:

• python3 -m pytest tests/test_inference_pytest.py tests/test_benchmark_detectors_pytest.py -v -k "not TestAnswerStartToken" (47 passed, 2 deselected)
• python3 -m ruff format --check --diff lettucedetect/ tests/ scripts/benchmark_detectors.py
• python3 -m ruff check lettucedetect/ tests/ scripts/benchmark_detectors.py --extend-exclude lettucedetect/integrations/