Bad-line MAD detection fixes + sub-pixel Gaussian blur with physical readout#38
Merged
Merged
Conversation
…adout
The image-viewer "Smooth: Gaussian" control was an integer slider (1-20 px) and
reported nothing about the physical extent of the kernel, so a user could not
tell what distance was being blurred or do gentle sub-pixel smoothing. The blur
itself (scipy gaussian_filter, sigma in px, kernel truncated at +-4 sigma) is
sound and ~equivalent to ImageJ for sigma >= 1 px on float data; the gap was the
GUI control and units feedback. Backend already accepts float sigma, so this is
GUI-only.
Where to look:
- probeflow/gui/processing.py:
* format_gaussian_readout(sigma_px, px_nm) — pure, unit-tested formatter
(FWHM = 2.3548*sigma; kernel half-width = int(4*sigma+0.5), matching scipy
truncate=4.0).
* _sub_slider() gained a `scale` arg; the smooth slider now spans 0.2-20.0 px
in 0.1 steps (_SMOOTH_SIGMA_SCALE). state()/set_state() convert accordingly.
* set_pixel_size_nm() + _update_smooth_readout() drive a readout label that
shows sigma/FWHM/kernel extent in nm (or px-only when uncalibrated).
- probeflow/gui/dialogs/image_viewer.py: _refresh_display_array pushes the loaded
scan's pixel size into the panel via set_pixel_size_nm.
Tests: test_format_gaussian_readout_pure, test_viewer_full_smooth_sigma_is_subpixel_float.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…xtended lines The MAD/outlier bad-line method failed on real STM/AFM scans (createc qPlus AFM) in three compounding ways. All three are fixed in probeflow/processing/bad_lines.py (detect_bad_scanline_segments MAD branch + helpers). 1. Correction could not be applied. _split_segments_by_adjacent_limit was column-blind: a line counted as "bad" if it had a segment anywhere, so noise across many lines formed one long consecutive-line run that exceeded "max adjacent" and ALL segments were skipped (nothing repaired). It is now column-overlap-aware (union-find over segments linked only when adjacent rows overlap in columns); only a genuine vertical stack taller than the limit is skipped. The repair itself was already column-aware. 2. Long segments were invisible/shattered. The method subtracted each row's own median before thresholding, so a defect covering >~half the row became the row median and vanished (lowering the threshold only surfaced noise). It now thresholds the neighbour-referenced residual directly, bridges short noise gaps (_close_small_gaps, _NOISE_GAP_TOL_PX) and keeps a run only if it is a defect on its MEDIAN — robust to per-pixel noise; no upper-length cap. 3. Low-contrast lines were undetectable. The real bright lines are only ~1.5 sigma per pixel, so per-pixel thresholding can't separate them from texture. The residual is now smoothed along each row first (_smooth_rows_nanaware, _MAD_SMOOTH_WINDOW_PX = 11) as a matched filter for extended defects: SNR gain ~sqrt(window) lets a faint-but-long line clear a clean threshold while texture averages out. On the sample file, threshold-5 detection went from 0 to ~80 segments on exactly the bright rows, repaired with 0 skipped. The `step` method is unchanged and remains the sharp/short-scar detector. Tests: test_adjacent_limit_is_column_aware, test_mad_detects_long_partial_segment_against_noise, test_mad_detects_whole_line_offset_not_just_internal_segments, and an updated (now matched-filter-aware) test_mad_outlier_method_repairs_only_segment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent, self-contained improvements (one commit each; reviewable separately).
1. Bad-line MAD detection — works on real low-contrast scans
Verified by the user on
createc_scan_qplus_10ch_afm(Z forward). Three compounding bugs fixed inprobeflow/processing/bad_lines.py:_split_segments_by_adjacent_limit) — only a genuine vertical stack taller than the limit is skipped._smooth_rows_nanaware,_MAD_SMOOTH_WINDOW_PX) before thresholding. On the sample file, threshold-5 detection went 0 → ~80 segments on exactly the bright rows, repaired with 0 skipped, flattening the bright excess 0.4 pm → 0.0 pm while touching ~2% of pixels. Thestepmethod is unchanged (sharp-scar detector).2. Sub-pixel Gaussian blur + physical readout
The "Smooth: Gaussian" control was integer-only (1–20 px) with no feedback on the kernel's physical extent. Now: float σ 0.2–20.0 px in 0.1 steps, and a readout showing σ/FWHM/kernel-extent in nm (FWHM = 2.3548·σ; kernel ±4σ, matching scipy
truncate=4.0). GUI-only — the backend already accepted float σ. Seeformat_gaussian_readoutandset_pixel_size_nminprobeflow/gui/processing.py.Tests
New/updated regression tests in
tests/test_processing.pyandtests/test_gui_processing_panel.py; full suite 254 passed,ruffclean.🤖 Generated with Claude Code