Pin surface_distance against scipy csgraph and document the allocation tie-break - #3730
Pin surface_distance against scipy csgraph and document the allocation tie-break#3730brendancol wants to merge 2 commits into
Conversation
…n tie-break (#3726) Validated surface_distance/allocation/direction against an independent reference: scipy.sparse.csgraph.dijkstra over an explicitly built grid graph with sqrt(horizontal^2 + dz^2) edge weights. All four backends reproduce it bit for bit. Adds golden-value tests pinning that parity, plus a tilted-plane case checked against the closed-form surface distance, since the module had no reference test of any kind. Also documents that surface_allocation and surface_direction resolve exact distance ties differently on each backend (#3726). The distances agree; only the reported source varies.
|
Verification trail for this PR. Local, on a CUDA host, against the branch code ( That F841 predates this branch (confirmed by stashing the diff and re-running); left alone under the surgical-changes rule, worth a separate cleanup. The golden values were produced by the scipy reference and then checked against xrspatial, not the other way round. The generator asserted The cupy test is not skipped here — it ran and passed on the local GPU. Scope check on the two open CRITICALs: neither is touched. #3722 is heap sizing in CI: 6 checks green, 0 failing, the rest queued. |
Closes #3726.
surface_distancehad no test comparing it to anything outside xrspatial.Its existing suite checks internal consistency (backends against each
other, flat terrain against
cost_distance) and a couple of hand-computedvalues, so a shared bug in the cost model would pass unnoticed.
What was validated
I ran the module against
scipy.sparse.csgraph.dijkstra(scipy 1.16.1)over an explicitly constructed grid graph whose edge weights are
sqrt(horizontal**2 + dz**2), the cost model the docstring claims. Thatis an independent implementation of the same algorithm, so agreement is
meaningful.
Scenes: a 64x64 Gaussian hill, a tilted plane, a seeded rough surface, and
a rough surface with NaN barriers. Each at connectivity 4 and 8, with and
without a finite
max_distance, with and withouttarget_values.map_overlapsurface_directionbearings match the reference bearings to float32precision (5.9e-8 relative) and the compass convention is right on
north-up rasters: due north reads 360, due east 90.
Geodesic mode matches an independent haversine graph at R = 6378137
exactly, which is the radius the rest of the library uses. Against the
WGS84 ellipsoid via pyproj it differs by at most 0.17% locally with a
mean bias of +0.003% — the ordinary spherical approximation.
The tilted plane also has a closed form. Along the row, column and main
diagonal through the source the octile path is the straight line, so
xrspatial's answer should equal
sqrt(dx**2 + dy**2 + dz**2)with nometrication error. It does, to float32 epsilon. Worth checking separately:
xrspatial and scipy agreeing with each other but not with the math would
be two bugs, not zero.
What this PR adds
Golden-value tests pinning that parity: the reference distance matrices
for connectivity 4 and 8 on an 8x8 scene with a NaN barrier and two
competing sources, the matching allocation labels, and the analytic
tilted-plane case. Expected values come from scipy, not from xrspatial's
output, and the comment says so, so a future regression cannot be papered
over by regenerating them.
The one divergence, and the docstring change
Allocation diverged from the reference on 32 pixels of one symmetric
scene. All 32 are exact ties: computing the distance to each source
separately gives
max |d1 - d2| == 0.0over them.Chasing that showed the three backends break ties by three unrelated
mechanisms — heap pop order on numpy and dask, relaxation convergence
order on cupy, tile sweep order on the dask iterative path — and none of
them is documented. On a symmetric 64x64 scene, 29 of 4096 allocation
pixels differ between numpy and cupy and 25 between numpy and dask.
proximity.allocationstates its rule. The surface trio stated nothing,so this adds a Notes section to
surface_allocationandsurface_directionsaying the tie-break is unspecified andbackend-dependent. Documentation only; no behaviour changes. #3726 leaves
open the alternative of adopting
proximity's lowest-flat-index ruleeverywhere, which would change results and needs a call on the GPU cost.
Not in this PR
Validation also surfaced two CRITICAL defects that parallel sweep agents
had already filed, so I added evidence rather than duplicate issues:
H*Wand overflows. Reproduced hereon a 40x40 raster with a 50% dense source mask:
IndexErrorunderNUMBA_BOUNDSCHECK=1,corrupted size vs. prev_sizeand a core dumpwithout it. Peak heap occupancy measured at 1.07x
H*W. (surface_distance Dijkstra heap is sized height*width and overflows on dense target rasters, corrupting memory #3723 is aduplicate; commented there.)
H + W. I added a case showing it isnot limited to barrier mazes and does not always fail loudly: with no
impassable pixels at all, only steep ridges, cupy returns finite
distances up to 173.8% too large on 1314 of 1600 pixels.
Neither is touched here. The golden values in these tests are the correct
ones, so they stay valid once those fixes land.
Verification
37 pre-existing plus 9 new, on a CUDA host so the cupy test ran for real.
flake8on both changed files is clean apart from one pre-existing F841at
test_surface_distance.py:234, left alone.