Skip to content

Pin surface_distance against scipy csgraph and document the allocation tie-break - #3730

Open
brendancol wants to merge 2 commits into
mainfrom
deep-sweep-reference-validation-surface_distance-2026-08-16
Open

Pin surface_distance against scipy csgraph and document the allocation tie-break#3730
brendancol wants to merge 2 commits into
mainfrom
deep-sweep-reference-validation-surface_distance-2026-08-16

Conversation

@brendancol

Copy link
Copy Markdown
Contributor

Closes #3726.

surface_distance had no test comparing it to anything outside xrspatial.
Its existing suite checks internal consistency (backends against each
other, flat terrain against cost_distance) and a couple of hand-computed
values, so a shared bug in the cost model would pass unnoticed.

What was validated

I ran the module against scipy.sparse.csgraph.dijkstra (scipy 1.16.1)
over an explicitly constructed grid graph whose edge weights are
sqrt(horizontal**2 + dz**2), the cost model the docstring claims. That
is an independent implementation of the same algorithm, so agreement is
meaningful.

Scenes: a 64x64 Gaussian hill, a tilted plane, a seeded rough surface, and
a rough surface with NaN barriers. Each at connectivity 4 and 8, with and
without a finite max_distance, with and without target_values.

backend distance allocation
numpy exact (max_abs 0.0) exact
dask+numpy, bounded map_overlap exact exact
dask+numpy, unbounded iterative exact exact
cupy exact exact
dask+cupy exact exact

surface_direction bearings match the reference bearings to float32
precision (5.9e-8 relative) and the compass convention is right on
north-up rasters: due north reads 360, due east 90.

Geodesic mode matches an independent haversine graph at R = 6378137
exactly, which is the radius the rest of the library uses. Against the
WGS84 ellipsoid via pyproj it differs by at most 0.17% locally with a
mean bias of +0.003% — the ordinary spherical approximation.

The tilted plane also has a closed form. Along the row, column and main
diagonal through the source the octile path is the straight line, so
xrspatial's answer should equal sqrt(dx**2 + dy**2 + dz**2) with no
metrication error. It does, to float32 epsilon. Worth checking separately:
xrspatial and scipy agreeing with each other but not with the math would
be two bugs, not zero.

What this PR adds

Golden-value tests pinning that parity: the reference distance matrices
for connectivity 4 and 8 on an 8x8 scene with a NaN barrier and two
competing sources, the matching allocation labels, and the analytic
tilted-plane case. Expected values come from scipy, not from xrspatial's
output, and the comment says so, so a future regression cannot be papered
over by regenerating them.

The one divergence, and the docstring change

Allocation diverged from the reference on 32 pixels of one symmetric
scene. All 32 are exact ties: computing the distance to each source
separately gives max |d1 - d2| == 0.0 over them.

Chasing that showed the three backends break ties by three unrelated
mechanisms — heap pop order on numpy and dask, relaxation convergence
order on cupy, tile sweep order on the dask iterative path — and none of
them is documented. On a symmetric 64x64 scene, 29 of 4096 allocation
pixels differ between numpy and cupy and 25 between numpy and dask.

proximity.allocation states its rule. The surface trio stated nothing,
so this adds a Notes section to surface_allocation and
surface_direction saying the tie-break is unspecified and
backend-dependent. Documentation only; no behaviour changes. #3726 leaves
open the alternative of adopting proximity's lowest-flat-index rule
everywhere, which would change results and needs a call on the GPU cost.

Not in this PR

Validation also surfaced two CRITICAL defects that parallel sweep agents
had already filed, so I added evidence rather than duplicate issues:

Neither is touched here. The golden values in these tests are the correct
ones, so they stay valid once those fixes land.

Verification

$ python -m pytest xrspatial/tests/test_surface_distance.py -q
46 passed

37 pre-existing plus 9 new, on a CUDA host so the cupy test ran for real.
flake8 on both changed files is clean apart from one pre-existing F841
at test_surface_distance.py:234, left alone.

…n tie-break (#3726)

Validated surface_distance/allocation/direction against an independent
reference: scipy.sparse.csgraph.dijkstra over an explicitly built grid
graph with sqrt(horizontal^2 + dz^2) edge weights. All four backends
reproduce it bit for bit. Adds golden-value tests pinning that parity,
plus a tilted-plane case checked against the closed-form surface
distance, since the module had no reference test of any kind.

Also documents that surface_allocation and surface_direction resolve
exact distance ties differently on each backend (#3726). The distances
agree; only the reported source varies.
@brendancol

Copy link
Copy Markdown
Contributor Author

Verification trail for this PR.

Local, on a CUDA host, against the branch code (PYTHONPATH pinned to the worktree, xrspatial.__file__ confirmed to resolve inside it):

$ python -m pytest xrspatial/tests/test_surface_distance.py -q
46 passed in 3.68s
$ flake8 xrspatial/surface_distance.py
$ flake8 xrspatial/tests/test_surface_distance.py
xrspatial/tests/test_surface_distance.py:234:5: F841 local variable 'sd' is assigned to but never used

That F841 predates this branch (confirmed by stashing the diff and re-running); left alone under the surgical-changes rule, worth a separate cleanup.

The golden values were produced by the scipy reference and then checked against xrspatial, not the other way round. The generator asserted scipy == xrspatial for distance, allocation, and the connectivity-4 variant before printing the arrays, so the committed numbers are the reference's.

The cupy test is not skipped here — it ran and passed on the local GPU.

Scope check on the two open CRITICALs: neither is touched. #3722 is heap sizing in _dijkstra, #3721 is the cupy iteration cap. The 8x8 golden scene is far too small and too sparse to reach the heap bound, and it converges in well under H + W cupy passes, so these tests pass both before and after those fixes and will keep pinning the right answers afterwards.

CI: 6 checks green, 0 failing, the rest queued. mergeable: MERGEABLE; BLOCKED is REVIEW_REQUIRED, which is the maintainer's to clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

surface_allocation/surface_direction tie-break is backend-dependent and undocumented

1 participant