Fix surface_direction bearings on north-up rasters and on chunked dask input - #3731
Fix surface_direction bearings on north-up rasters and on chunked dask input#3731brendancol wants to merge 4 commits into
Conversation
…k input (#3719) _finalize_direction built the offset to the source out of row/column indices scaled by an absolute cell size, so the row index stood in for the y coordinate. On a north-up raster, where y descends as the row index rises, every bearing came out mirrored across the east-west axis. _compute now derives the sign of each axis from the coordinate and hands the backends signed cell sizes; the edge-cost paths only ever square or abs() them, so the costs are unchanged. The dask iterative path had a second problem: _run_tile records source positions as global indices, but _finalize_direction measured them against the block's own np.arange grid, leaving the chunk offset in the answer. Every chunk but (0, 0) reported the bearing it would have had as a standalone raster. _assemble_sd now rebases to block-local indices. Rebasing exposed a third: the no-source mask tested src_row < 0, which a source sitting in the chunk above rebases into. Since a finite in-budget distance always comes with an assigned source, the mask now keys on dist alone, and direction ends up with the same NaN mask as distance and allocation. Verified against numpy, cupy, dask+numpy, and dask+cupy.
brendancol
left a comment
There was a problem hiding this comment.
PR Review: Fix surface_direction bearings on north-up rasters and on chunked dask input
Reviewed the full surface_distance.py and test_surface_distance.py at the
head of deep-sweep-accuracy-surface_distance-2026-08-16-01, not just the
diff. All three defects are real and the reproductions in #3719 replay against
main. Distances and allocations are untouched, which the existing 37 tests
confirm.
Blockers (must fix before merge)
None.
Suggestions (should fix, not blocking)
-
xrspatial/surface_distance.py:642— the cupy DIRECTION branch still
masks onsrow < 0._finalize_directiondropped that clause at line
399-406 because a rebased block-local index can legitimately be negative.
The eager cupy path never rebases, sosrowis always non-negative wherever
distis finite and the two masks agree today. But leaving one backend
keying on the source index and the other ondistis the asymmetry that
produced this bug in the first place. Make itcp.isinf(dist) | (dist > max_distance)so both paths state the same rule. -
xrspatial/surface_distance.py:1355-1359— the signed cell size relies on
an unwritten invariant. Passing signed values through the existing
cellsize_x/cellsize_yparameters is correct only because every other
consumer squares them (_compute_seeds_sdbuildsdiag_distfrom
cellsize_x ** 2 + cellsize_y ** 2,_edge_seedsusescardinal_dist ** 2) or takesabs()first (_surface_distance_dask,
_surface_distance_dask_bounded,_surface_distance_dask_cupy). I walked
every call site and the invariant holds, but nothing enforces it, and the
next person to add a cell-size consumer has no way to know. A test asserting
that a descending-y raster produces the same DISTANCE and ALLOCATION output
as an ascending-y one would pin it cheaply.
Nits (optional improvements)
-
xrspatial/surface_distance.py:344—_coord_step_signreads the span,
not a step.values[-1] - values[0]gives the right answer for any
monotonic axis and matches howcalc_resalready derives resolution, but
surface_distancenever checks monotonicity the way
proximity._check_monotonic_coordsdoes. On a scrambled coordinate the sign
is whatever the endpoints happen to say. Not worth a validation pass, but
worth a word in the docstring that the axis is assumed monotonic. -
Geodesic direction is still index-scaled. In
method='geodesic'mode
_finalize_directionscales index offsets by a cell size in degrees, so
bearings drift away from the equator where a degree of longitude is shorter
than a degree of latitude. This PR fixes the sign but not the scale. It
predates the change and is not made worse by it; worth a separate issue
rather than growing this diff.
What looks good
- The three defects are separated cleanly, and the commit message explains how
fixing the second exposed the third rather than presenting it as one insight. - Distances and allocations come out byte-identical, which is the right
outcome for a direction-only fix and is what makes the signed-cell-size
approach defensible. test_direction_matches_proximity_directionpins agreement with the sibling
API instead of hard-coding numbers, so the two functions cannot drift apart
again silently.test_direction_nan_mask_matches_distancecovers bounded and unbounded dask
and is what caught the third defect mid-fix. Keeping it in the suite is more
valuable than the bearing assertions.- The verification ran all four backends on a descending-y raster rather than
reasoning about the GPU path from source.
Checklist
- Algorithm matches reference — bearings now agree with
proximity.direction, the in-repo reference for this convention - All implemented backends produce consistent results — numpy, cupy,
dask+numpy, dask+cupy checked bounded and unbounded, zero difference - NaN handling is correct — direction now shares the distance NaN mask
- Edge cases covered — both axis directions, no-source raster, chunks
whose source lives in a neighbouring chunk - Dask chunk boundaries handled correctly — the tiled path was the bug
- No premature materialization —
srow - row_offsetis a per-block numpy
op inside an existingmap_blockschunk function - Benchmark exists —
benchmarks/benchmarks/surface_distance.py, and this
change has no throughput effect - README feature matrix — not applicable, no new function and no backend
support change - Docstrings present and accurate — the
surface_directioncompass
convention in the public docstring is now what the function returns
…e invariant (#3719) The cupy DIRECTION branch still masked on srow < 0 while _finalize_direction had moved to keying on dist. Both paths now state the same rule, so the asymmetry that produced this bug cannot come back through the GPU side. Passing signed cell sizes through the existing cellsize_x/cellsize_y parameters works only because every edge-cost consumer squares them or takes abs() first. That held at every call site but nothing enforced it, so test_distance_and_allocation_ignore_axis_direction now flips the y axis and asserts distance and allocation come back unchanged. Also noted in _coord_step_sign's docstring that the axis is assumed monotonic, the same assumption calc_res already makes. The remaining review nit, geodesic bearings scaling index offsets by degrees without correcting for meridian convergence, is filed as #3733: it predates this change and the fix needs a convention decision this PR should not make.
brendancol
left a comment
There was a problem hiding this comment.
PR Review: follow-up pass after da98601
Re-read the changed hunks and re-ran the suite. Disposition of the first pass:
Fixed
- The cupy DIRECTION branch no longer masks on
srow < 0. Both it and
_finalize_directionnow key ondist, so the two backends state the same
rule and the GPU side cannot reintroduce the asymmetry. test_distance_and_allocation_ignore_axis_directionpins the invariant the
signed cell size leans on: flip the y axis and distance and allocation come
back unchanged. Parametrized over numpy and dask+numpy, and over
surface_distanceandsurface_allocation. 50 tests pass (37 onmain)._coord_step_sign's docstring now says the axis is assumed monotonic and
points atcalc_resmaking the same assumption.
Deferred, with a linked issue
- Geodesic bearings still scale index offsets by a cell size in degrees, so
they tilt withcos(lat)away from the equator. Filed as #3733. It predates
this PR, and the fix needs a decision this one should not make: whether to
scale the longitude leg bycos(lat)at the pixel, or compute a forward
azimuth from the two lat/lon pairs, which changes what "bearing" means over
a long path since a great circle's initial azimuth is not its final one.
No new findings. Nothing outstanding from my side.
|
Overlap notice: PR #3729 (test-coverage sweep) and PR #3731 (accuracy sweep) both fix the dask They are not equivalent, and the difference matters:
The trap in merging #3729 alone: its direction tests assert that dask matches numpy. If numpy is itself mirrored, that parity holds while both backends are wrong, so the suite would go green over a live defect. #3731's Suggested resolution: take #3731's source changes as the base (superset of the direction fixes), keep #3729's test additions, and reconcile the overlapping hunks in Note for sequencing: PR #3724 adds a docstring caveat describing this bug as unfixed, so it should land after whichever fix merges, with that paragraph dropped. |
Closes #3719
surface_direction()was returning wrong bearings in two unrelated ways, andfixing the second one uncovered a third.
surface_distance()andsurface_allocation()were correct throughout.What changed
_finalize_direction()measured the offset to the source in row/columnindices scaled by an absolute cell size, so the row index stood in for y.
On a north-up raster (y descending as the row index rises) every bearing
came out mirrored across the east-west axis, and
surface_direction()disagreed with
proximity.direction()on identical input._compute()nowreads the sign of each axis off the coordinate and passes the backends
signed cell sizes. Every edge-cost site either squares the cell size or
takes
abs()of it first, so distances and allocations are byte-identicalto before.
_assemble_sd()handed_finalize_direction()the global source indicesthat
_run_tile()records, while the bearing was measured against theblock's own
np.arangegrid. The chunk offset stayed in the answer andevery chunk except (0, 0) reported the bearing it would have had as a
standalone raster. It now rebases to block-local indices. This is the
default path:
max_distancedefaults tonp.inf, which routes to_sd_dask_iterative().Rebasing broke the no-source mask, which tested
src_row < 0— a sourcein the chunk above rebases to a legitimately negative local index. A finite
in-budget distance always comes with an assigned source, so the mask now
keys on
distalone. Direction ends up with the same NaN mask as distanceand allocation, which is what the docstring already promised.
Backends
numpy, cupy, dask+numpy, dask+cupy. All four checked against the numpy
baseline on a descending-y raster, bounded and unbounded, with zero NaN-mask
mismatches and zero numeric difference. The cupy eager path shares
_finalize_direction's index math and was affected by the orientation bugtoo; dask+cupy inherits the tiled fix through the CPU iterative path.
Test plan
pytest xrspatial/tests/test_surface_distance.py— 46 passed (37 before)pytest xrspatial/tests/test_proximity.py xrspatial/tests/test_cost_distance.py— 686 passedflake8clean on both changed files (the F841 attest_surface_distance.py:234predates this branch and is untouched)test_direction_follows_y_axis_orientationpins both axisdirections;
test_direction_matches_proximity_directionpins agreementwith the sibling API;
test_dask_direction_matches_numpyandtest_direction_nan_mask_matches_distancecover bounded and unboundeddask;
test_cupy_direction_matches_numpycloses the GPU gapmain'ssurface_distance.pyunder the new tests:test_direction_follows_y_axis_orientation,test_direction_matches_proximity_direction, and bothtest_dask_direction_matches_numpycases fail there and pass here.test_direction_nan_mask_matches_distanceandtest_cupy_direction_matches_numpypass onmainas well — they pininvariants this change had to preserve, and the mask one is what caught
the third defect mid-fix.
Found by
/sweep-accuracyagainstxrspatial/surface_distance.py. The stateCSV row for the module is in the second commit.