The cycle walk never asserts the API refusals it collects, so a real one cannot fail the run

[JArmandoAnaya]

The real-server browser walk collects every API call the application made that the API refused, and never asserts anything about the collection. apiRefusals is written to and read once, and the read is a suppression: a console error whose location matches a member of the set is dropped rather than recorded. Nothing else ever looks at it.

const apiRefusals = new Set<string>();
page.on("response", (response) => {
  const kind = response.request().resourceType();
  if (response.status() < 400) return;
  if (kind === "fetch" || kind === "xhr") apiRefusals.add(response.url());
  else badRequests.push(`${response.status()} ${kind} ${response.url()}`);
});
page.on("console", (message) => {
  if (message.type() !== "error") return;
  if (apiRefusals.has(message.location().url)) return;
  consoleErrors.push(`${message.text()} @ ${message.location().url}`);
});

So every response at or above 400 on a fetch or xhr takes the first branch, enters the set, and is thereby exempted from the console assertion — whatever its status, whatever route produced it, however many times.

Why it was built this way, and where the asymmetry is

The suppression is not arbitrary, and the comment above it explains itself: the walk contains refused calls by design, because GET /projects/{id}/schema answers 404 for a project that has no schema yet, and that 404 is how the schema editor knows to open on an empty draft. Chrome logs a console error for each one. Without the set, the final step's console assertion would fail on the product working correctly.

What is missing is the other half. The walk's two neighbouring collectors are both asserted in the final step, badRequests against an empty array and abortedApiCalls against an exact two-element list, with a comment explaining which aborted calls are expected and why. apiRefusals gets neither treatment. It is the only one of the three that exists solely to make an assertion pass, and the only one that grows without bound.

The step those assertions live in is titled "the whole walk produced a clean console". The console is clean in part because this set makes it so.

What that costs, concretely

The walk currently produces a refusal nothing accounts for. On an installation without the optional local-inference extra — which is what the browser cycle (chromium) job installs — the connection dialog's size probe answers 500:

GET /inference/download-size?model_id=facebook%2Fsam2.1-hiera-base-plus&model_revision=b7320756a13354e7530a63935656d35b2f91a290 500

That status is deliberate. LocalInferenceUnavailable is mapped in src/visionset/server/errors.py to ErrorRule(500, "LOCAL_INFERENCE_UNAVAILABLE", expose_message=True), and the route's docstring says it is refused with the install command when the local runtime is absent. So this is the product behaving as designed, and it is not a defect.

It is the illustration, though. A genuine 500 — an unhandled exception in a route the walk exercises, a regression that turns a working call into a refusal — reaches this suite as exactly the same thing: a fetch response at or above 400, added to a set, never looked at. The run stays green and the report says the console was clean. The suite cannot presently distinguish the refusal it was built to tolerate from the one it exists to catch, and there is no output anywhere in a passing run that would let a person notice the difference either.

This is the walk that has three separate times been the only suite to catch a regression every other check reported green, which is what makes the blind spot worth closing rather than noting.

Directions

Both of these follow shapes already in the file, and choosing between them is part of the work rather than settled here.

The set could be asserted the way abortedApiCalls is: record method, path and status rather than a bare URL, and pin the result against the refusals the walk is known to produce — the schema 404s, and the size probe's 500 on an installation without the extra. That names the expected set in one place, and anything new fails with the route and status in the message.

Or the tolerated refusals could be narrowed at the collection site, the way image-byte cancellations already are in the requestfailed handler: exempt the routes whose refusals are load-bearing, and let everything else fall through to an assertion. That keeps the exempt list next to the reason for each exemption.

Either way the size probe's refusal has to be expressible, since it depends on which extras are installed and the two inference CI jobs install differently.

Related

Found while giving the walk's retry a workspace it can pass (#688), whose pull request (#696) records this among the things it did not fix.