Add rate limits initial documentation by nicu-da · Pull Request #1450 · canton-network/cf-docs

nicu-da · 2026-08-24T08:19:06Z

https://github.com/DACH-NY/canton-network-internal/issues/6475

Signed-off-by: Nicu Reut <nicu.reut@digitalasset.com>

mintlify · 2026-08-24T08:19:09Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
cantonfoundation	🟢 Ready	View Preview	Aug 24, 2026, 8:22 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

martinflorian-da

Thanks a lot! I do expect we'll need to tweak this more going forward but this is super helpful to maintain an overview.

My main nudge at this time would be to remove references to SV app...

martinflorian-da · 2026-08-24T13:29:57Z

+description: "Rate limiting requirements for Super Validator nodes"
+---
+
+SV nodes must protect their HTTP and gRPC endpoints against excessive load with two complementary layers:


Suggested change

SV nodes must protect their HTTP and gRPC endpoints against excessive load with two complementary layers:

SV nodes protect their HTTP and gRPC endpoints against excessive load with two complementary layers:

martinflorian-da · 2026-08-24T13:50:02Z

+</Warning>
+
+```hocon
+canton.scan-apps.scan-app.parameters.rate-limiting.client-ip-headers = ["x-envoy-external-address"]


This seems to miss a sentence about "what is this snippet here telling me".

It will also confuse people not used to thinking about the low-level HOCON configs; WDYT of making these two name: ADDITIONAL_CONFIG_... style snippets instead?

...and possibly we also want to expose these via Helm values at some point? (So might need a hook here to replace with Helm value overrides if we end up enabling these.)

martinflorian-da · 2026-08-24T13:55:05Z

+
+### Requirements
+
+The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API, public SV app endpoints and the sequencer gRPC API.


Suggested change

The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API, public SV app endpoints and the sequencer gRPC API.

The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API and the sequencer gRPC API.

My understanding is that we're not planning to make any SV API endpoints public.

If I'm wrong let's please talk about this :)

If not let's not even mention the SV app anywhere here, to avoid raising eyebrows and also avoid SVs investing work into something that will matter less for dropping the whitelists on scan and sequencer.

My understanding is that we're not planning to make any SV API endpoints public.

No way around it.

Also check canton-network/splice#6934

No way around it.

Please let's unroll this with a bit more detail. What exactly do you mean?

I commented on the PR on all the validator-public endpoints. I don't think a single of those should remain at the time of dropping whitelists.

What about https://github.com/canton-network/splice/pull/6769/changes#diff-f0367a50f64ea65f9e6d528b04834cc4e14b4eddd120864fcdf03039364b1affR496?

That is a fair point! But this is DevNet-only, a single endpoint and effectively doesn't exist yet on main.

So I think I'm still leaning towards "let's keep the SV app out of this for now"; we can explain why we need this one single endpoint and the rate limits for it once we introduce that endpoint.

Yes, but DevNet is still a prod cluster that exposes that endpoint.
Note that I am totally fine with removing the mentions of sv app from here, but this goes in the wrong direction of making sure that the svs only allow exactly what needed as accessible by the validators.

So I think I'm still leaning towards "let's keep the SV app out of this for now"; we can explain why we need this one single endpoint and the rate limits for it once we introduce that endpoint.

Well yes, but the current docs are made for the current version, we need rate limits to get rid of whitelists but having whitelists imo doesn't mean we don't need rate limits. So just leaving the SV app uncovered does sound a bit tricky.

OK then how about we add some sentence somewhere that the current plan is to reduce the need to make SV app endpoints reachable, so that in the end we only need to do that on DevNet?

martinflorian-da · 2026-08-24T13:57:59Z

+
+Both limits must be enforced over a **fixed window of 60 seconds**. A client is allowed at most `limit` requests per 60s window; once the budget for the current window is exhausted, all further requests from that client (or, for the global limit, to that service) are rejected with `429 Too Many Requests` until the window rolls over.
+
+For the sequencer, limits must be applied to gRPC traffic requests.


How confident are you that you can throw "gRPC traffic request" at operators and they'll end up configuring the right thing? Can this be mapped to something on a lower layer (HTTP2?)?

I expect that most proxies have native support for gRPC. Will just switch to gRPC (HTTP2)

martinflorian-da · 2026-08-24T13:59:49Z

+
+Beyond the global and per-IP limits above, operators are encouraged to keep the **ability to apply more fine-grained, custom rate limits on demand** — for example per URL path or path prefix, per gRPC service or method, or for an individual client IP or IP range. This makes it possible to react quickly to a misbehaving client or an expensive endpoint without taking the whole node offline, and to lift limits temporarily for a specific trusted peer.
+
+Since the apps derive the client IP from an HTTP header, the same infrastructure that enforces these limits must also set that header in a way the client cannot influence.


Shouldn't this sentence (if we at all want to keep it) move up where "Per-source-IP limits" are introduced?

martinflorian-da · 2026-08-24T14:01:00Z

+
+### Optional: banning aggressively retrying clients
+
+Clients that ignore `429` responses and immediately retry keep consuming ingress capacity even though every request is rejected. Operators are therefore encouraged to additionally **ban clients that retry too quickly**.


I wonder if we can't just skip that here and later expend extra banning when we discuss WAF / DDoS protection... not a strong opinion.

martinflorian-da · 2026-08-24T14:43:41Z

+
+#### Monitoring
+
+- `splice_splice_rate_limiting` — metrics prefix that identifies the rate limiting metrics.


Are you sure it's splice_splice_ and not just splice_?

nicu-da · 2026-08-24T14:57:16Z

Thanks a lot! I do expect we'll need to tweak this more going forward but this is super helpful to maintain an overview.

My main nudge at this time would be to remove references to SV app...

I definitely expect we keep evolving this in the short to medium term, but wanted to front run this so that SVs can also get a head start on basic rate limits.

Add rate limits initial documentation

aa232cb

Signed-off-by: Nicu Reut <nicu.reut@digitalasset.com>

mintlify Bot deployed to staging - docs-main August 24, 2026 08:22 View deployment

nicu-da requested a review from martinflorian-da August 24, 2026 09:58

martinflorian-da approved these changes Aug 24, 2026

View reviewed changes

martinflorian-da mentioned this pull request Aug 24, 2026

Expose sv app endpoints to the minimum users canton-network/splice#6934

Open

8 tasks

	SV nodes must protect their HTTP and gRPC endpoints against excessive load with two complementary layers:
	SV nodes protect their HTTP and gRPC endpoints against excessive load with two complementary layers:


		### Requirements

		The following infrastructure level rate limits should be configured. These limits should apply individually to the Scan API, public SV app endpoints and the sequencer gRPC API.


		Both limits must be enforced over a fixed window of 60 seconds. A client is allowed at most `limit` requests per 60s window; once the budget for the current window is exhausted, all further requests from that client (or, for the global limit, to that service) are rejected with `429 Too Many Requests` until the window rolls over.

		For the sequencer, limits must be applied to gRPC traffic requests.


		Beyond the global and per-IP limits above, operators are encouraged to keep the ability to apply more fine-grained, custom rate limits on demand — for example per URL path or path prefix, per gRPC service or method, or for an individual client IP or IP range. This makes it possible to react quickly to a misbehaving client or an expensive endpoint without taking the whole node offline, and to lift limits temporarily for a specific trusted peer.

		Since the apps derive the client IP from an HTTP header, the same infrastructure that enforces these limits must also set that header in a way the client cannot influence.


		### Optional: banning aggressively retrying clients

		Clients that ignore `429` responses and immediately retry keep consuming ingress capacity even though every request is rejected. Operators are therefore encouraged to additionally ban clients that retry too quickly.


		#### Monitoring

		- `splice_splice_rate_limiting` — metrics prefix that identifies the rate limiting metrics.

Uh oh!

Conversation

nicu-da commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mintlify Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martinflorian-da left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicu-da commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nicu-da commented Aug 24, 2026 •

edited

Loading

mintlify Bot commented Aug 24, 2026 •

edited

Loading