A measurement-grade HTTP/1.1 and HTTP/2 load generator for benchmarking web servers.
loadgen ships its own H1 and H2 clients rather than wrapping net/http, so the numbers it reports describe the server under test and not the client's overhead. Latency is recorded per worker in sharded HdrHistograms with no lock contention on the hot path, and percentiles come from a merged histogram spanning 1µs to 30s at three significant digits. It is the reference load generator for the goceleris/probatorium cluster benchmark.
Part of the goceleris family — celeris (the server), probatorium (the harness), loadgen (this), and docs. Full documentation lives at goceleris.dev.
- Custom H1/H2 clients. Pre-formatted request bytes, pre-encoded HPACK headers, and allocation-free HPACK status extraction keep client-side overhead out of your measurements.
- Sharded latency recording. One HdrHistogram per worker, merged once at completion — no lock on the cumulative hot path.
- Honest percentiles. p50 through p99.99 from a merged HdrHistogram, plus the full distribution exported as a V2-compressed HdrHistogram for downstream re-aggregation.
- Streaming drivers. Built-in WebSocket and SSE modes measure echo round-trips, broadcast fan-out, and inter-event delivery — not just request/response.
- Rated mode. Closed-loop constant-rate scheduling with Gil Tene coordinated-omission correction: backlog surfaces as honest tail latency instead of vanishing.
- Federation. Drive load from two hosts and merge their histograms client-side, for when a single box can't saturate the target.
- Self-instrumentation. A 1Hz process-CPU sampler and a Linux recv-queue probe flag the runs where loadgen itself — not the server — was the bottleneck.
- Cluster-ready CLI.
cmd/loadgenemits a single JSONResulton stdout and progress on stderr, built to be driven remotely over SSH or Ansible.
Library:
go get github.com/goceleris/loadgen
CLI:
go install github.com/goceleris/loadgen/cmd/loadgen@latest
package main
import (
"context"
"fmt"
"time"
"github.com/goceleris/loadgen"
)
func main() {
cfg := loadgen.Config{
URL: "http://localhost:8080/",
Duration: 15 * time.Second,
Connections: 256,
Workers: 256,
}
b, err := loadgen.New(cfg)
if err != nil {
panic(err)
}
result, err := b.Run(context.Background())
if err != nil {
panic(err)
}
fmt.Printf("%.0f req/s, p99=%v\n", result.RequestsPerSec, result.Latency.P99)
}cfg := loadgen.Config{
URL: "http://localhost:8080/",
Duration: 15 * time.Second,
HTTP2: true,
HTTP2Options: loadgen.HTTP2Options{
Connections: 16,
MaxStreams: 100,
},
Workers: 64,
}
b, err := loadgen.New(cfg)
// ...cfg := loadgen.Config{
URL: "https://api.example.com/health",
Duration: 30 * time.Second,
Connections: 128,
Workers: 128,
InsecureSkipVerify: true, // self-signed certs only
}For client certificates, a custom CA pool, or a pinned cipher list, set Config.TLSConfig to a *tls.Config instead of using InsecureSkipVerify.
cfg := loadgen.Config{
URL: "http://localhost:8080/",
Duration: 30 * time.Second,
Workers: 64,
Rate: 10000, // 10k req/s, coordinated-omission corrected
}cfg := loadgen.Config{
// ...
OnProgress: func(elapsed time.Duration, snapshot loadgen.Result) {
fmt.Printf("\r%s: %d req, %.0f req/s",
elapsed.Round(time.Second),
snapshot.Requests,
snapshot.RequestsPerSec)
},
}loadgen selects one driver per run. HTTP protocol selection (-h2 / -h2c-upgrade / -mix) and the streaming drivers (-mode) are mutually exclusive — a run is one or the other.
| Mode | Flag | What it measures |
|---|---|---|
| HTTP/1.1 (default) | (none) | Baseline request/response throughput. One persistent TCP connection per worker, pre-formatted request bytes written in a single syscall. For Connection: close workloads (-close), each worker round-robins a pool of connections so reconnect latency stays off the critical path. |
| HTTP/2 | -h2 |
Multiplexed throughput. Prior-knowledge h2c: workers share connections and dispatch streams lock-free, with pre-encoded HPACK headers and batched WINDOW_UPDATE. |
| h2c upgrade | -h2c-upgrade |
The RFC 7540 §3.2 cleartext upgrade path. Each connection starts as HTTP/1.1 carrying Connection: Upgrade, HTTP2-Settings + Upgrade: h2c, reads 101 Switching Protocols, then switches to HTTP/2 on the same socket. Exercises the handshake that -h2 skips. Cleartext only — TLS servers negotiate H2 via ALPN. |
| Protocol mix | -mix h1:h2:upgrade=N:N:N |
A realistic traffic blend. Each new connection is assigned a protocol by weighted random draw and stays on it for life. |
Set -mode (or Config.Mode) to drive a non-HTTP protocol. One long-lived connection is held open per worker; each unit of work flows through the same worker / latency / timeseries pipeline as an HTTP request.
| Mode | What it measures |
|---|---|
ws-echo |
RFC 6455 WebSocket echo round-trip latency (small text frame). |
ws-large-echo |
Echo round-trip with a 64 KiB payload — exercises framing and buffering under large frames. |
ws-hub |
Broadcast fan-out latency: the worker only reads, blocking until the server broadcasts the next frame. No outbound write per unit. |
sse-fanout |
Server-Sent Events delivery. One GET stream per worker; each unit blocks until the next event arrives, so inter-event delivery time is the recorded latency. |
Streaming modes are mutually exclusive with -h2, -h2c-upgrade, and -mix.
By default loadgen runs open-loop: dispatch as fast as the server will accept, and report throughput at saturation.
Pass -rate <req/s> (or set Config.Rate) to switch to a closed-loop constant-rate scheduler that emits one request every 1/rate seconds. Crucially, latency is measured from the request's intended dispatch time, not the moment it actually left — Gil Tene's coordinated-omission correction. When the server stalls, the backlog of intended-but-unsent requests accumulates and surfaces as honest tail latency, instead of being hidden by a client that simply slowed down to match. Rated runs set rated_mode: true, target_rps, and mode: "rated" in the result.
-rate (closed-loop, coordinated-omission corrected) and -max-rps (open-loop token-bucket cap) are independent rate controls; -rate is the right one for latency-SLO measurement.
When one host can't generate enough load to saturate the target, run loadgen on two boxes and merge their measurements client-side.
- Start a sidecar on the second host:
loadgen -sidecar :9099. It binds, waits for a coordinator, and takes its entire workload from the coordinator's start frame. - Start the coordinator with
-peer <sidecar-host>:9099plus the usual flags. It dials the sidecar, both sides run the identical workload against the target at an agreed start time, and at end-of-run the coordinator pulls the sidecar's V2-compressed HdrHistogram and merges it into its own.
The wire protocol is deliberately minimal: a loadgen-fed-v1 magic handshake followed by length-prefixed JSON frames (start / ack / result / err) over one TCP connection, no retries. It is best-effort — if any step fails, the coordinator records federation.merge_succeeded = false and reports its local-only histogram. Merge outcomes appear in the federation block of the result.
Every result carries the full latency distribution, not just the summary percentiles:
Result.Histogramis a V2-compressed HdrHistogram (base64 in JSON), covering 1µs–30s at three significant digits. Empty when no samples were recorded.loadgen.DecodeHistogram([]byte)and the recorder'sEncodeHistogram()are exported so downstream tools (the probatorium cluster aggregator) can decode, re-merge across cells, and recompute percentiles without re-running the benchmark.
This is the same encoding federation uses on the wire, so a merged multi-host distribution round-trips losslessly.
Two probes flag the runs where loadgen — not the server — was the limiting factor:
- CPU sampler (
-cpu-monitor, on by default). Samples the loadgen process's own CPU at 1Hz and reports the P95 ascpu_pct_p95(percentage of available cores). A high value means loadgen was compute-bound and the throughput number is a client ceiling, not a server ceiling. - Recv-queue probe (
-recvq-probe, on by default, Linux only). Reads per-socket receive-queue depths from/proc/net/tcp. If the median across loadgen's sockets stays above 64 KB for ≥10s it latchesrecvq_high: true— the client couldn't drain responses fast enough, so the observed tail latency is loadgen-side. A no-op on non-Linux platforms.
loadgen [flags] -url <target>
| Flag | Default | Meaning |
|---|---|---|
-url string |
(required) | Target URL. Must include scheme (http/https) and host. |
-duration duration |
15s |
Measured benchmark duration (excludes warmup). |
-warmup duration |
2s |
Warmup phase before measurement. 0 to skip. |
-connections int |
256 |
Number of H1 connections. |
-workers int |
0 |
Concurrent workers. 0 → connections for H1, NumCPU*4 for multiplexed modes. |
-method string |
GET |
HTTP method. |
-H "Key: Value" |
— | Custom header, repeatable. |
-body-file string |
— | Read the request body from a file. |
-close |
false |
Send Connection: close (H1 only). |
-h2 |
false |
HTTP/2 prior-knowledge (h2c / h2). |
-h2c-upgrade |
false |
HTTP/2 via the RFC 7540 §3.2 h2c upgrade handshake. |
-mix string |
— | Per-connection protocol mix, e.g. h1:h2:upgrade=4:4:1. |
-h2-conns int |
16 |
H2 connections. |
-h2-streams int |
100 |
Max concurrent H2 streams per connection. |
-mode string |
— | Streaming driver: ws-echo | ws-large-echo | ws-hub | sse-fanout. Mutually exclusive with -h2 / -h2c-upgrade / -mix. |
-insecure |
false |
Skip TLS certificate verification. |
-max-rps int |
0 |
Open-loop token-bucket cap on total req/s (0 = unlimited). |
-rate float |
0 |
Constant request rate (req/s). >0 enables rated mode with coordinated-omission correction. |
-peer host:port |
— | Federation coordinator: dial a sidecar and merge its histogram. |
-sidecar host:port |
— | Run as a federation sidecar; all workload settings come from the coordinator. |
-out string |
— | Also write the JSON result to this file (in addition to stdout). |
-cpu-monitor |
true |
Enable the 1Hz process-CPU sampler (cpu_pct_p95). |
-recvq-probe |
true |
Enable the per-socket recv-queue probe (Linux only; recvq_high). |
The CLI's
-warmupdefault is2s; the library'sDefaultConfig()uses5s.
# H1 with custom headers.
loadgen -url http://localhost:8080/api -duration 30s -connections 512 \
-H "Authorization: Bearer token" -H "Content-Type: application/json"
# H2 prior-knowledge.
loadgen -url http://localhost:8080/ -h2 -h2-conns 16 -h2-streams 200 -duration 30s
# Rated latency measurement at 20k req/s.
loadgen -url http://localhost:8080/ -rate 20000 -duration 60s
# WebSocket echo latency.
loadgen -url http://localhost:8080/ws -mode ws-echo -duration 30s
# HTTPS with an open-loop cap and a result file.
loadgen -url https://api.example.com/health -insecure -max-rps 5000 -duration 60s -out result.jsonStarts each connection as HTTP/1.1 carrying Connection: Upgrade, HTTP2-Settings + Upgrade: h2c, reads a 101 Switching Protocols response, then switches to HTTP/2 on the same TCP socket. This exercises the cleartext upgrade path that -h2 skips (prior-knowledge sends the H2 preface directly). Mutually exclusive with -h2 and -mix. Only defined over cleartext HTTP — TLS servers negotiate H2 via ALPN.
# Basic h2c upgrade run.
loadgen -url http://localhost:8080/ -h2c-upgrade -duration 30s
# h2c upgrade with browser-style fan-out.
loadgen -url http://localhost:8080/ -h2c-upgrade -h2-conns 16 -h2-streams 200 -duration 30sThe result includes an upgrade block, and stderr prints h2c upgrade: X/Y conns upgraded successfully.
Assigns each connection to a protocol by weighted random draw across H1, H2 prior-knowledge, and h2c-upgrade. Connections commit to their chosen protocol for life. Mutually exclusive with -h2 and -h2c-upgrade. Format: h1:h2:upgrade=N:N:N (the h1:h2:upgrade= prefix is optional; a bare N:N:N also parses). Weights are non-negative integers; 0:0:0 is rejected.
# Equal fan-out across all three protocols.
loadgen -url http://localhost:8080/ -mix h1:h2:upgrade=1:1:1 -duration 30s
# Browser-heavy H2 with a trickle of legacy upgrades.
loadgen -url http://localhost:8080/ -mix h1:h2:upgrade=4:4:1 -duration 60sThe mix block in the result reports per-protocol connection, request, and error counts; stderr prints a matching breakdown.
cmd/loadgen is built to be invoked remotely (over SSH or by an orchestrator like Ansible) on a dedicated load-generation host. The contract:
| Aspect | Behavior |
|---|---|
| stdout | A single pretty-printed JSON Result object — no progress noise, no log lines. |
| stderr | Human-readable progress (<elapsed> <reqs> <rps>), warmup and mix/upgrade summaries, error traces. |
| exit 0 | Benchmark ran to completion (per-run errors are reported inside the JSON, not via exit code). |
| exit non-zero | Configuration error (-url missing, mutually-exclusive flags, unreadable body-file, etc.). |
| SIGINT / SIGTERM | Cancels the run gracefully — the partial Result is still printed to stdout, exit 0. |
The shape is loadgen.Result in results.go. Fields marked (optional) are omitted when empty (omitempty).
| Field | Type | Meaning |
|---|---|---|
requests |
int64 | Successful requests during the measurement window. |
errors |
int64 | Errors during the measurement window. |
duration |
duration (ns) | Wall-clock measurement duration (post-warmup). |
requests_per_sec |
float64 | requests / duration.Seconds(). |
throughput_bps |
float64 | Response-body bytes per second. |
latency |
object | {avg, min, max, p50, p75, p90, p99, p99_9, p99_99}, each a duration in ns. |
loadgen_version |
string | loadgen build that produced the run. (optional) |
mode |
string | "saturation" or "rated". (optional) |
rated_mode |
bool | True when driven by the constant-rate scheduler. (optional) |
target_rps |
float64 | Requested rate; meaningful only when rated_mode. (optional) |
histogram |
base64 bytes | V2-compressed HdrHistogram of the full distribution (1µs–30s, 3 sig-digits). (optional) |
client_cpu_percent |
float64 | Client CPU% over the run, from /proc/stat. (optional) |
cpu_pct_p95 |
float64 | P95 of the 1Hz process-CPU sampler. (optional) |
recvq_high |
bool | True when the recv-queue probe latched (loadgen-side backpressure). (optional) |
dial_retries |
uint64 | TCP SYN retries after an RST (listener-replacement / engine-switch window). (optional) |
connect_errors |
uint64 | Dial/handshake failures (TCP, TLS, WS/SSE upgrade, H1 reconnect). (optional) |
timeseries |
array | 1-second snapshots: {t, rps, p99_ms, errors, connect_errors}. (optional) |
warmup |
object | Warmup-phase {requests, errors, connect_errors}; a zero-request/nonzero-error warmup means the target was never healthy. (optional) |
upgrade |
object | h2c-upgrade handshake tally (-h2c-upgrade). (optional) |
mix |
object | Per-protocol connection/request/error counts (-mix). (optional) |
federation |
object | Federated-run outcome (role, peer, peer_requests, peer_errors, merge_succeeded). (optional) |
GitHub Releases ship platform tarballs that orchestrators can fetch directly:
TAG=v1.4.13
OS=linux
ARCH=amd64
curl -fsSL "https://github.com/goceleris/loadgen/releases/download/${TAG}/loadgen_${OS}_${ARCH}.tar.gz" \
| tar xz -C /tmp/
chmod +x /tmp/loadgen-${OS}-${ARCH}
/tmp/loadgen-${OS}-${ARCH} -url http://target:8080/ -duration 10sGitHub displays the SHA-256 of each release asset on the release page — gh release download <tag> and the web UI verify checksums automatically, so no separate .sha256 sidecar ships.
The probatorium cluster bench drives this contract: it cross-compiles loadgen (or fetches a release tarball), pushes it to the load host, executes it, and parses the stdout JSON — decoding histogram for cross-cell re-aggregation. See goceleris/probatorium.
┌─────────────────────────────────────────────────────┐
│ Benchmarker │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Worker 0 │ │ Worker 1 │ ... │ Worker N │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ ┌────▼──────────────▼──────────────────▼─────┐ │
│ │ Client Interface │ │
│ │ ┌────────┐ ┌────────┐ ┌─────────────────┐ │ │
│ │ │H1Client│ │H2Client│ │ WS / SSE / custom│ │ │
│ │ └────────┘ └────────┘ └─────────────────┘ │ │
│ └────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────┐ │
│ │ ShardedLatencyRecorder │ │
│ │ [Shard 0] [Shard 1] ... [Shard N] │ │
│ │ (per-worker HdrHistogram, merged at end) │ │
│ └────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Each worker owns a dedicated TCP connection (keep-alive) or a round-robin pool (Connection: close, PoolSize = 16). Pre-formatted request bytes are written in a single syscall. No synchronization between workers.
N workers share M connections. Each connection has a write goroutine (serializes frame writes, allocates stream IDs) and a read goroutine (dispatches responses to waiting workers via lock-free stream slots). Workers acquire a stream semaphore, submit a write request, and wait on a pooled response channel.
Each worker records into its own shard. The cumulative HdrHistogram is single-writer and lock-free; a separate windowed histogram — used only for the 1-second p99_ms timeseries — takes a short per-shard lock so the ticker goroutine can read-and-reset it without racing the writer. Request and byte counters are batched into plain locals and flushed to atomics every 256 requests (H1) or 16 (H2) to keep ARM64 memory-barrier overhead off the hot path. Shards merge once at completion for percentiles and the exported histogram.
On "zero allocation." The design goal is to keep the request/response cycle allocation-free: request bytes are pre-formatted, HPACK headers are pre-encoded, and H2 status codes are extracted from the encoded header block with no allocation. This is a target the hot path is built around, not a hard invariant — the windowed-histogram path takes a short per-shard lock, and no test asserts literal full-cycle zero-alloc.
The H2 client uses aggressive flow-control settings tuned for benchmark throughput rather than production fairness.
| Parameter | Value | RFC 7540 default | Rationale |
|---|---|---|---|
| Initial window size | 16 MB | 64 KB | Prevents stalls with large response bodies. |
| Max frame size | 64 KB | 16 KB | Balances framing overhead against flow-control granularity. |
| Header table size | 0 | 4096 | Allocation-free status-code extraction from HPACK headers. |
During the handshake the client now captures the connection-level WINDOW_UPDATE(stream 0) a server sends in its preface — before its SETTINGS ack — and seeds connSendWindow with it. Servers such as Kestrel/ASP.NET grow the connection flow-control window this way (65535 → their configured InitialConnectionWindowSize, e.g. 1 MiB). Previously this update was dropped, stranding the send window at the RFC 7540 §6.9.2 floor of 65535 and deadlocking sustained request-body sends. Fixed in commit 29d8ff5 (PR #69).
The read loop accumulates consumed bytes in an atomic counter. The write loop flushes a single WINDOW_UPDATE between request batches and on a 1ms idle ticker, amortizing frame overhead while preventing flow-control stalls.
A semaphore sized to min(server MAX_CONCURRENT_STREAMS, configured MaxStreams) gates in-flight streams; stream slots are allocated at 2× that size for wrap-around headroom during bursts.
- CPU-bound: add connections — more TCP sockets means more kernel parallelism across cores.
- IO-bound: add streams — more multiplexed requests per connection saturates bandwidth.
- Large responses (>1 MB): fewer connections with more streams. The 16 MB window sustains ~16 in-flight 1 MB responses per connection before stalling.
Beyond the flags above, the library Config exposes knobs the CLI leaves at their defaults:
| Field | Default | Notes |
|---|---|---|
DialTimeout |
10s |
TCP connect / reconnect timeout. |
ReadBufferSize / WriteBufferSize |
256 KB (H1), 2 MB (H2) | Larger buffers cut syscall overhead on high-throughput runs. |
PoolSize |
16 |
Connections per worker in Connection: close mode. |
MaxResponseSize |
10 MB | Cap on response-body bytes read; -1 for unlimited. |
TLSConfig |
— | Client certs, custom CA pool, or cipher suites for HTTPS. |
See the package reference for the full Config and Result types.
For protocols loadgen does not ship (gRPC, QUIC, …), implement the exported Client interface and inject it via Config.Client. When set, the built-in H1/H2 client creation is skipped and your client drives every worker.
type Client interface {
DoRequest(ctx context.Context, workerID int) (bytesRead int, err error)
Close()
}
cfg := loadgen.Config{
URL: "http://localhost:50051/", // still required: Validate() enforces an http/https URL
Duration: 15 * time.Second,
Workers: 64,
Client: &myGRPCClient{}, // implements loadgen.Client
}Even with a custom Client, Config.URL must be a valid http/https URL — Config.Validate() enforces it before your client runs; your client is free to reinterpret the target however it needs. WebSocket and SSE are built in — reach for -mode / Config.Mode rather than a custom client for those.
| Feature | loadgen | wrk | h2load | k6 |
|---|---|---|---|---|
| HTTP/1.1 | Custom client | Lua-scriptable | Limited | Full net/http |
| HTTP/2 | Custom multiplexed | Not supported | nghttp2-based | Full net/http |
| WebSocket / SSE | Built-in modes | Not supported | Not supported | Scriptable |
| HTTPS / TLS | ALPN negotiation | Yes | Yes | Yes |
| Scripting | Go (compile-time) | Lua | None | JavaScript |
| Client overhead | Minimal hot path | Low | Low | Higher |
| Rate limiting | -max-rps + -rate (CO-corrected) |
None | None | Built-in |
| Use case | Server throughput/latency measurement | H1 benchmarking | H2 benchmarking | General load testing |
loadgen is focused on measuring a server with minimal client overhead. It is not a general-purpose load-testing framework — for scripted user journeys, reach for k6.
Apache License 2.0 — see LICENSE.
