pcopy

[jdatcmd]

as COPY is single threaded, let's consider creating a new command with pgColumnar, pcopy . It would take the same parameters as copy with one additional parameter W(orkers), where W = n number of workers to launch to appropriately load large files over multiple background workers.

[jdatcmd]

Adversarial read of this one, because I think most of what it asks for already
shipped and the issue does not know it.

The request is "the same parameters as COPY with one additional parameter
W(orkers), where W = n number of workers to launch to appropriately load large
files over multiple background workers". What is in main today:

pgcolumnar.parallel_copy(target regclass, filename text, workers int DEFAULT NULL)
  RETURNS bigint

Background workers, a worker count, atomic, and it also handles the
RANGE-partitioned case with one distinct partition set per worker (#300). On
ChronicallyJD's 11.1M-row ClickBench fixture it loads the same data in 89.1s
against heap's 144.5s, so it is not merely present, it is 1.6x faster than
heap COPY rather than slower.

So the substance is delivered. What is genuinely missing is the surface: a
pcopy statement rather than a function call.

And that part deserves a straight answer rather than an estimate, because it is
not a matter of effort. PostgreSQL's grammar is not extensible — an extension
cannot add a new top-level SQL command. ProcessUtility_hook can only intercept
statements the core parser already parses, so pcopy ... would fail at parse
time before any hook of ours ran. The routes that do exist are all worse than the
function: shadowing COPY itself through the utility hook (surprising, and it
changes the behaviour of a core command), or a client-side rewrite (not the
server). This is adjacent to [[never-change-wal-behavior]] in kind: a design that
needs core to change is not a deferred task, it is a different project.

My suggestion is to close this as delivered-by-parallel_copy and, if the
ergonomics are the real ask, open a narrow issue for those instead — the
workers int DEFAULT NULL default, an error message that names the file and the
worker count, and documentation that puts parallel_copy where someone looking
for "fast COPY" will actually find it. Happy to be told the command syntax
matters more than I think, but I would rather say plainly that we cannot have it
than leave it open as though we could.

One correction while I am here, since it bears on the ranking: #445 says Citus
loads 3x faster than we do. On the same fixture that gap is a property of the
serial loader, not of columnar storage — serial is 3.1x slower than heap
while parallel_copy is 1.6x faster. Measured by ChronicallyJD, not by me.