From af35872a644091f322805aa66fd5c313a06053f2 Mon Sep 17 00:00:00 2001 From: Polyglot AI <293096396+polyglotAI-bot@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:26:02 +0000 Subject: [PATCH 1/4] feat(client-v2): decode QBit in the Native format Reading a plain top-level QBit(Float32|Float64|BFloat16, dimension) column over the Native format previously failed fast: the server transmits QBit there using its internal bit-plane-transposed Tuple(FixedString) layout rather than the Array-like encoding used over RowBinary, which the reader could not decode. Reconstruct the vector by reversing that transposition (the exact inverse of the server's SerializationQBit transpose): read the element_size bit-plane FixedString columns column-major and, for each row, scatter each plane's bits back to their element/bit positions. Values are materialized into the same float[]/double[] ArrayValue representation as the RowBinary path, so a QBit column round-trips identically through either format. Strided QBit(E, dim, stride), Nullable/LowCardinality-wrapped QBit, and QBit nested inside a container remain rejected with a clear error directing callers to a RowBinary format. Implements: https://github.com/ClickHouse/clickhouse-java/issues/2610 --- CHANGELOG.md | 8 +- .../api/data_formats/NativeFormatReader.java | 78 ++++++++--- .../internal/BinaryStreamReader.java | 92 ++++++++++++ .../client/datatypes/DataTypeTests.java | 132 +++++++++++++++--- docs/features.md | 2 +- 5 files changed, 272 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 02f2674ea..dae72a8d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,8 +23,12 @@ binary type encoding is read back to the concrete `QBit(...)` type). In the JDBC driver (`jdbc-v2`) `QBit` maps to `java.sql.Types.ARRAY` and is returned as a `java.sql.Array` from `getObject`/`getArray`. Previously `QBit` was an unimplemented type constant and reading or writing such a column - failed. Reading `QBit` through the `Native` output format is not supported — the server transmits it there using a - different internal layout — and fails fast with a clear error; use a `RowBinary` format instead. + failed. A plain top-level `QBit` column is also read through the `Native` output format: there the server transmits + it using its internal bit-plane-transposed `Tuple(FixedString(...))` layout, and the client reverses that + transposition to reconstruct the same `float[]`/`double[]` vector as `RowBinary`. A `QBit` that is strided + (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, or nested inside another type + (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`) is not yet decoded over `Native` and fails fast with a clear error + directing you to a `RowBinary` format such as `RowBinaryWithNamesAndTypes`. (https://github.com/ClickHouse/clickhouse-java/issues/2610) - **[client-v2, jdbc-v2]** Added TLS cipher suite selection. `Client.Builder.setSSLCipherSuites(String...)` (client-v2) and the comma-separated `ssl_cipher_suites` connection property (client-v2 and jdbc-v2) restrict the cipher suites diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java index 6bd910f99..01472df9c 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java @@ -91,23 +91,26 @@ private boolean readBlock() throws IOException { names.add(column.getColumnName()); types.add(column.getDataType().name()); - if (containsQBit(column)) { - // QBit is transmitted in the Native format using its internal bit-transposed - // Tuple(FixedString(...)) layout, which is NOT the Array(element_type)-like - // representation used in RowBinary (the only representation this reader decodes for - // QBit). Reading it through the columnar/per-row paths below would misread those bytes - // and desynchronize the block, corrupting the columns that follow. Fail loudly instead - // of silently decoding garbage. This also covers a QBit nested inside another type - // (e.g. Map(String, QBit(...))). QBit can be read through a RowBinary format. + List values; + if (isNativeDecodableQBit(column)) { + // A plain top-level QBit column is transmitted in the Native format using its internal + // bit-plane-transposed layout, decoded here into one vector per row (see + // BinaryStreamReader#readQBitColumn). + values = binaryStreamReader.readQBitColumn(column, nRows); + } else if (containsQBit(column)) { + // A QBit that is strided, wrapped in Nullable/LowCardinality, or nested inside another + // type (e.g. Map(String, QBit(...))) uses a Native layout this reader does not decode. + // Reading it through the columnar/per-row paths below would misread those bytes and + // desynchronize the block, corrupting the columns that follow, so fail loudly instead of + // silently decoding garbage. Such QBit values can still be read through a RowBinary format. throw new ClientException("Reading column '" + column.getColumnName() + "' (" - + column.getOriginalTypeName() + ") from the Native format is not supported " - + "because it contains a QBit type: QBit is serialized in the Native format " - + "using an internal layout this reader does not decode. Use a RowBinary format " - + "(e.g. RowBinaryWithNamesAndTypes) to read QBit values"); - } - - List values = new ArrayList<>(nRows); - if (column.isArray()) { + + column.getOriginalTypeName() + ") from the Native format is not supported: " + + "this reader decodes a plain top-level QBit column but not a QBit that is " + + "strided, wrapped in Nullable/LowCardinality, or nested inside another type " + + "(e.g. Array/Tuple/Map). Use a RowBinary format (e.g. RowBinaryWithNamesAndTypes) " + + "to read such QBit values"); + } else if (column.isArray()) { + values = new ArrayList<>(nRows); int[] sizes = new int[nRows]; for (int j = 0; j < nRows; j++) { sizes[j] = Math.toIntExact(binaryStreamReader.readLongLE()); @@ -116,6 +119,7 @@ private boolean readBlock() throws IOException { values.add(binaryStreamReader.readArrayItem(column.getNestedColumns().get(0), sizes[0])); } } else { + values = new ArrayList<>(nRows); for (int j = 0; j < nRows; j++) { Object value = binaryStreamReader.readValue(column); values.add(value); @@ -131,13 +135,47 @@ private boolean readBlock() throws IOException { return true; } + /** + * Returns {@code true} when {@code column} is a plain top-level {@code QBit} column that this + * reader can decode from the Native bit-plane layout: a {@code QBit(Float32|Float64|BFloat16, + * dimension)} that is not {@code Nullable}/{@code LowCardinality} and carries no stride argument. + *

+ * A strided {@code QBit(element, dimension, stride)} (whose Native layout has + * {@code element_size * (dimension / stride)} bit planes) and a {@code QBit} wrapped in + * {@code Nullable}/{@code LowCardinality} or nested inside another type are intentionally excluded + * here; those are caught by {@link #containsQBit} and rejected in {@link #readBlock}. + */ + private static boolean isNativeDecodableQBit(ClickHouseColumn column) { + if (column.getDataType() != ClickHouseDataType.QBit + || column.isNullable() || column.isLowCardinality() + || column.getNestedColumns().isEmpty()) { + return false; + } + // A strided QBit carries a third type argument (element type, dimension, stride); its Native + // layout has element_size * (dimension / stride) bit planes, which this decoder does not + // reconstruct. A non-strided QBit only ever has [element type, dimension]. + if (column.getParameters().size() > 2) { + return false; + } + switch (column.getNestedColumns().get(0).getDataType()) { + case Float32: + case Float64: + case BFloat16: + return true; + default: + return false; + } + } + /** * Returns {@code true} if {@code column} is a {@code QBit} or contains a {@code QBit} anywhere in * its nested type tree (e.g. {@code Array(QBit(...))}, {@code Tuple(..., QBit(...))}, - * {@code Map(String, QBit(...))}). {@code QBit} uses a different, internal wire layout in the - * Native format than in RowBinary, so this reader cannot decode it and rejects such columns - * up-front rather than misreading the block. {@code Nullable}/{@code LowCardinality} wrappers are - * flags on the column, so a wrapped {@code QBit} still reports {@code dataType == QBit} here. + * {@code Map(String, QBit(...))}). A plain top-level {@code QBit} is decoded (see + * {@link #isNativeDecodableQBit}); this predicate additionally flags the QBit variants this reader + * does not decode in the Native format — strided, {@code Nullable}/{@code LowCardinality}-wrapped, + * or container-nested QBit — so {@link #readBlock} can reject them up-front rather than misreading + * the block. {@code Nullable}/{@code LowCardinality} wrappers are flags on the column, so a wrapped + * {@code QBit} still reports {@code dataType == QBit} here. */ private static boolean containsQBit(ClickHouseColumn column) { if (column.getDataType() == ClickHouseDataType.QBit) { diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java index 7a244e956..dad602e16 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java @@ -662,6 +662,98 @@ private T readQBit(ClickHouseColumn column, Class typeHint) throws IOExce return convertArray(array, typeHint == null ? arrayDefaultTypeHint : typeHint); } + /** + * Reads a plain, non-strided {@code QBit(element_type, dimension)} column from the Native + * (columnar) format, decoding the server's internal bit-plane-transposed representation into one + * vector per row of the current block. + *

+ * Unlike RowBinary — where a {@code QBit} is transmitted element-by-element exactly like + * {@code Array(element_type)} (see {@link #readQBit}) — the Native format serializes a + * {@code QBit} column as its internal {@code Tuple(FixedString(S), ...)} of {@code element_size} + * bit planes, column-major: each plane holds {@code S = ceil(dimension / 8)} bytes per row and the + * planes are laid out one after another for the whole block + * ({@code plane0[row0..rowN-1] plane1[row0..rowN-1] ...}). Bit plane {@code p} carries bit + * {@code (element_size - 1 - p)} of every element (most-significant plane first); within a plane's + * {@code S} bytes the element at index {@code j} sits at byte {@code (bitIndex >> 3)}, bit + * {@code (bitIndex & 7)}, where {@code bitIndex = (S * 8 - 1) - (j ^ 7)} (the {@code ^ 7} flips the + * order inside each group of eight). This is the exact inverse of ClickHouse's + * {@code SerializationQBit::transposeBits}. + *

+ * Values are materialized identically to {@link #readQBit} — a {@code float[]} for + * {@code Float32}/{@code BFloat16} and a {@code double[]} for {@code Float64}, wrapped through + * {@link #convertArray} with the same default type hint — so a {@code QBit} read over the Native + * and RowBinary formats yields equal values. + * + * @param column QBit column information (element type in the nested column, dimension in precision) + * @param nRows number of rows in the current block + * @return one materialized vector per row, in row order + * @throws IOException when an IO error occurs + */ + public List readQBitColumn(ClickHouseColumn column, int nRows) throws IOException { + ClickHouseDataType elementType = column.getNestedColumns().get(0).getDataType(); + final boolean isDouble = elementType == ClickHouseDataType.Float64; + final boolean isBFloat16 = elementType == ClickHouseDataType.BFloat16; + if (!isDouble && !isBFloat16 && elementType != ClickHouseDataType.Float32) { + throw new ClientException("QBit Native decoding supports only Float32, Float64 and BFloat16 " + + "element types, got: " + elementType); + } + + final int elementBits = elementType.getByteLength() * 8; // 16 (BFloat16), 32 (Float32), 64 (Float64) + final int dimension = column.getPrecision(); + final int bytesPerPlane = (dimension + 7) / 8; + final int totalBits = bytesPerPlane * 8; + + // The nested Tuple(FixedString(bytesPerPlane)) is serialized column-major, so each of the + // element_size bit planes occupies nRows * bytesPerPlane contiguous bytes; row r's slice for a + // plane starts at r * bytesPerPlane. + byte[][] planes = new byte[elementBits][]; + for (int p = 0; p < elementBits; p++) { + planes[p] = readNBytes(input, nRows * bytesPerPlane); + } + + // Precompute each element's byte offset and bit mask within a plane row (constant across + // planes and rows), reversing the server's (j ^ 7) row-flip and MSB-first FixedString layout. + int[] elementByte = new int[dimension]; + int[] elementMask = new int[dimension]; + for (int j = 0; j < dimension; j++) { + int bitIndex = (totalBits - 1) - (j ^ 7); + elementByte[j] = bitIndex >> 3; + elementMask[j] = 1 << (bitIndex & 7); + } + + List values = new ArrayList<>(nRows); + for (int r = 0; r < nRows; r++) { + final int rowBase = r * bytesPerPlane; + long[] bits = new long[dimension]; + for (int p = 0; p < elementBits; p++) { + final byte[] plane = planes[p]; + final long planeBit = 1L << (elementBits - 1 - p); + for (int j = 0; j < dimension; j++) { + if ((plane[rowBase + elementByte[j]] & elementMask[j]) != 0) { + bits[j] |= planeBit; + } + } + } + + ArrayValue vector; + if (isDouble) { + vector = new ArrayValue(double.class, dimension); + for (int j = 0; j < dimension; j++) { + vector.set(j, Double.longBitsToDouble(bits[j])); + } + } else { + vector = new ArrayValue(float.class, dimension); + for (int j = 0; j < dimension; j++) { + // BFloat16 carries the high 16 bits of a Float32; widen it the same way readBFloat16LE does. + int intBits = isBFloat16 ? ((int) bits[j] << 16) : (int) bits[j]; + vector.set(j, Float.intBitsToFloat(intBits)); + } + } + values.add(convertArray(vector, arrayDefaultTypeHint)); + } + return values; + } + /** * Reads a array into an ArrayValue object. * @param column - column information diff --git a/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java b/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java index 5e330a6a3..cdb322e11 100644 --- a/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java @@ -200,30 +200,18 @@ public void testQBit() throws Exception { } @Test(groups = {"integration"}) - public void testQBitNativeFormatRejected() throws Exception { + public void testQBitNativeFormatNestedRejected() throws Exception { if (isVersionMatch(QBIT_UNSUPPORTED_VERSIONS)) { throw new SkipException("QBit requires ClickHouse 25.10+"); } - // In the Native format the server transmits a QBit column using its internal bit-transposed - // layout, which is NOT the Array(element_type)-like representation the client decodes for QBit - // over RowBinary. Reading QBit via Native must therefore fail loudly with a clear error rather - // than silently decoding garbage and misaligning the trailing column that follows it. + // A plain top-level QBit column IS decoded from the Native format (see testQBitNativeFormatRead), + // but a QBit nested inside another type (here Map(String, QBit)) uses a Native layout this reader + // does not decode. Reading it must fail loudly with a clear error rather than silently decoding + // garbage and misaligning the columns that follow it. QuerySettings settings = new QuerySettings() .setFormat(ClickHouseFormat.Native) .serverSetting("allow_experimental_qbit_type", "1"); - try (QueryResponse response = client.query( - "SELECT CAST([1, 2, 3, 4, 5, 6, 7, 8] AS QBit(Float32, 8)) AS q, 42 AS tail", settings).get()) { - ClientException ex = Assert.expectThrows(ClientException.class, - () -> client.newBinaryFormatReader(response)); - Assert.assertTrue(ex.getMessage().contains("QBit"), - "Expected a clear QBit message, got: " + ex.getMessage()); - Assert.assertTrue(ex.getMessage().contains("Native"), - "Expected the message to mention the Native format, got: " + ex.getMessage()); - } - - // The same rejection applies to a QBit nested inside another type (here Map(String, QBit)), - // which the server does support and would otherwise be misread column-by-column. try (QueryResponse response = client.query( "SELECT CAST(map('a', [1, 2, 3]) AS Map(String, QBit(Float32, 3))) AS m", settings).get()) { ClientException ex = Assert.expectThrows(ClientException.class, @@ -235,6 +223,116 @@ public void testQBitNativeFormatRejected() throws Exception { } } + @DataProvider(name = "qbitNativeCases") + public static Object[][] qbitNativeCases() { + // elementType, ClickHouse array literal, expected vector, isDouble. + // Dimensions deliberately span dim<8 (single-byte bit plane), dim not a multiple of 8 with + // dim>8 (two-byte plane, partial last byte), and dim a multiple of 8, plus negative and + // fractional values, across all three supported element types. + return new Object[][] { + {"Float32", "[1, -2, 3.5, 4, 5, 6, 7, 8]", + new float[]{1f, -2f, 3.5f, 4f, 5f, 6f, 7f, 8f}, false}, + {"Float64", "[1, -2, 3.5, 4, 5, 6, 7, 8]", + new double[]{1d, -2d, 3.5d, 4d, 5d, 6d, 7d, 8d}, true}, + // Integers up to 256 are exactly representable in BFloat16, so these round-trip bit-for-bit. + {"BFloat16", "[1, 2, 4, 8, 16, 32, 64, 128]", + new float[]{1f, 2f, 4f, 8f, 16f, 32f, 64f, 128f}, false}, + {"Float32", "[1, -2, 3.5]", new float[]{1f, -2f, 3.5f}, false}, + {"Float32", "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]", + new float[]{1f, 2f, 3f, 4f, 5f, 6f, 7f, 8f, 9f, 10f}, false}, + {"Float64", "[-1, 2, -3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]", + new double[]{-1d, 2d, -3d, 4d, 5d, 6d, 7d, 8d, 9d, 10d, 11d, 12d, 13d, 14d, 15d, 16d}, true}, + }; + } + + @Test(groups = {"integration"}, dataProvider = "qbitNativeCases") + public void testQBitNativeFormatRead(String elementType, String valueLiteral, Object expectedVector, + boolean isDouble) throws Exception { + if (isVersionMatch(QBIT_UNSUPPORTED_VERSIONS)) { + throw new SkipException("QBit requires ClickHouse 25.10+"); + } + + int dimension = isDouble ? ((double[]) expectedVector).length : ((float[]) expectedVector).length; + // rowId ... vec ... tail: the fixed-width columns around the QBit shift (and the assertions + // fail) if the bit-plane decode consumes the wrong number of bytes and desynchronizes the block. + String sql = "SELECT toInt64(7) AS rowId, CAST(" + valueLiteral + " AS QBit(" + elementType + ", " + + dimension + ")) AS vec, toInt32(42) AS tail SETTINGS allow_experimental_qbit_type = 1"; + + // Native: the QBit column arrives in the server's internal bit-plane-transposed layout and must + // be reconstructed to the same vector as RowBinary. + assertSingleQBitRow(sql, ClickHouseFormat.Native, expectedVector, isDouble); + // Parity: the same query over a RowBinary format must decode to an equal vector. + assertSingleQBitRow(sql, ClickHouseFormat.RowBinaryWithNamesAndTypes, expectedVector, isDouble); + } + + private void assertSingleQBitRow(String sql, ClickHouseFormat format, Object expectedVector, + boolean isDouble) throws Exception { + QuerySettings settings = new QuerySettings().setFormat(format); + try (QueryResponse response = client.query(sql, settings).get()) { + ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + Assert.assertEquals(reader.getLong("rowId"), 7L); + if (isDouble) { + Assert.assertEquals(reader.getDoubleArray("vec"), (double[]) expectedVector); + } else { + Assert.assertEquals(reader.getFloatArray("vec"), (float[]) expectedVector); + } + Assert.assertEquals(reader.getInteger("tail"), 42); + Assert.assertFalse(reader.hasNext()); + } + } + + @Test(groups = {"integration"}) + public void testQBitNativeFormatMultiRow() throws Exception { + if (isVersionMatch(QBIT_UNSUPPORTED_VERSIONS)) { + throw new SkipException("QBit requires ClickHouse 25.10+"); + } + + // In the Native format a QBit column is serialized column-major (each bit plane holds all rows' + // bytes contiguously). Reading several rows in one block exercises the per-row slicing across + // the shared bit planes; the trailing Int32 guards against desync. Covered for a single-byte + // bit plane (Float32, dimension 3) and for a wider element with a two-byte, partially-filled + // plane (Float64, dimension 10), so the per-row offset arithmetic is checked for both + // bytes-per-plane = 1 and 2. + List f32Rows = Arrays.asList( + new float[]{1f, 2f, 3f}, new float[]{4f, 5f, 6f}, new float[]{7f, 8f, 9f}); + assertMultiRowQBit("SELECT number AS rowId, " + + "CAST([number * 3 + 1, number * 3 + 2, number * 3 + 3] AS QBit(Float32, 3)) AS vec, " + + "toInt32(42) AS tail FROM numbers(3) ORDER BY rowId " + + "SETTINGS allow_experimental_qbit_type = 1", f32Rows, false); + + List f64Rows = new ArrayList<>(); + for (int r = 0; r < 3; r++) { + double[] row = new double[10]; + for (int j = 0; j < 10; j++) { + row[j] = (j + 1) + r * 10; + } + f64Rows.add(row); + } + assertMultiRowQBit("SELECT number AS rowId, " + + "CAST(arrayMap(x -> x + number * 10, range(1, 11)) AS QBit(Float64, 10)) AS vec, " + + "toInt32(42) AS tail FROM numbers(3) ORDER BY rowId " + + "SETTINGS allow_experimental_qbit_type = 1", f64Rows, true); + } + + private void assertMultiRowQBit(String sql, List expectedPerRow, boolean isDouble) throws Exception { + QuerySettings settings = new QuerySettings().setFormat(ClickHouseFormat.Native); + try (QueryResponse response = client.query(sql, settings).get()) { + ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response); + for (int r = 0; r < expectedPerRow.size(); r++) { + Assert.assertNotNull(reader.next()); + Assert.assertEquals(reader.getLong("rowId"), (long) r); + if (isDouble) { + Assert.assertEquals(reader.getDoubleArray("vec"), (double[]) expectedPerRow.get(r)); + } else { + Assert.assertEquals(reader.getFloatArray("vec"), (float[]) expectedPerRow.get(r)); + } + Assert.assertEquals(reader.getInteger("tail"), 42); + } + Assert.assertFalse(reader.hasNext()); + } + } + @Test(groups = {"integration"}) public void testQBitInDynamicColumn() throws Exception { if (isVersionMatch(QBIT_UNSUPPORTED_VERSIONS)) { diff --git a/docs/features.md b/docs/features.md index 128321eff..c563774ce 100644 --- a/docs/features.md +++ b/docs/features.md @@ -22,7 +22,7 @@ This document lists stable, user-visible behavior in `client-v2` and `jdbc-v2` t - JSONEachRow text reader: Can stream `JSONEachRow` responses through a caller-supplied `JsonParser`, with Jackson and Gson parser factory implementations available as optional classpath dependencies, and infers a best-effort schema from the first row. - Data type conversion: Maps ClickHouse types to Java values for binary reads, POJO binding, and SQL parameter formatting, including date/time handling. - BFloat16 type support: For ClickHouse `24.11+`, reads and writes the `BFloat16` type through generic records, binary readers, POJO binding, and `Nullable`/`Dynamic`/`Variant` wrappers. `BFloat16` maps to the Java `float` type: a read widens the stored 16-bit value losslessly, while a write keeps the high 16 bits of the `float`. In `jdbc-v2`, `BFloat16` maps to `java.sql.Types.FLOAT` / `java.lang.Float` and is read and written through the standard `getFloat`/`setFloat` and `getObject` accessors. -- QBit type support: For ClickHouse `25.10+` (requires the `allow_experimental_qbit_type` server setting to create a column), reads and writes the experimental `QBit(element_type, dimension[, stride])` vector type. The type-name parser accepts two or three parameters (the optional third parameter is the stride) and recognizes the documented element types `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that documented set is parsed with a warning rather than rejected, so a newer server-side element type does not require a client change to parse. On the wire a `QBit` value is encoded exactly like `Array(element_type)` (a length-prefixed list of elements), so the client reads and writes it as a Java array of the element type — `float[]` for `BFloat16`/`Float32`, `double[]` for `Float64` — through generic records, binary readers, and POJO binding, using a dedicated `QBit` read/serialize path (the shared `Array`-like wire encoding is an implementation detail, not a type equivalence). A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is decoded on read: its binary type encoding (`0x36 `) is read back to the concrete `QBit(...)` type. The client never infers a `QBit` from a Java value, so writing a `QBit` into a `Dynamic` column is not supported and is rejected with a clear `ClientException` rather than emitting an incomplete tag that would desynchronize the stream. This `Array`-like encoding is what the server uses over `RowBinary` formats; the `Native` format instead transmits `QBit` using its internal bit-transposed `Tuple(FixedString(...))` layout, which the client does not decode, so reading any column that is or contains a `QBit` (including a nested `QBit`, e.g. `Map(String, QBit(...))`) through the `Native` format is rejected with a clear `ClientException` (use a `RowBinary` format such as `RowBinaryWithNamesAndTypes` instead). +- QBit type support: For ClickHouse `25.10+` (requires the `allow_experimental_qbit_type` server setting to create a column), reads and writes the experimental `QBit(element_type, dimension[, stride])` vector type. The type-name parser accepts two or three parameters (the optional third parameter is the stride) and recognizes the documented element types `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that documented set is parsed with a warning rather than rejected, so a newer server-side element type does not require a client change to parse. On the wire a `QBit` value is encoded exactly like `Array(element_type)` (a length-prefixed list of elements), so the client reads and writes it as a Java array of the element type — `float[]` for `BFloat16`/`Float32`, `double[]` for `Float64` — through generic records, binary readers, and POJO binding, using a dedicated `QBit` read/serialize path (the shared `Array`-like wire encoding is an implementation detail, not a type equivalence). A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is decoded on read: its binary type encoding (`0x36 `) is read back to the concrete `QBit(...)` type. The client never infers a `QBit` from a Java value, so writing a `QBit` into a `Dynamic` column is not supported and is rejected with a clear `ClientException` rather than emitting an incomplete tag that would desynchronize the stream. This `Array`-like encoding is what the server uses over `RowBinary` formats; the `Native` format instead transmits `QBit` using its internal bit-plane-transposed `Tuple(FixedString(ceil(dimension/8)))` layout (`element_size` bit planes, most-significant plane first, each plane an `element_size`-independent `FixedString` serialized column-major). A plain top-level `QBit(element_type, dimension)` column is decoded from that layout on read: the client reverses the bit-plane transposition (the exact inverse of the server's `SerializationQBit` transpose) to reconstruct the same `float[]`/`double[]` vector it produces over `RowBinary`, so a `QBit` column round-trips identically through either format. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, or nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`) is not decoded over `Native` and is rejected with a clear `ClientException` (use a `RowBinary` format such as `RowBinaryWithNamesAndTypes` instead). - Geometry type support: For ClickHouse `25.11+`, where `Geometry` changed from a string alias to `Variant(Point, Ring, LineString, MultiLineString, Polygon, MultiPolygon)`, the client reads and writes `Geometry` values through generic records, binary readers, POJO binding, and SQL parameter formatting, using Java array dimensionality to represent the geometry shape. - Insert APIs: Supports inserting registered POJOs, raw streams, and callback-driven writers, with optional column lists and format selection. - Insert controls: Supports insert-specific settings such as deduplication token, query id, compression behavior, and request headers. From 688cf07334dc9d38d293642f9dc115a63a8a8d4a Mon Sep 17 00:00:00 2001 From: Polyglot AI <293096396+polyglotAI-bot@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:51:38 +0000 Subject: [PATCH 2/4] Address review: rename readQBitColumn->readQBitNative, shorten javadoc Per @chernser's review on PR #2983: the method name now reflects that it decodes QBit only in the Native format (dropping "column"), and its javadoc is trimmed to the essentials (the deep byte/bit layout stays in the inline method-body comments). Pure rename + doc change; no behavior change. --- .../api/data_formats/NativeFormatReader.java | 4 +-- .../internal/BinaryStreamReader.java | 27 +++++-------------- 2 files changed, 8 insertions(+), 23 deletions(-) diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java index 01472df9c..efe0b9703 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java @@ -95,8 +95,8 @@ private boolean readBlock() throws IOException { if (isNativeDecodableQBit(column)) { // A plain top-level QBit column is transmitted in the Native format using its internal // bit-plane-transposed layout, decoded here into one vector per row (see - // BinaryStreamReader#readQBitColumn). - values = binaryStreamReader.readQBitColumn(column, nRows); + // BinaryStreamReader#readQBitNative). + values = binaryStreamReader.readQBitNative(column, nRows); } else if (containsQBit(column)) { // A QBit that is strided, wrapped in Nullable/LowCardinality, or nested inside another // type (e.g. Map(String, QBit(...))) uses a Native layout this reader does not decode. diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java index dad602e16..e0d6cbac6 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java @@ -663,33 +663,18 @@ private T readQBit(ClickHouseColumn column, Class typeHint) throws IOExce } /** - * Reads a plain, non-strided {@code QBit(element_type, dimension)} column from the Native - * (columnar) format, decoding the server's internal bit-plane-transposed representation into one - * vector per row of the current block. - *

- * Unlike RowBinary — where a {@code QBit} is transmitted element-by-element exactly like - * {@code Array(element_type)} (see {@link #readQBit}) — the Native format serializes a - * {@code QBit} column as its internal {@code Tuple(FixedString(S), ...)} of {@code element_size} - * bit planes, column-major: each plane holds {@code S = ceil(dimension / 8)} bytes per row and the - * planes are laid out one after another for the whole block - * ({@code plane0[row0..rowN-1] plane1[row0..rowN-1] ...}). Bit plane {@code p} carries bit - * {@code (element_size - 1 - p)} of every element (most-significant plane first); within a plane's - * {@code S} bytes the element at index {@code j} sits at byte {@code (bitIndex >> 3)}, bit - * {@code (bitIndex & 7)}, where {@code bitIndex = (S * 8 - 1) - (j ^ 7)} (the {@code ^ 7} flips the - * order inside each group of eight). This is the exact inverse of ClickHouse's - * {@code SerializationQBit::transposeBits}. - *

- * Values are materialized identically to {@link #readQBit} — a {@code float[]} for - * {@code Float32}/{@code BFloat16} and a {@code double[]} for {@code Float64}, wrapped through - * {@link #convertArray} with the same default type hint — so a {@code QBit} read over the Native - * and RowBinary formats yields equal values. + * Reads a plain, non-strided {@code QBit(element_type, dimension)} column from the Native format, + * decoding the server's internal bit-plane-transposed layout (the exact inverse of ClickHouse's + * {@code SerializationQBit::transposeBits}) into one vector per row of the current block. Values + * are materialized identically to the RowBinary path ({@link #readQBit}), so a {@code QBit} read + * over either format yields equal values. The per-byte/bit layout is described inline below. * * @param column QBit column information (element type in the nested column, dimension in precision) * @param nRows number of rows in the current block * @return one materialized vector per row, in row order * @throws IOException when an IO error occurs */ - public List readQBitColumn(ClickHouseColumn column, int nRows) throws IOException { + public List readQBitNative(ClickHouseColumn column, int nRows) throws IOException { ClickHouseDataType elementType = column.getNestedColumns().get(0).getDataType(); final boolean isDouble = elementType == ClickHouseDataType.Float64; final boolean isBFloat16 = elementType == ClickHouseDataType.BFloat16; From 9b0eaaa27116a4a6384c0fa1272f07a71695f160 Mon Sep 17 00:00:00 2001 From: Polyglot AI <293096396+polyglotAI-bot@users.noreply.github.com> Date: Thu, 30 Jul 2026 12:32:47 +0000 Subject: [PATCH 3/4] Address review: narrow QBit Native support to float element types, guard plane-size overflow - Reject message + docs (features.md, CHANGELOG) now state that only a plain top-level QBit with a Float32/Float64/BFloat16 element type is decoded over Native; a QBit with any other element type is rejected alongside the strided/Nullable/LowCardinality/nested cases. The server constrains QBit element types to those three, so the non-float case is only reachable via the client's forward-compat lenient type parser. - Guard the per-plane byte count (nRows * ceil(dimension/8)) against int overflow in readQBitNative: compute it as a long and throw a clear ClientException instead of wrapping to a negative/short readNBytes length that would desynchronize the stream. - Add a unit test pinning the overflow guard. --- CHANGELOG.md | 13 +++++----- .../api/data_formats/NativeFormatReader.java | 18 +++++++------- .../internal/BinaryStreamReader.java | 13 ++++++++-- .../internal/BinaryStreamReaderTests.java | 24 +++++++++++++++++++ docs/features.md | 2 +- 5 files changed, 53 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dae72a8d5..9fbb2e133 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,12 +23,13 @@ binary type encoding is read back to the concrete `QBit(...)` type). In the JDBC driver (`jdbc-v2`) `QBit` maps to `java.sql.Types.ARRAY` and is returned as a `java.sql.Array` from `getObject`/`getArray`. Previously `QBit` was an unimplemented type constant and reading or writing such a column - failed. A plain top-level `QBit` column is also read through the `Native` output format: there the server transmits - it using its internal bit-plane-transposed `Tuple(FixedString(...))` layout, and the client reverses that - transposition to reconstruct the same `float[]`/`double[]` vector as `RowBinary`. A `QBit` that is strided - (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, or nested inside another type - (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`) is not yet decoded over `Native` and fails fast with a clear error - directing you to a `RowBinary` format such as `RowBinaryWithNamesAndTypes`. + failed. A plain top-level `QBit` column with a `Float32`, `Float64`, or `BFloat16` element type is also read through + the `Native` output format: there the server transmits it using its internal bit-plane-transposed + `Tuple(FixedString(...))` layout, and the client reverses that transposition to reconstruct the same + `float[]`/`double[]` vector as `RowBinary`. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), + wrapped in `Nullable`/`LowCardinality`, nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`), + or carrying any other element type is not yet decoded over `Native` and fails fast with a clear error directing you + to a `RowBinary` format such as `RowBinaryWithNamesAndTypes`. (https://github.com/ClickHouse/clickhouse-java/issues/2610) - **[client-v2, jdbc-v2]** Added TLS cipher suite selection. `Client.Builder.setSSLCipherSuites(String...)` (client-v2) and the comma-separated `ssl_cipher_suites` connection property (client-v2 and jdbc-v2) restrict the cipher suites diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java index efe0b9703..738c4aed8 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java @@ -98,17 +98,19 @@ private boolean readBlock() throws IOException { // BinaryStreamReader#readQBitNative). values = binaryStreamReader.readQBitNative(column, nRows); } else if (containsQBit(column)) { - // A QBit that is strided, wrapped in Nullable/LowCardinality, or nested inside another - // type (e.g. Map(String, QBit(...))) uses a Native layout this reader does not decode. - // Reading it through the columnar/per-row paths below would misread those bytes and - // desynchronize the block, corrupting the columns that follow, so fail loudly instead of - // silently decoding garbage. Such QBit values can still be read through a RowBinary format. + // A QBit that has a non-float element type, is strided, is wrapped in + // Nullable/LowCardinality, or is nested inside another type (e.g. Map(String, QBit(...))) + // uses a Native layout this reader does not decode. Reading it through the columnar/per-row + // paths below would misread those bytes and desynchronize the block, corrupting the columns + // that follow, so fail loudly instead of silently decoding garbage. Such QBit values can + // still be read through a RowBinary format. throw new ClientException("Reading column '" + column.getColumnName() + "' (" + column.getOriginalTypeName() + ") from the Native format is not supported: " - + "this reader decodes a plain top-level QBit column but not a QBit that is " + + "this reader decodes only a plain top-level QBit column with a Float32, Float64 " + + "or BFloat16 element type. A QBit whose element type is none of those, or that is " + "strided, wrapped in Nullable/LowCardinality, or nested inside another type " - + "(e.g. Array/Tuple/Map). Use a RowBinary format (e.g. RowBinaryWithNamesAndTypes) " - + "to read such QBit values"); + + "(e.g. Array/Tuple/Map), is not decoded. Use a RowBinary format " + + "(e.g. RowBinaryWithNamesAndTypes) to read such QBit values"); } else if (column.isArray()) { values = new ArrayList<>(nRows); int[] sizes = new int[nRows]; diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java index e0d6cbac6..dbef3f379 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java @@ -690,10 +690,19 @@ public List readQBitNative(ClickHouseColumn column, int nRows) throws IO // The nested Tuple(FixedString(bytesPerPlane)) is serialized column-major, so each of the // element_size bit planes occupies nRows * bytesPerPlane contiguous bytes; row r's slice for a - // plane starts at r * bytesPerPlane. + // plane starts at r * bytesPerPlane. Compute that per-plane byte count in a long and guard it: + // both factors come off the wire, and a plain int multiply could overflow to a negative or + // wrapped value, which would then be passed to readNBytes and desynchronize the stream. + final long planeBytesLong = (long) nRows * bytesPerPlane; + if (planeBytesLong > Integer.MAX_VALUE) { + throw new ClientException("QBit Native block too large to decode: " + nRows + " rows x " + + bytesPerPlane + " bytes per bit plane exceeds the maximum array size (" + + Integer.MAX_VALUE + ")"); + } + final int planeBytes = (int) planeBytesLong; byte[][] planes = new byte[elementBits][]; for (int p = 0; p < elementBits; p++) { - planes[p] = readNBytes(input, nRows * bytesPerPlane); + planes[p] = readNBytes(input, planeBytes); } // Precompute each element's byte offset and bit mask within a plane row (constant across diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java index dca014999..e783a51fb 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java @@ -1,5 +1,6 @@ package com.clickhouse.client.api.data_formats.internal; +import com.clickhouse.client.api.ClientException; import com.clickhouse.data.ClickHouseColumn; import com.clickhouse.data.format.BinaryStreamUtils; @@ -310,4 +311,27 @@ private void assertEmptyArrayComponentType(String columnType, Class expectedC Assert.assertEquals(array.getArray().getClass().getComponentType(), expectedComponentType, "Failed for " + columnType); } + + @Test + public void testReadQBitNativeRejectsIntOverflowPlaneSize() { + // Each QBit Native bit plane is nRows * ceil(dimension/8) bytes. With a large dimension and row + // count that product overflows a 32-bit int; the reader must reject it with a clear ClientException + // rather than wrapping to a negative/short length and desynchronizing the stream. The guard fires + // before any bytes are read, so an empty input stream is sufficient. + // dimension 200000 -> 25000 bytes/plane; 90000 rows -> 2_250_000_000 bytes > Integer.MAX_VALUE. + BinaryStreamReader reader = new BinaryStreamReader( + new ByteArrayInputStream(new byte[0]), + TimeZone.getTimeZone("UTC"), + null, + new BinaryStreamReader.CachingByteBufferAllocator(), + false, + null, + false); + + ClickHouseColumn column = ClickHouseColumn.of("vec", "QBit(Float32, 200000)"); + ClientException ex = Assert.expectThrows(ClientException.class, + () -> reader.readQBitNative(column, 90000)); + Assert.assertTrue(ex.getMessage().contains("too large"), + "Expected an overflow rejection message, got: " + ex.getMessage()); + } } diff --git a/docs/features.md b/docs/features.md index c563774ce..3e600b8f2 100644 --- a/docs/features.md +++ b/docs/features.md @@ -22,7 +22,7 @@ This document lists stable, user-visible behavior in `client-v2` and `jdbc-v2` t - JSONEachRow text reader: Can stream `JSONEachRow` responses through a caller-supplied `JsonParser`, with Jackson and Gson parser factory implementations available as optional classpath dependencies, and infers a best-effort schema from the first row. - Data type conversion: Maps ClickHouse types to Java values for binary reads, POJO binding, and SQL parameter formatting, including date/time handling. - BFloat16 type support: For ClickHouse `24.11+`, reads and writes the `BFloat16` type through generic records, binary readers, POJO binding, and `Nullable`/`Dynamic`/`Variant` wrappers. `BFloat16` maps to the Java `float` type: a read widens the stored 16-bit value losslessly, while a write keeps the high 16 bits of the `float`. In `jdbc-v2`, `BFloat16` maps to `java.sql.Types.FLOAT` / `java.lang.Float` and is read and written through the standard `getFloat`/`setFloat` and `getObject` accessors. -- QBit type support: For ClickHouse `25.10+` (requires the `allow_experimental_qbit_type` server setting to create a column), reads and writes the experimental `QBit(element_type, dimension[, stride])` vector type. The type-name parser accepts two or three parameters (the optional third parameter is the stride) and recognizes the documented element types `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that documented set is parsed with a warning rather than rejected, so a newer server-side element type does not require a client change to parse. On the wire a `QBit` value is encoded exactly like `Array(element_type)` (a length-prefixed list of elements), so the client reads and writes it as a Java array of the element type — `float[]` for `BFloat16`/`Float32`, `double[]` for `Float64` — through generic records, binary readers, and POJO binding, using a dedicated `QBit` read/serialize path (the shared `Array`-like wire encoding is an implementation detail, not a type equivalence). A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is decoded on read: its binary type encoding (`0x36 `) is read back to the concrete `QBit(...)` type. The client never infers a `QBit` from a Java value, so writing a `QBit` into a `Dynamic` column is not supported and is rejected with a clear `ClientException` rather than emitting an incomplete tag that would desynchronize the stream. This `Array`-like encoding is what the server uses over `RowBinary` formats; the `Native` format instead transmits `QBit` using its internal bit-plane-transposed `Tuple(FixedString(ceil(dimension/8)))` layout (`element_size` bit planes, most-significant plane first, each plane an `element_size`-independent `FixedString` serialized column-major). A plain top-level `QBit(element_type, dimension)` column is decoded from that layout on read: the client reverses the bit-plane transposition (the exact inverse of the server's `SerializationQBit` transpose) to reconstruct the same `float[]`/`double[]` vector it produces over `RowBinary`, so a `QBit` column round-trips identically through either format. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, or nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`) is not decoded over `Native` and is rejected with a clear `ClientException` (use a `RowBinary` format such as `RowBinaryWithNamesAndTypes` instead). +- QBit type support: For ClickHouse `25.10+` (requires the `allow_experimental_qbit_type` server setting to create a column), reads and writes the experimental `QBit(element_type, dimension[, stride])` vector type. The type-name parser accepts two or three parameters (the optional third parameter is the stride) and recognizes the documented element types `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that documented set is parsed with a warning rather than rejected, so a newer server-side element type does not require a client change to parse. On the wire a `QBit` value is encoded exactly like `Array(element_type)` (a length-prefixed list of elements), so the client reads and writes it as a Java array of the element type — `float[]` for `BFloat16`/`Float32`, `double[]` for `Float64` — through generic records, binary readers, and POJO binding, using a dedicated `QBit` read/serialize path (the shared `Array`-like wire encoding is an implementation detail, not a type equivalence). A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is decoded on read: its binary type encoding (`0x36 `) is read back to the concrete `QBit(...)` type. The client never infers a `QBit` from a Java value, so writing a `QBit` into a `Dynamic` column is not supported and is rejected with a clear `ClientException` rather than emitting an incomplete tag that would desynchronize the stream. This `Array`-like encoding is what the server uses over `RowBinary` formats; the `Native` format instead transmits `QBit` using its internal bit-plane-transposed `Tuple(FixedString(ceil(dimension/8)))` layout (`element_size` bit planes, most-significant plane first, each plane an `element_size`-independent `FixedString` serialized column-major). A plain top-level `QBit(element_type, dimension)` column whose element type is `Float32`, `Float64`, or `BFloat16` (the element types ClickHouse supports for `QBit`) is decoded from that layout on read: the client reverses the bit-plane transposition (the exact inverse of the server's `SerializationQBit` transpose) to reconstruct the same `float[]`/`double[]` vector it produces over `RowBinary`, so a `QBit` column round-trips identically through either format. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`), or carrying any other element type is not decoded over `Native` and is rejected with a clear `ClientException` (use a `RowBinary` format such as `RowBinaryWithNamesAndTypes` instead). - Geometry type support: For ClickHouse `25.11+`, where `Geometry` changed from a string alias to `Variant(Point, Ring, LineString, MultiLineString, Polygon, MultiPolygon)`, the client reads and writes `Geometry` values through generic records, binary readers, POJO binding, and SQL parameter formatting, using Java array dimensionality to represent the geometry shape. - Insert APIs: Supports inserting registered POJOs, raw streams, and callback-driven writers, with optional column lists and format selection. - Insert controls: Supports insert-specific settings such as deduplication token, query id, compression behavior, and request headers. From f71744e79daa657b0e7b1ece7996ceec91fda177 Mon Sep 17 00:00:00 2001 From: Polyglot AI <293096396+polyglotAI-bot@users.noreply.github.com> Date: Thu, 30 Jul 2026 18:31:15 +0000 Subject: [PATCH 4/4] Address review: raise QBit Native decode coverage, shorten comments, add encoding doc - Add unit tests for BinaryStreamReader#readQBitNative covering all supported element types, single/two-byte bit planes, multi-row slicing, special float values (NaN/Inf/-0.0), and the unsupported-element-type and int-overflow guards, driven by Native-format QBit bytes captured from the server (readQBitNative new-code coverage 41% -> 100%). - Add NativeFormatReaderQBitTest covering the block-level decode-vs-reject routing (isNativeDecodableQBit / containsQBit) without a live server. - Move the bit-plane transpose math into docs/qbit-encoding.md and trim the inline comments in readQBitNative / NativeFormatReader to short pointers; link the doc from docs/features.md. Addresses @chernser review feedback on #2983. --- .../api/data_formats/NativeFormatReader.java | 42 +++---- .../internal/BinaryStreamReader.java | 19 ++- .../NativeFormatReaderQBitTest.java | 99 ++++++++++++++++ .../internal/BinaryStreamReaderTests.java | 108 +++++++++++++++--- docs/features.md | 2 +- docs/qbit-encoding.md | 71 ++++++++++++ 6 files changed, 285 insertions(+), 56 deletions(-) create mode 100644 client-v2/src/test/java/com/clickhouse/client/api/data_formats/NativeFormatReaderQBitTest.java create mode 100644 docs/qbit-encoding.md diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java index 738c4aed8..2871571cb 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java @@ -93,17 +93,12 @@ private boolean readBlock() throws IOException { List values; if (isNativeDecodableQBit(column)) { - // A plain top-level QBit column is transmitted in the Native format using its internal - // bit-plane-transposed layout, decoded here into one vector per row (see - // BinaryStreamReader#readQBitNative). + // Decode the Native bit-plane layout (docs/qbit-encoding.md). values = binaryStreamReader.readQBitNative(column, nRows); } else if (containsQBit(column)) { - // A QBit that has a non-float element type, is strided, is wrapped in - // Nullable/LowCardinality, or is nested inside another type (e.g. Map(String, QBit(...))) - // uses a Native layout this reader does not decode. Reading it through the columnar/per-row - // paths below would misread those bytes and desynchronize the block, corrupting the columns - // that follow, so fail loudly instead of silently decoding garbage. Such QBit values can - // still be read through a RowBinary format. + // Any other QBit shape (non-float element, strided, Nullable/LowCardinality, or nested) + // uses a Native layout this reader does not decode; fail loudly rather than misread the + // block and desynchronize the columns that follow (docs/qbit-encoding.md). throw new ClientException("Reading column '" + column.getColumnName() + "' (" + column.getOriginalTypeName() + ") from the Native format is not supported: " + "this reader decodes only a plain top-level QBit column with a Float32, Float64 " @@ -138,14 +133,10 @@ private boolean readBlock() throws IOException { } /** - * Returns {@code true} when {@code column} is a plain top-level {@code QBit} column that this - * reader can decode from the Native bit-plane layout: a {@code QBit(Float32|Float64|BFloat16, - * dimension)} that is not {@code Nullable}/{@code LowCardinality} and carries no stride argument. - *

- * A strided {@code QBit(element, dimension, stride)} (whose Native layout has - * {@code element_size * (dimension / stride)} bit planes) and a {@code QBit} wrapped in - * {@code Nullable}/{@code LowCardinality} or nested inside another type are intentionally excluded - * here; those are caught by {@link #containsQBit} and rejected in {@link #readBlock}. + * Returns {@code true} for a plain top-level {@code QBit(Float32|Float64|BFloat16, dimension)} the + * reader can decode from the Native bit-plane layout — not {@code Nullable}/{@code LowCardinality}, + * not strided, not nested. Other QBit shapes are caught by {@link #containsQBit} and rejected in + * {@link #readBlock} (see {@code docs/qbit-encoding.md}). */ private static boolean isNativeDecodableQBit(ClickHouseColumn column) { if (column.getDataType() != ClickHouseDataType.QBit @@ -153,9 +144,7 @@ private static boolean isNativeDecodableQBit(ClickHouseColumn column) { || column.getNestedColumns().isEmpty()) { return false; } - // A strided QBit carries a third type argument (element type, dimension, stride); its Native - // layout has element_size * (dimension / stride) bit planes, which this decoder does not - // reconstruct. A non-strided QBit only ever has [element type, dimension]. + // A strided QBit has a third parameter (stride) and a plane count this decoder does not handle. if (column.getParameters().size() > 2) { return false; } @@ -170,14 +159,11 @@ private static boolean isNativeDecodableQBit(ClickHouseColumn column) { } /** - * Returns {@code true} if {@code column} is a {@code QBit} or contains a {@code QBit} anywhere in - * its nested type tree (e.g. {@code Array(QBit(...))}, {@code Tuple(..., QBit(...))}, - * {@code Map(String, QBit(...))}). A plain top-level {@code QBit} is decoded (see - * {@link #isNativeDecodableQBit}); this predicate additionally flags the QBit variants this reader - * does not decode in the Native format — strided, {@code Nullable}/{@code LowCardinality}-wrapped, - * or container-nested QBit — so {@link #readBlock} can reject them up-front rather than misreading - * the block. {@code Nullable}/{@code LowCardinality} wrappers are flags on the column, so a wrapped - * {@code QBit} still reports {@code dataType == QBit} here. + * Returns {@code true} if {@code column} is or contains a {@code QBit} anywhere in its nested type + * tree (e.g. {@code Array}/{@code Tuple}/{@code Map(String, QBit(...))}). Used by {@link #readBlock} + * to reject the QBit shapes {@link #isNativeDecodableQBit} does not decode. {@code Nullable}/ + * {@code LowCardinality} are column flags, so a wrapped {@code QBit} still reports + * {@code dataType == QBit} here. */ private static boolean containsQBit(ClickHouseColumn column) { if (column.getDataType() == ClickHouseDataType.QBit) { diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java index dbef3f379..d2ccf7e69 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java @@ -663,11 +663,10 @@ private T readQBit(ClickHouseColumn column, Class typeHint) throws IOExce } /** - * Reads a plain, non-strided {@code QBit(element_type, dimension)} column from the Native format, - * decoding the server's internal bit-plane-transposed layout (the exact inverse of ClickHouse's - * {@code SerializationQBit::transposeBits}) into one vector per row of the current block. Values - * are materialized identically to the RowBinary path ({@link #readQBit}), so a {@code QBit} read - * over either format yields equal values. The per-byte/bit layout is described inline below. + * Reads a plain, non-strided {@code QBit(Float32|Float64|BFloat16, dimension)} column from the + * Native format, reversing the server's bit-plane transpose into one vector per row. Values match + * the RowBinary path ({@link #readQBit}), so a {@code QBit} round-trips equally through either + * format. See {@code docs/qbit-encoding.md} for the wire layout and the transpose math. * * @param column QBit column information (element type in the nested column, dimension in precision) * @param nRows number of rows in the current block @@ -688,11 +687,8 @@ public List readQBitNative(ClickHouseColumn column, int nRows) throws IO final int bytesPerPlane = (dimension + 7) / 8; final int totalBits = bytesPerPlane * 8; - // The nested Tuple(FixedString(bytesPerPlane)) is serialized column-major, so each of the - // element_size bit planes occupies nRows * bytesPerPlane contiguous bytes; row r's slice for a - // plane starts at r * bytesPerPlane. Compute that per-plane byte count in a long and guard it: - // both factors come off the wire, and a plain int multiply could overflow to a negative or - // wrapped value, which would then be passed to readNBytes and desynchronize the stream. + // Planes are column-major: each holds nRows * bytesPerPlane bytes (docs/qbit-encoding.md). + // Size it in a long and guard against int overflow, since both factors come off the wire. final long planeBytesLong = (long) nRows * bytesPerPlane; if (planeBytesLong > Integer.MAX_VALUE) { throw new ClientException("QBit Native block too large to decode: " + nRows + " rows x " @@ -705,8 +701,7 @@ public List readQBitNative(ClickHouseColumn column, int nRows) throws IO planes[p] = readNBytes(input, planeBytes); } - // Precompute each element's byte offset and bit mask within a plane row (constant across - // planes and rows), reversing the server's (j ^ 7) row-flip and MSB-first FixedString layout. + // Per-element byte offset and bit mask within a plane row (docs/qbit-encoding.md). int[] elementByte = new int[dimension]; int[] elementMask = new int[dimension]; for (int j = 0; j < dimension; j++) { diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/NativeFormatReaderQBitTest.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/NativeFormatReaderQBitTest.java new file mode 100644 index 000000000..fc56f81ec --- /dev/null +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/NativeFormatReaderQBitTest.java @@ -0,0 +1,99 @@ +package com.clickhouse.client.api.data_formats; + +import com.clickhouse.client.api.ClientException; +import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; +import com.clickhouse.client.api.query.QuerySettings; +import com.clickhouse.data.format.BinaryStreamUtils; + +import org.testng.Assert; +import org.testng.annotations.DataProvider; +import org.testng.annotations.Test; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; + +/** + * QBit handling in {@link NativeFormatReader}, driven by Native-format blocks captured from ClickHouse + * 26.5.1 and by minimal hand-built block headers. These run without a live server, so the block-level + * routing (decode a plain float QBit, reject every other shape) stays covered on the coverage build, + * whose older server skips the QBit integration tests. + */ +public class NativeFormatReaderQBitTest { + + // Full single-row Native blocks (SELECT CAST( AS QBit()) FORMAT Native), one per decodable + // element type, with the vector each encodes. + @DataProvider(name = "decodableQBitBlocks") + public static Object[][] decodableQBitBlocks() { + return new Object[][] { + {"Float32", + "010103766563105142697428466c6f617433322c2033290206010101010101010404000000000000000000000000000000000000000000", + false, new float[] {1f, -2f, 3.5f}, null}, + {"Float64", + "010103766563105142697428466c6f617436342c20332902060101010101010101010104040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + true, null, new double[] {1d, -2d, 3.5d}}, + {"BFloat16", + "01010376656311514269742842466c6f617431362c20382900fe01010101e1995500000000000000", + false, new float[] {1f, 2f, 4f, 8f, 16f, 32f, 64f, 128f}, null}, + }; + } + + @Test(dataProvider = "decodableQBitBlocks") + public void testDecodesPlainQBitColumn(String label, String blockHex, boolean isDouble, + float[] expectedFloat, double[] expectedDouble) { + NativeFormatReader reader = nativeReader(fromHex(blockHex)); + + Assert.assertNotNull(reader.next(), label); + if (isDouble) { + Assert.assertEquals(reader.getDoubleArray("vec"), expectedDouble, label); + } else { + Assert.assertEquals(reader.getFloatArray("vec"), expectedFloat, label); + } + Assert.assertNull(reader.next(), label); + } + + // QBit shapes the Native reader cannot decode: nested inside a container, strided, wrapped in Nullable, + // or a non-float element type. Each must be rejected up front instead of misreading the block. + @DataProvider(name = "undecodableQBitTypes") + public static Object[][] undecodableQBitTypes() { + return new Object[][] { + {"m", "Map(String, QBit(Float32, 3))"}, + {"vec", "QBit(Float32, 4, 2)"}, + {"vec", "Nullable(QBit(Float32, 3))"}, + {"vec", "QBit(Int8, 3)"}, + }; + } + + @Test(dataProvider = "undecodableQBitTypes") + public void testRejectsUndecodableQBitColumn(String colName, String colType) throws IOException { + // Only the block header (column name + type) is needed: readBlock rejects the column before reading + // any payload. + byte[] block = nativeBlockHeader(colName, colType); + ClientException ex = Assert.expectThrows(ClientException.class, () -> nativeReader(block)); + Assert.assertTrue(ex.getMessage().contains("QBit") && ex.getMessage().contains("Native"), + colType + ": " + ex.getMessage()); + } + + private static NativeFormatReader nativeReader(byte[] block) { + return new NativeFormatReader(new ByteArrayInputStream(block), + new QuerySettings().setUseTimeZone("UTC"), + new BinaryStreamReader.CachingByteBufferAllocator()); + } + + private static byte[] nativeBlockHeader(String columnName, String columnType) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + BinaryStreamUtils.writeVarInt(out, 1); // one column + BinaryStreamUtils.writeVarInt(out, 1); // one row + BinaryStreamUtils.writeString(out, columnName); + BinaryStreamUtils.writeString(out, columnType); + return out.toByteArray(); + } + + private static byte[] fromHex(String hex) { + byte[] out = new byte[hex.length() / 2]; + for (int i = 0; i < out.length; i++) { + out[i] = (byte) Integer.parseInt(hex.substring(2 * i, 2 * i + 2), 16); + } + return out; + } +} diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java index e783a51fb..1904fbe2d 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java @@ -10,6 +10,7 @@ import java.time.ZoneId; import java.time.ZonedDateTime; import java.time.temporal.ChronoUnit; +import java.util.List; import java.util.TimeZone; import org.testng.Assert; @@ -312,26 +313,103 @@ private void assertEmptyArrayComponentType(String columnType, Class expectedC Assert.assertEquals(array.getArray().getClass().getComponentType(), expectedComponentType, "Failed for " + columnType); } + // Native-format QBit column payloads captured from ClickHouse 26.5.1 (the block header stripped, + // leaving the Tuple(FixedString) bit-plane bytes readQBitNative consumes) paired with the vector the + // server encoded. Pins the decode against the server's real layout as a unit test, because the QBit + // integration tests skip on the coverage build's older server. + @DataProvider(name = "qbitNativeGoldenBytes") + public static Object[][] qbitNativeGoldenBytes() { + return new Object[][] { + {"Float32 dim 3", "QBit(Float32, 3)", 1, + "0206010101010101010404000000000000000000000000000000000000000000", + false, new Object[] {new float[] {1f, -2f, 3.5f}}}, + {"Float32 dim 8 (full single-byte plane)", "QBit(Float32, 8)", 1, + "02fe010101010181796454000000000000000000000000000000000000000000", + false, new Object[] {new float[] {1f, -2f, 3.5f, 4f, 5f, 6f, 7f, 8f}}}, + {"Float32 dim 10 (two-byte plane, partial last byte)", "QBit(Float32, 10)", 1, + "02aa03fe000100010001000100010381007900640254010000000000000000000000000000000000000000000000000000000000000000000000000000000000", + false, new Object[] {new float[] {1f, -2f, 3.5f, -4f, 5f, -6f, 7f, -8f, 9f, -10f}}}, + {"Float32 dim 1", "QBit(Float32, 1)", 1, + "0001000000000100000001000100010000000000000000000000000000000000", + false, new Object[] {new float[] {42.5f}}}, + {"Float64 dim 3 (64 bit planes)", "QBit(Float64, 3)", 1, + "02060101010101010101010104040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + true, new Object[] {new double[] {1d, -2d, 3.5d}}}, + {"BFloat16 dim 8 (16 bit planes)", "QBit(BFloat16, 8)", 1, + "00fe01010101e1995500000000000000", + false, new Object[] {new float[] {1f, 2f, 4f, 8f, 16f, 32f, 64f, 128f}}}, + {"Float32 dim 3, three rows (column-major slicing)", "QBit(Float32, 3)", 3, + "000000060707010000010000010000010000010000010006010701040401000201000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + false, new Object[] { + new float[] {1f, 2f, 3f}, new float[] {4f, 5f, 6f}, new float[] {7f, 8f, 9f}}}, + }; + } + + @Test(dataProvider = "qbitNativeGoldenBytes") + public void testReadQBitNativeDecodesGoldenBytes(String label, String columnType, int nRows, + String columnDataHex, boolean isDouble, Object[] expectedRows) throws Exception { + List rows = qbitReader(fromHex(columnDataHex)) + .readQBitNative(ClickHouseColumn.of("vec", columnType), nRows); + + Assert.assertEquals(rows.size(), nRows, label); + for (int r = 0; r < nRows; r++) { + Object vector = ((BinaryStreamReader.ArrayValue) rows.get(r)).getArray(); + if (isDouble) { + Assert.assertEquals((double[]) vector, (double[]) expectedRows[r], label + " row " + r); + } else { + Assert.assertEquals((float[]) vector, (float[]) expectedRows[r], label + " row " + r); + } + } + } + @Test - public void testReadQBitNativeRejectsIntOverflowPlaneSize() { - // Each QBit Native bit plane is nRows * ceil(dimension/8) bytes. With a large dimension and row - // count that product overflows a 32-bit int; the reader must reject it with a clear ClientException - // rather than wrapping to a negative/short length and desynchronizing the stream. The guard fires - // before any bytes are read, so an empty input stream is sufficient. - // dimension 200000 -> 25000 bytes/plane; 90000 rows -> 2_250_000_000 bytes > Integer.MAX_VALUE. - BinaryStreamReader reader = new BinaryStreamReader( - new ByteArrayInputStream(new byte[0]), - TimeZone.getTimeZone("UTC"), - null, - new BinaryStreamReader.CachingByteBufferAllocator(), - false, - null, - false); + public void testReadQBitNativeDecodesSpecialFloatValues() throws Exception { + // NaN, +Inf, -Inf and -0.0 must survive the bit-plane transpose bit-for-bit. + float[] vector = (float[]) ((BinaryStreamReader.ArrayValue) qbitReader(fromHex( + "0c07070707070707070100000000000000000000000000000000000000000000")) + .readQBitNative(ClickHouseColumn.of("vec", "QBit(Float32, 4)"), 1).get(0)).getArray(); + + float[] expected = {Float.NaN, Float.POSITIVE_INFINITY, Float.NEGATIVE_INFINITY, -0.0f}; + Assert.assertEquals(vector.length, expected.length); + for (int j = 0; j < expected.length; j++) { + Assert.assertEquals(Float.floatToRawIntBits(vector[j]), Float.floatToRawIntBits(expected[j]), + "element " + j); + } + } + + @Test + public void testReadQBitNativeRejectsUnsupportedElementType() { + // readQBitNative reconstructs only float element types; a non-float element is rejected by its own + // guard (the Native reader also filters these earlier), which fires before any read. + ClickHouseColumn column = ClickHouseColumn.of("vec", "QBit(Int8, 3)"); + ClientException ex = Assert.expectThrows(ClientException.class, + () -> qbitReader(new byte[0]).readQBitNative(column, 1)); + Assert.assertTrue(ex.getMessage().contains("Float32"), + "Expected an unsupported-element-type message, got: " + ex.getMessage()); + } + @Test + public void testReadQBitNativeRejectsIntOverflowPlaneSize() { + // nRows * ceil(dimension/8) per bit plane must not overflow a 32-bit int; the reader rejects it + // before allocating or reading, so an empty stream is sufficient (200000 -> 25000 bytes/plane, + // 90000 rows -> 2_250_000_000 bytes > Integer.MAX_VALUE). ClickHouseColumn column = ClickHouseColumn.of("vec", "QBit(Float32, 200000)"); ClientException ex = Assert.expectThrows(ClientException.class, - () -> reader.readQBitNative(column, 90000)); + () -> qbitReader(new byte[0]).readQBitNative(column, 90000)); Assert.assertTrue(ex.getMessage().contains("too large"), "Expected an overflow rejection message, got: " + ex.getMessage()); } + + private static BinaryStreamReader qbitReader(byte[] columnData) { + return new BinaryStreamReader(new ByteArrayInputStream(columnData), TimeZone.getTimeZone("UTC"), + null, new BinaryStreamReader.CachingByteBufferAllocator(), false, null, false); + } + + private static byte[] fromHex(String hex) { + byte[] out = new byte[hex.length() / 2]; + for (int i = 0; i < out.length; i++) { + out[i] = (byte) Integer.parseInt(hex.substring(2 * i, 2 * i + 2), 16); + } + return out; + } } diff --git a/docs/features.md b/docs/features.md index 3e600b8f2..7d5350c5c 100644 --- a/docs/features.md +++ b/docs/features.md @@ -22,7 +22,7 @@ This document lists stable, user-visible behavior in `client-v2` and `jdbc-v2` t - JSONEachRow text reader: Can stream `JSONEachRow` responses through a caller-supplied `JsonParser`, with Jackson and Gson parser factory implementations available as optional classpath dependencies, and infers a best-effort schema from the first row. - Data type conversion: Maps ClickHouse types to Java values for binary reads, POJO binding, and SQL parameter formatting, including date/time handling. - BFloat16 type support: For ClickHouse `24.11+`, reads and writes the `BFloat16` type through generic records, binary readers, POJO binding, and `Nullable`/`Dynamic`/`Variant` wrappers. `BFloat16` maps to the Java `float` type: a read widens the stored 16-bit value losslessly, while a write keeps the high 16 bits of the `float`. In `jdbc-v2`, `BFloat16` maps to `java.sql.Types.FLOAT` / `java.lang.Float` and is read and written through the standard `getFloat`/`setFloat` and `getObject` accessors. -- QBit type support: For ClickHouse `25.10+` (requires the `allow_experimental_qbit_type` server setting to create a column), reads and writes the experimental `QBit(element_type, dimension[, stride])` vector type. The type-name parser accepts two or three parameters (the optional third parameter is the stride) and recognizes the documented element types `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that documented set is parsed with a warning rather than rejected, so a newer server-side element type does not require a client change to parse. On the wire a `QBit` value is encoded exactly like `Array(element_type)` (a length-prefixed list of elements), so the client reads and writes it as a Java array of the element type — `float[]` for `BFloat16`/`Float32`, `double[]` for `Float64` — through generic records, binary readers, and POJO binding, using a dedicated `QBit` read/serialize path (the shared `Array`-like wire encoding is an implementation detail, not a type equivalence). A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is decoded on read: its binary type encoding (`0x36 `) is read back to the concrete `QBit(...)` type. The client never infers a `QBit` from a Java value, so writing a `QBit` into a `Dynamic` column is not supported and is rejected with a clear `ClientException` rather than emitting an incomplete tag that would desynchronize the stream. This `Array`-like encoding is what the server uses over `RowBinary` formats; the `Native` format instead transmits `QBit` using its internal bit-plane-transposed `Tuple(FixedString(ceil(dimension/8)))` layout (`element_size` bit planes, most-significant plane first, each plane an `element_size`-independent `FixedString` serialized column-major). A plain top-level `QBit(element_type, dimension)` column whose element type is `Float32`, `Float64`, or `BFloat16` (the element types ClickHouse supports for `QBit`) is decoded from that layout on read: the client reverses the bit-plane transposition (the exact inverse of the server's `SerializationQBit` transpose) to reconstruct the same `float[]`/`double[]` vector it produces over `RowBinary`, so a `QBit` column round-trips identically through either format. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`), or carrying any other element type is not decoded over `Native` and is rejected with a clear `ClientException` (use a `RowBinary` format such as `RowBinaryWithNamesAndTypes` instead). +- QBit type support: For ClickHouse `25.10+` (requires the `allow_experimental_qbit_type` server setting to create a column), reads and writes the experimental `QBit(element_type, dimension[, stride])` vector type. The type-name parser accepts two or three parameters (the optional third parameter is the stride) and recognizes the documented element types `Int8`, `BFloat16`, `Float32`, and `Float64`; an element type outside that documented set is parsed with a warning rather than rejected, so a newer server-side element type does not require a client change to parse. On the wire a `QBit` value is encoded exactly like `Array(element_type)` (a length-prefixed list of elements), so the client reads and writes it as a Java array of the element type — `float[]` for `BFloat16`/`Float32`, `double[]` for `Float64` — through generic records, binary readers, and POJO binding, using a dedicated `QBit` read/serialize path (the shared `Array`-like wire encoding is an implementation detail, not a type equivalence). A `QBit` held inside a `Dynamic`/`Variant`/`JSON` column is decoded on read: its binary type encoding (`0x36 `) is read back to the concrete `QBit(...)` type. The client never infers a `QBit` from a Java value, so writing a `QBit` into a `Dynamic` column is not supported and is rejected with a clear `ClientException` rather than emitting an incomplete tag that would desynchronize the stream. This `Array`-like encoding is what the server uses over `RowBinary` formats; the `Native` format instead transmits `QBit` using its internal bit-plane-transposed `Tuple(FixedString(ceil(dimension/8)))` layout (`element_size` bit planes, most-significant plane first, each plane an `element_size`-independent `FixedString` serialized column-major). A plain top-level `QBit(element_type, dimension)` column whose element type is `Float32`, `Float64`, or `BFloat16` (the element types ClickHouse supports for `QBit`) is decoded from that layout on read: the client reverses the bit-plane transposition (the exact inverse of the server's `SerializationQBit` transpose) to reconstruct the same `float[]`/`double[]` vector it produces over `RowBinary`, so a `QBit` column round-trips identically through either format. A `QBit` that is strided (`QBit(element_type, dimension, stride)`), wrapped in `Nullable`/`LowCardinality`, nested inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`), or carrying any other element type is not decoded over `Native` and is rejected with a clear `ClientException` (use a `RowBinary` format such as `RowBinaryWithNamesAndTypes` instead). See [qbit-encoding.md](qbit-encoding.md) for the RowBinary and Native wire layouts. - Geometry type support: For ClickHouse `25.11+`, where `Geometry` changed from a string alias to `Variant(Point, Ring, LineString, MultiLineString, Polygon, MultiPolygon)`, the client reads and writes `Geometry` values through generic records, binary readers, POJO binding, and SQL parameter formatting, using Java array dimensionality to represent the geometry shape. - Insert APIs: Supports inserting registered POJOs, raw streams, and callback-driven writers, with optional column lists and format selection. - Insert controls: Supports insert-specific settings such as deduplication token, query id, compression behavior, and request headers. diff --git a/docs/qbit-encoding.md b/docs/qbit-encoding.md new file mode 100644 index 000000000..501a4c82c --- /dev/null +++ b/docs/qbit-encoding.md @@ -0,0 +1,71 @@ +# QBit encoding: RowBinary vs Native + +`QBit(element_type, dimension)` is a fixed-length vector type (ClickHouse `25.10+`). The same +logical value is put on the wire two different ways depending on the output format, which is why +`client-v2` has a dedicated read path for each. This document describes both encodings and the +"problem" the Native path has to solve. + +See also: [ClickHouse QBit docs](https://clickhouse.com/docs/sql-reference/data-types/qbit) and the +server serializer `SerializationQBit` (`transposeBits` / `restoreBits`). + +## RowBinary — the simple case + +Over `RowBinary` formats (`RowBinary`, `RowBinaryWithNamesAndTypes`, …) a `QBit(E, N)` value is +encoded **exactly like `Array(E)`**: a var-int length prefix (always `N`) followed by `N` element +values in order. `client-v2` reads it straight into a Java array of the element type — `float[]` for +`BFloat16`/`Float32`, `double[]` for `Float64` — in `BinaryStreamReader#readQBit`. There is no +transposition; the bytes are the elements. (The shared `Array`-like encoding is an implementation +detail of the wire format, not a type equivalence.) + +## Native — the bit-plane transposed layout + +Over the `Native` format the server does **not** send the elements contiguously. It stores a `QBit` +column as a nested `Tuple(FixedString(ceil(N/8)))` with one field per **bit plane**, and serializes +that tuple **column-major**. Concretely, for a block of `nRows` rows: + +- Let `element_size` be the element bit width: `16` (`BFloat16`), `32` (`Float32`), `64` (`Float64`). +- Let `bytesPerPlane = ceil(N / 8)` and `totalBits = bytesPerPlane * 8`. +- There are `element_size` **bit planes**, most-significant plane first: plane `p` (0-based) carries + element bit `element_size - 1 - p` of every element. +- Each plane is a `FixedString(bytesPerPlane)` per row, and the tuple is column-major, so plane `p` + occupies `nRows * bytesPerPlane` contiguous bytes; row `r`'s slice for a plane starts at + `r * bytesPerPlane`. +- Within a plane row, element `j`'s bit is stored MSB-first with a `j ^ 7` flip inside each byte: + its bit index is `bitIndex = (totalBits - 1) - (j ^ 7)`, i.e. byte `bitIndex >> 3`, bit + `bitIndex & 7`. + +### Why the client can't reuse the Array path + +Because the Native layout is bit-transposed and column-major, the QBit bytes are **not** the +`Array(E)` element bytes RowBinary sends. Feeding them through the per-row/columnar reader would +misread the column and desynchronize the rest of the block (every following column shifts). So the +Native reader must either fully reconstruct the vector or fail loudly. + +### Decoding (the inverse transpose) + +`BinaryStreamReader#readQBitNative` reverses the transpose, producing the **same** `float[]`/`double[]` +`readQBit` produces over RowBinary (so a `QBit` round-trips identically through either format): + +1. Read the `element_size` planes, each `nRows * bytesPerPlane` bytes. +2. Precompute, per element `j`, its byte offset `bitIndex >> 3` and bit mask `1 << (bitIndex & 7)` + within a plane row (constant across planes and rows). +3. For each row `r` and plane `p`, for each element `j`: if the masked bit is set, OR bit + `element_size - 1 - p` into element `j`'s accumulated bits. +4. Reinterpret each element's bits as the value: `Double.longBitsToDouble` for `Float64`, + `Float.intBitsToFloat` for `Float32`, and for `BFloat16` widen the 16 stored bits to a `float` by + shifting them into the high half (`bits << 16`), matching `readBFloat16LE`. + +This is the exact inverse of the server's `SerializationQBit::transposeBits`. + +## What client-v2 decodes over Native + +`client-v2` decodes a **plain, top-level** `QBit(Float32|Float64|BFloat16, dimension)` column from the +Native format (`NativeFormatReader#isNativeDecodableQBit` selects it). Every other shape is **rejected +up front** with a clear `ClientException` (rather than misread) — read it through a `RowBinary` format +instead: + +- **strided** `QBit(element_type, dimension, stride)` — its Native layout has + `element_size * (dimension / stride)` planes, which this decoder does not reconstruct; +- `QBit` wrapped in **`Nullable`** / **`LowCardinality`**; +- `QBit` **nested** inside another type (e.g. `Array`/`Tuple`/`Map(String, QBit(...))`); +- a `QBit` with any **non-float element type**.