Search before asking
Paimon version
master @ a5e87eb (2.1-SNAPSHOT)
Compute Engine
Engine-agnostic (format table read path)
Minimal reproduce step
- Create a CSV format table with
'csv.include-header' = 'true'.
- Write one uncompressed CSV file larger than
source.split.target-size (default 128MB).
SELECT COUNT(*).
What doesn't meet your expectations?
Expected: the count equals the number of data rows. Actual: it is short by (number of splits - 1).
SplitEnumerator.preferToSplitFile() splits such a file into (offset, length) segments, each read by a CsvFileReader. CsvFileReader#setupReading (line 62-69) calls readLine() on every split, ignoring offset. The header only exists at byte 0, so where offset > 0 the discarded line is that split's first data row.
Anything else?
The record straddling a boundary is already dropped by StandardLineReader#skipFirstLine and read in full by the previous split, so this extra readLine() is pure over-discard. A header plus 20 data rows, read as two segments, returns 19 rows.
Are you willing to submit a PR?
Search before asking
Paimon version
master @ a5e87eb (2.1-SNAPSHOT)
Compute Engine
Engine-agnostic (format table read path)
Minimal reproduce step
'csv.include-header' = 'true'.source.split.target-size(default 128MB).SELECT COUNT(*).What doesn't meet your expectations?
Expected: the count equals the number of data rows. Actual: it is short by (number of splits - 1).
SplitEnumerator.preferToSplitFile()splits such a file into(offset, length)segments, each read by aCsvFileReader.CsvFileReader#setupReading(line 62-69) callsreadLine()on every split, ignoringoffset. The header only exists at byte 0, so whereoffset > 0the discarded line is that split's first data row.Anything else?
The record straddling a boundary is already dropped by
StandardLineReader#skipFirstLineand read in full by the previous split, so this extrareadLine()is pure over-discard. A header plus 20 data rows, read as two segments, returns 19 rows.Are you willing to submit a PR?