Search before asking
Paimon version
master, 142f823
Compute Engine
Engine independent.
Minimal reproduce step
Found by code inspection rather than from a failing job. Read a row format file whose tail is truncated, or whose footer is corrupt, so that footer or block index parsing throws.
What doesn't meet your expectations?
RowFormatReaderFactory.createReader opens the stream first and only transfers ownership to RowFormatReader on the last line:
SeekableInputStream in = fileIO.newInputStream(path);
int tailSize = (int) Math.min(TAIL_PREFETCH_SIZE, fileSize);
long tailOffset = fileSize - tailSize;
in.seek(tailOffset);
byte[] tailBuf = new byte[tailSize];
IOUtils.readFully(in, tailBuf);
RowFileFooter footer = RowFileFooter.readFrom(tailBuf, tailSize - RowFileFooter.FOOTER_SIZE);
RowBlockIndex blockIndex;
...
blockIndex = RowBlockIndex.readFrom(in, footer.indexOffset, footer.indexLength);
return new RowFormatReader(in, path, footer, blockIndex, rowType, projection, context.selection());
There is no try / catch anywhere in the method, so a throw from in.seek, IOUtils.readFully, RowFileFooter.readFrom or RowBlockIndex.readFrom leaves in open with no owner. Scanning a set of files where several are corrupt leaks one stream per file.
Anything else?
Same interaction as the sibling report on CachingSeekableInputStream: with the lifetime tracking added in #8962, a stream that is never closed holds its lease forever, so that entry's FileIO is never released. That matches today's behaviour rather than regressing it, but it does keep the fix from reaching the affected entries.
Fix shape: close in before rethrowing.
Are you willing to submit a PR?
Search before asking
Paimon version
master, 142f823
Compute Engine
Engine independent.
Minimal reproduce step
Found by code inspection rather than from a failing job. Read a row format file whose tail is truncated, or whose footer is corrupt, so that footer or block index parsing throws.
What doesn't meet your expectations?
RowFormatReaderFactory.createReaderopens the stream first and only transfers ownership toRowFormatReaderon the last line:There is no
try/catchanywhere in the method, so a throw fromin.seek,IOUtils.readFully,RowFileFooter.readFromorRowBlockIndex.readFromleavesinopen with no owner. Scanning a set of files where several are corrupt leaks one stream per file.Anything else?
Same interaction as the sibling report on
CachingSeekableInputStream: with the lifetime tracking added in #8962, a stream that is never closed holds its lease forever, so that entry'sFileIOis never released. That matches today's behaviour rather than regressing it, but it does keep the fix from reaching the affected entries.Fix shape: close
inbefore rethrowing.Are you willing to submit a PR?