[client]Fix memory leak if oom when decompress data.#2647
[client]Fix memory leak if oom when decompress data.#2647loserwang1024 merged 3 commits intoapache:mainfrom
Conversation
dd1a57b to
6daa427
Compare
|
@platinumhamburg @wuchong , CC |
| // vectorSchemaRootMap, | ||
| // for example temporary buffers left behind when an OOM occurs during decompression | ||
| // (see https://github.com/apache/fluss/issues/2646). | ||
| bufferAllocator.releaseBytes(bufferAllocator.getAllocatedMemory()); |
There was a problem hiding this comment.
Thanks for the fix! The approach is consistent with RecordAccumulator.close().
One concern: releaseBytes() only adjusts the allocator's accounting but doesn't actually free the underlying direct memory - those buffers will
remain until GC. This fix prevents the IllegalStateException on close, which is good for avoiding cascading failures, but the actual memory may still
leak temporarily.
For a more robust solution, consider adding try-finally in ZstdArrowCompressionCodec.doDecompress() to ensure temp buffers are released on failure.
But the current fix is acceptable as a defensive measure.
There was a problem hiding this comment.
@platinumhamburg , I modified this PR and add a patched version of Arrow's {@code VectorLoader} to fix this problem. Detail see VectorLoaderTest
0b07ab6 to
0e740b6
Compare
platinumhamburg
left a comment
There was a problem hiding this comment.
Great work identifying and fixing this Arrow buffer leak! I verified that the upstream Arrow-Java main branch
still has this bug in VectorLoader.loadBuffers() — the decompression loop sits outside the try block, and
buf.close() only runs on the success path. So this patch is definitely necessary.
The fix logic is correct: moving the decompression loop inside the try block and releasing buffers in the finally
clause properly handles both the success path (loadFieldBuffers retains +1, finally close -1) and the error path
(finally close frees all already-decompressed buffers).
Below are only one suggestion to improve maintainability.
...common/src/main/java/org/apache/fluss/shaded/arrow/org/apache/arrow/vector/VectorLoader.java
Outdated
Show resolved
Hide resolved
0e740b6 to
eaca9e5
Compare
eaca9e5 to
27dabc4
Compare

Purpose
Linked issue: close #2646
Brief change log
Tests
API and Format
Documentation