OutOfMemoryError when copyFileToContainer used on large file

[findepi]

I want to copy a large installation archive into a container using GenericContainer#withCopyFileToContainer
Since the process runs with certain ("reasonable") -Xmx, it hits a limit:

Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
	at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
	at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:48)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:244)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:92)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:86)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:122)
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:454)
	at java.base/java.io.InputStream.transferTo(InputStream.java:705)
	at java.base/java.nio.file.Files.copy(Files.java:3119)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:326)
	at org.testcontainers.utility.MountableFile.transferTo(MountableFile.java:300)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:257)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:234)
	at org.testcontainers.containers.GenericContainer$$Lambda$434/0x0000000800408840.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:397)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:317)
	at org.testcontainers.containers.GenericContainer$$Lambda$414/0x0000000800402440.call(Unknown Source)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:315)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
	at org.testcontainers.lifecycle.Startables$$Lambda$411/0x0000000800401440.run(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:783)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[findepi]

.

[jandry]

I ran into this issue last friday after upgrading from testcontainers 1.18.0 to 1.18.1. (same gitlab runner)
I reproduced it on my project and solved it just reverting the testcontainers version

[eugene-kuntsevich]

The same issue for me after migrating from 1.18.0 to 1.18.3. So 1.18.0 works good. But with 1.18.3 I got:

Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
	at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:62)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:91)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:259)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:169)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:206)
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:713)
	at java.base/java.io.InputStream.transferTo(InputStream.java:783)
	at java.base/java.nio.file.Files.copy(Files.java:3213)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:362)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.transferTo(MountableFile.java:327)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:306)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:282)
	at org.testcontainers.containers.GenericContainer$$Lambda$422/0x0000000800eaa290.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:721)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:430)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at org.testcontainers.containers.GenericContainer$$Lambda$391/0x0000000800e8e828.call(Unknown Source)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at org.testcontainers.containers.ContainerisedDockerCompose.invoke(DockerComposeContainer.java:727)
	at org.testcontainers.containers.DockerComposeContainer.runWithCompose(DockerComposeContainer.java:339)
	at org.testcontainers.containers.DockerComposeContainer.createServices(DockerComposeContainer.java:260)
	at org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:191)
	at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:274)

[jandry]

I confirm, try 1.18.1, 1.18.2, 1.18.3, same issue. Stay in 1.18.0 is only solution I found for the moment

[eddumelendez]

Thanks for reporting the issue. O do not have any change related to this on top of my hrad but will take a deep look on what changed between 1.18.0 and 1.18.1

[findepi]

@eddumelendez would you be interested in taking a look at #2864 ?

[eddumelendez]

I just notice this is an old issue but some users hit into it in 1.18.1. Also, I see an update in the dependency that can be related to this issue. Since I do not have a reproducer, Can you please use org.apache.commons:commons-compress:1.22.0 instead of org.apache.commons:commons-compress:1.23.0?

[eugene-kuntsevich]

I just notice this is an old issue but some users hit into it in 1.18.1. Also, I see an update in the dependency that can be related to this issue. Since I do not have a reproducer, Can you please use org.apache.commons:commons-compress:1.22.0 instead of org.apache.commons:commons-compress:1.23.0?

Thanks for proposal, I tried but it didn't help

[jandry]

Still have the issue on 1.19.0. Only working with 1.18.0. And like mentioned, starting 1.18.1 dependency is org.apache.commons:commons-compress:1.23.0 instead of 1.22.0

[jandry]

Seems same as #7239