When verify signature receive java.io.EOFException: Unexpected end of ZIP input stream

[brett-walker]

In verify the signer of a PGP encrypted message I receive the following stacktrace:

Exception in thread "main" java.io.EOFException: Unexpected end of ZIP input stream
       at org.bouncycastle.openpgp.PGPCompressedData$2.fill(PGPCompressedData.java:155)
       at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
       at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
       at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
       at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
       at org.bouncycastle.bcpg.BCPGInputStream.read(BCPGInputStream.java:99)
       at org.bouncycastle.bcpg.BCPGInputStream$PartialInputStream.read(BCPGInputStream.java:432)
       at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
       at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
       at org.bouncycastle.bcpg.BCPGInputStream.read(BCPGInputStream.java:82)
       at org.bouncycastle.bcpg.SignaturePacket.<init>(SignaturePacket.java:43)
       at org.bouncycastle.bcpg.BCPGInputStream.readPacket(BCPGInputStream.java:271)
       at org.bouncycastle.openpgp.PGPSignature.<init>(PGPSignature.java:74)
       at org.bouncycastle.openpgp.PGPObjectFactory.nextObject(PGPObjectFactory.java:101)
       at com.geometryit.dhhs.hms.utils.PgpDecryptor.decrypt(PgpDecryptor.java:116)
       at com.geometryit.dhhs.hms.utils.PgpDecryptor.main(PgpDecryptor.java:308)

The message was decrypted successfully with the expected message. And it was verified correctly too.
The message had the expected PGPOnePassSignatureList block before the PGPLiteralData block. It contained the ID of the expected signer.

Using version 1.77 of the Bouncy Castle libraries.
Using a Java version of Temurin-11.0.22+7
The OS platform is MacOS using Sonoma 14.2.1

[chibenwa]

Writing a PGP extension on top of Apache James server, using bouncyCastle 1.77 we encounter the very same issue.

Downgrade to 1.70 solves the issue.

An alternative could be .setWithIntegrityPacket(false) but it looks unsafe.

[brett-walker]

Have done some investigation using older versions.

I did not have to roll back so far, only to 1.71.1 for my code to work.

Version	Integrity verification	Signature verification
1.71	passed	passed
1.71.1	passed	passed
1.72	failed	passed
1.73	passed	failed
1.74	passed	failed
1.75	passed	failed
1.76	passed	failed
1.77	passed	failed

The release notes for version 1.72 indicate a change in the BZIP2 implementation.

The release notes for version 1.73 indicate a change in the BCPGInputStream class.

I wonder if the latter is the source of the problem I am encountering.

[brett-walker]

@dghgit

Sorry to call you in. Need a bit of advice.

The change of interest in the BCPGInputStream class for v1.73; in the readPacket() method, the general change was in.read() calls change to this.read() calls. (d594ee2 08/10/2022).
I'm wondering if the same sort of change needs to happen in the readPacketTag() method too to help resolve the problem I'm encountering.
Part of the reason I'm asking is I would have to set-up a gradle environment to investigate. Using gradle which would be new too me. I use maven regularly.

Hope you can help, Thanks.

[dghgit]

Hmm. Possibly, change should appear on github shortly, but try what's in https://www.bouncycastle.org/betas and let me know how it goes.

[trondaks]

Similar issue, works in 1.71.1 and earlier, fails afterwards... Tried the 1.78-SNAPSHOT without luck.

            InputStream dIn = pgpLiteralData.getInputStream();
            byte[] byteArray = dIn.readAllBytes();  //Fails on this

Caused by: java.io.EOFException: premature end of stream in PartialInputStream
at org.bouncycastle.bcpg.BCPGInputStream$PartialInputStream.read(BCPGInputStream.java:435)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
at org.bouncycastle.bcpg.BCPGInputStream.read(BCPGInputStream.java:99)
at org.bouncycastle.openpgp.PGPCompressedData$2.fill(PGPCompressedData.java:158)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at org.bouncycastle.bcpg.BCPGInputStream.read(BCPGInputStream.java:99)
at org.bouncycastle.bcpg.BCPGInputStream$PartialInputStream.read(BCPGInputStream.java:432)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
at org.bouncycastle.bcpg.BCPGInputStream.read(BCPGInputStream.java:99)
at java.base/java.io.InputStream.readNBytes(InputStream.java:409)
at java.base/java.io.InputStream.readAllBytes(InputStream.java:346)
at decryptAndOrValidateSigStream(PgpUtils.java:530)
... 38 more

[dghgit]

Looking at the stack traces these aren't using the BC zip library. Is it possible to provide something that reproduces the issue? I think the changes are coincidental, it's likely to be somewhere else.

[brett-walker]

I could probably create a test case with the code that generated my stack trace. It will take a week or two, as I'll need clearance from the client to include some of their test data that helped generate the issue.

I tried to grok the code base, but the commit history, and the vastness of the code base made this difficult to comprehend what is going on.

[brett-walker]

I roughly do the same but still experience the problem. A couple of things to note from the example provided by @trondaks

• The code is from 11 years ago, May, 2013. The APIs have changes.
• I use instance of JcaPGPObjectFactory rather than PGPObjectFactory. Same error occurs if I use BcPGPObjectfactory.
• It verifies the signature before the verifying the data integrity. I do it the other way around.

Another thing of note, not yet mentioned, is that my test data was generated by Gnu PG (windows).

[brett-walker]

In trying to get to a simple piece of code as part of a reproduction case I have solved my issue.

I swapped the order things. Initially I had data integrity verification followed by signature verification. In swapping the order to signature verification followed by data integrity verification solved my issue.

I thought I had tried this variation before. 😒

The stack trace I received was not helpful.

Thanks for people input.
LLAP 🖖

[brett-walker]

@dghgit
Before I close this case, I will add some final comments.

I would have thought that the questions: 'Has the data been tampered with? (data integrity)' and 'Has the message been sent by a know user? (signature verification)' would have been independant from each other. Conceptually I see it this way. Actual implementation may be different.

I coded my solution from examples I found on the web. I adapted the code as I saw fit. It mostly worked. The example were very poorly documented, and I did not read any of the online supporting documentation from BC.

The way I see it, from resolving my problem; it is the internal design of BC that has dictated the external usage of its API. That is signature verification needs to be done before data verification. If this is documented then it is so easily missed.

The stack trace I encountered did not indicate the real cause of my problem. I was actually looking for the solution in an entirely different area.

While the internal design of BC dictating external usage of its API may be considered poor practise; it may be because of other considerations such as algorthim, speed, memory, or other security considerations.

What would have been helpful here is some sort if internal check, if possible, in the signature verification step that it's source data has not been altered, and if it is altered throw an informative excpetion early.

I have suggested this tentative improvement because if internal design of BC dictates the external usage of its API then an informative exception when it is being use contray to expections would be helpful.

[dghgit]

I think we could better document this - the reason for it happening is because the BC PGP API is a streaming API, so order of events is tied in with the order of things as they appear in the PGP protocol. This allows the API to work with very large files, but on the other hand does introduce a level of eccentricity as order of operations really does matter.