[External] : Re: PEM API github repo

Tue May 28 16:48:33 UTC 2024

Hi,

PEM didn't make JDK 23 as there were further comments about the API.  It 
hasn't been updated because of that.  Hopefully in June I will be 
updating the github repo.

Tony

On 5/18/24 3:55 AM, Karl Scheibelhofer wrote:
> Hi Tony,
> 
> today, I had a look at the recent sources of the PEM API 
> implementation - i.e. https://github.com/ascarpino/jdk/commits/pem 
> <https://urldefense.com/v3/__https://github.com/ascarpino/jdk/commits/pem__;!!ACWV5N9M2RV99hQ!KryKT5MjRlHpyEwcDJR5vP5ST53pFGfYJ80ocGPInq5HkglLO0076V7QKD8hckW5TqF_ZNc1b1mNq6Si7vCKEwRx8XUKiQCF$>
> 
> It seems that the PEM API Tests are out of sync. They fail with
> 
> JT Harness : Tests that failed
> java/security/PEM/PEMDecoderTest.java: Testing PEM decodings
> java/security/PEM/PEMEncoderTest.java: Testing PEM decoding
> 
> I wanted to update the PEM keystore implementation to the latest code 
> base. Can you tell about the planned next steps?
> 
> Regards, Karl
> 
> On Sat, Mar 16, 2024 at 10:36 AM Karl Scheibelhofer 
> <karl.scheibelhofer.75 at gmail.com 
> <mailto:karl.scheibelhofer.75 at gmail.com>> wrote:
> 
>     Hi Tony,
> 
>     find my replies inline...
> 
>     On Mon, Mar 11, 2024 at 6:13 AM Anthony Scarpino
>     <anthony.scarpino at oracle.com <mailto:anthony.scarpino at oracle.com>>
>     wrote:
>      >
>      >
>      >
>      > On Mar 9, 2024, at 8:09 AM, Karl Scheibelhofer
>     <karl.scheibelhofer.75 at gmail.com
>     <mailto:karl.scheibelhofer.75 at gmail.com>> wrote:
>      >
>      > 
>      > ... try again from from my subscribed mail account...
>      >
>      >> Hi Tony,
>      >>
>      >> in my jdk fork, I created a branch named pem-feedback-karl.
>      >>
>      >> https://github.com/KarlScheibelhofer/jdk/tree/pem-feedback-karl
>     <https://urldefense.com/v3/__https://github.com/KarlScheibelhofer/jdk/tree/pem-feedback-karl__;!!ACWV5N9M2RV99hQ!KryKT5MjRlHpyEwcDJR5vP5ST53pFGfYJ80ocGPInq5HkglLO0076V7QKD8hckW5TqF_ZNc1b1mNq6Si7vCKEwRx8QAJkEvq$>
>      >>
>      >> It is based on the pem branch of your jdk fork.
>      >> In this pem-feedback-karl branch, I did some cleanup without
>     changing
>      >> the API. Your tests pass as before.
>      >>
>      >> My original pem-keystore implementation for the SUN provider is
>     in this branch
>      >>
>      >> https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore
>     <https://urldefense.com/v3/__https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore__;!!ACWV5N9M2RV99hQ!KryKT5MjRlHpyEwcDJR5vP5ST53pFGfYJ80ocGPInq5HkglLO0076V7QKD8hckW5TqF_ZNc1b1mNq6Si7vCKEwRx8evoXyho$>
>      >>
>      >> It did not use the PEM API.
>      >>
>      >> In the branch
>      >>
>      >>
>     https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore-pem-api
>     <https://urldefense.com/v3/__https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore-pem-api__;!!ACWV5N9M2RV99hQ!KryKT5MjRlHpyEwcDJR5vP5ST53pFGfYJ80ocGPInq5HkglLO0076V7QKD8hckW5TqF_ZNc1b1mNq6Si7vCKEwRx8ZydhYWH$>
>      >>
>      >> I modified the PEM keystore implementation to use the PEMDecoder
>     and PEMEncoder.
>      >> To make it pass all tests, I had to fix some issues with the PEM
>     api:
>      >>
>      >> * fix missing line-breaks when encoding certificates (and CRLs)
>      >> * use uniform line length for PEM encoding keys and certificates
>      >
>      >
>      > It sounds like I did my repo update to use MimeEncoder after you
>     had pulled the changes.
> 
>     In which repo can i see the version using MimeEncoder that you
>     mentioned? i can't find this in
>     https://github.com/ascarpino/jdk/commits/pem
>     <https://urldefense.com/v3/__https://github.com/ascarpino/jdk/commits/pem__;!!ACWV5N9M2RV99hQ!KryKT5MjRlHpyEwcDJR5vP5ST53pFGfYJ80ocGPInq5HkglLO0076V7QKD8hckW5TqF_ZNc1b1mNq6Si7vCKEwRx8XUKiQCF$>
> 
>      >
>      >> During this work, I took some notes regarding the PEM api:
>      >>
>      >> * Consider decoupling the raw PEM encoding and decoding from
>     SecurityObject.
>      >>   This would make the API usable for general purpose PEM
>     encoding and
>      >> decoding, not just for keys and certificates, as it is now.
>      >
>      >
>      > There has been discussions about adding a generic PEM object that
>     would have methods to return the header, footer, and PEM text,
>     instead of processing into a class or KeySpec. Is there something
>     more “general purpose” you had it mind?
> 
>     In addition to header, footer and PEM text, for my keystore
>     implementation, I would need the preceding PEM explanatory text lines,
>     i.e. the lines before the header line
> 
>      >
>      >> * For this PEM keystore implementation it is essential to parse the
>      >> preceding explanatory text lines.
>      >>   The PEM decoder should support this.
>      >>   As it is now, the keystore implementation must parse the PEM
>     objects
>      >> itself, including reading PEM header and footer lines.
>      >>   Having to handle half the work in the application diminishes the
>      >> value of the PEMDecoder.
>      >>   It only delegates the decoding of certificates and keys to the
>     PEMDecoder.
>      >>
>      >> * PEMDecoder should be able to handle or pass through unknown PEM
>      >> objects gracefully.
>      >>   Otherwise the application has to check the PEM labels in advance
>      >> itself, which does not make sense.
>      >
>      >
>      > So do you not have a structured data file?  I expected the
>     application would parse its own metadata, then when the structured
>     code got to a PEM tag, it would pass the InputStream to PEMDecoder.
>      >
>      > I am concerned about a generic PEM object storing an unknown
>     amount of application metadata and returning it back.  I feel
>     handling non-PEM should be in the scope of the encoder/decoder.
> 
> 
>     No, there is no structured data file. Just a PEM file which contains
>     multiple PEM objects, typically certificates and private keys. It is
>     common practice to have related certificates and keys in a single
>     file. The explanatory text lines contain the name of the key, because
>     this is the place specified in PEM RFC to hold meta information like
>     this. To enable implementing a consistent Java Keystore with PEM
>     format, I see this necessary. At least, I found no other way for
>     handling key alias names as required  by Java Keystore. If you know
>     another way, please let me know.
> 
>     The files that my keystore implementation reads and writes are just
>     completely PEM format, from first to last line.
> 
>      >
>      >>
>      >> * Though supporting InputStream/OutputStream for reading and writing
>      >> makes sense,
>      >>   supporting Reader/Writer feels even more essential for robust
>      >> support for all character encodings.
>      >>   Multi-Byte character encodings won't work with
>     InputStream/OutputStream.
>      >
>      >
>      > A Reader will read ahead, buffering the input data.  I saw this
>     when I had `decode(Reader)` in the API.  It would return the first
>     PEM object, but the read pointer was at the end of the file, missing
>     the remaining PEM objects.
> 
>     An application or a Keystore implementation gets a stream providing
>     multiple concatenated PEM objects that are in this keystore object. It
>     needs to get all the PEM objects handing in that stream, i.e. in
>     essence hand in the input stream and get out a list of pem objects
>     containing certificates, keys and their names.
> 
>      >
>      > Is the multi-byte characters for the keystore metadata?  I don’t
>     see how this affects Base64.
> 
>     Try feeding in a UTF-16 encoded PEM file. I guess it won't work with
>     InputStream. Because the current implementation expects one character
>     in each byte.
> 
>      >
>      >>
>      >> * The standard line separator for PEM is "\r\n".
>      >>   For PEM files stored in a typical linux file system, "\n" is
>      >> typically used, however.
>      >>   See the output of openssl, for example.
>      >>   Because there are still several command line utilities that do not
>      >> work well with "\r\n" line breaks.
>      >>   Supporting a different line-separator than "\r\n" is a good
>     idea in
>      >> my opinion.
>      >>   Base64.getMimeEncoder also supports selecting a custom line
>     separator.
>      >
>      >
>      > The standards I saw says the line separator is “\r\n”.  I realize
>     decoders have to be more flexible because many may not follow the
>     line separators or cut-n-paste removes them.  I think having a
>     configurable line separator for encoding is likely to create more
>     incompatibility rather than lessen for cross-platform and using with
>     other applications.
> 
>     Yes, the PEM standard say"\r\n". But widely used tools like openssl
>     produce "\n". In practice, this is just the better solution. It goes
>     better with most other tools and version control.
>     I would opt for "\r\n" by default, but provide an option for "\n" only.
>     There must be a reason why Base64.getMimeEncoder supports custom line
>     separators.
> 
>      >
>      >>
>      >> * The PEMEncoder encodes the predefined SecurityObjects only.
>      >>   There is no way to use it to PEM encoded any other type of object.
>      >>   Consider opening a path to generic use.
>      >
>      >
>      > The generic PEM object I mentioned previously I think fits this
>     case.  It would still be a SecurityObject as I don’t see value in
>     passing in any object.
> 
>     I will have a look at it. Can you provide a link to your implementation?
> 
>      >
>      >>
>      >> * If an application has a DER encoded certificate, it has to decode
>      >> and parse the certificate before
>      >>   it can encode it using PEMEncoder.
>      >>   This is inconvenient.
>      >
>      >
>      > Yeah, there isn’t an EncodedKeySpec equivalent.  I’d have to
>     think if there is an easy way to do this without causing more
>     problems.  Given the purpose is going between Java Objects and PEM,
>     accepting random data isn’t a goal.  Maybe something that can be
>     addressed with a generic PEM object.
> 
>     As a fallback simply allow passing in a byte[] that is base64 encoded,
>     i.e. the result of Certificate.getEncoded().
> 
>     Thank you!
> 
>     Karl
> 
>     ...
> 