[External] : Re: PEM API github repo

Sat May 18 10:55:37 UTC 2024

Hi Tony,

today, I had a look at the recent sources of the PEM API implementation -
i.e.  https://github.com/ascarpino/jdk/commits/pem

It seems that the PEM API Tests are out of sync. They fail with

JT Harness : Tests that failed
java/security/PEM/PEMDecoderTest.java: Testing PEM decodings
java/security/PEM/PEMEncoderTest.java: Testing PEM decoding

I wanted to update the PEM keystore implementation to the latest code base.
Can you tell about the planned next steps?

Regards, Karl

On Sat, Mar 16, 2024 at 10:36 AM Karl Scheibelhofer <
karl.scheibelhofer.75 at gmail.com> wrote:

> Hi Tony,
>
> find my replies inline...
>
> On Mon, Mar 11, 2024 at 6:13 AM Anthony Scarpino
> <anthony.scarpino at oracle.com> wrote:
> >
> >
> >
> > On Mar 9, 2024, at 8:09 AM, Karl Scheibelhofer <
> karl.scheibelhofer.75 at gmail.com> wrote:
> >
> > 
> > ... try again from from my subscribed mail account...
> >
> >> Hi Tony,
> >>
> >> in my jdk fork, I created a branch named pem-feedback-karl.
> >>
> >> https://github.com/KarlScheibelhofer/jdk/tree/pem-feedback-karl
> >>
> >> It is based on the pem branch of your jdk fork.
> >> In this pem-feedback-karl branch, I did some cleanup without changing
> >> the API. Your tests pass as before.
> >>
> >> My original pem-keystore implementation for the SUN provider is in this
> branch
> >>
> >> https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore
> >>
> >> It did not use the PEM API.
> >>
> >> In the branch
> >>
> >> https://github.com/KarlScheibelhofer/jdk/tree/pem-keystore-pem-api
> >>
> >> I modified the PEM keystore implementation to use the PEMDecoder and
> PEMEncoder.
> >> To make it pass all tests, I had to fix some issues with the PEM api:
> >>
> >> * fix missing line-breaks when encoding certificates (and CRLs)
> >> * use uniform line length for PEM encoding keys and certificates
> >
> >
> > It sounds like I did my repo update to use MimeEncoder after you had
> pulled the changes.
>
> In which repo can i see the version using MimeEncoder that you
> mentioned? i can't find this in
> https://github.com/ascarpino/jdk/commits/pem
>
> >
> >> During this work, I took some notes regarding the PEM api:
> >>
> >> * Consider decoupling the raw PEM encoding and decoding from
> SecurityObject.
> >>   This would make the API usable for general purpose PEM encoding and
> >> decoding, not just for keys and certificates, as it is now.
> >
> >
> > There has been discussions about adding a generic PEM object that would
> have methods to return the header, footer, and PEM text, instead of
> processing into a class or KeySpec. Is there something more “general
> purpose” you had it mind?
>
> In addition to header, footer and PEM text, for my keystore
> implementation, I would need the preceding PEM explanatory text lines,
> i.e. the lines before the header line
>
> >
> >> * For this PEM keystore implementation it is essential to parse the
> >> preceding explanatory text lines.
> >>   The PEM decoder should support this.
> >>   As it is now, the keystore implementation must parse the PEM objects
> >> itself, including reading PEM header and footer lines.
> >>   Having to handle half the work in the application diminishes the
> >> value of the PEMDecoder.
> >>   It only delegates the decoding of certificates and keys to the
> PEMDecoder.
> >>
> >> * PEMDecoder should be able to handle or pass through unknown PEM
> >> objects gracefully.
> >>   Otherwise the application has to check the PEM labels in advance
> >> itself, which does not make sense.
> >
> >
> > So do you not have a structured data file?  I expected the application
> would parse its own metadata, then when the structured code got to a PEM
> tag, it would pass the InputStream to PEMDecoder.
> >
> > I am concerned about a generic PEM object storing an unknown amount of
> application metadata and returning it back.  I feel handling non-PEM should
> be in the scope of the encoder/decoder.
>
>
> No, there is no structured data file. Just a PEM file which contains
> multiple PEM objects, typically certificates and private keys. It is
> common practice to have related certificates and keys in a single
> file. The explanatory text lines contain the name of the key, because
> this is the place specified in PEM RFC to hold meta information like
> this. To enable implementing a consistent Java Keystore with PEM
> format, I see this necessary. At least, I found no other way for
> handling key alias names as required  by Java Keystore. If you know
> another way, please let me know.
>
> The files that my keystore implementation reads and writes are just
> completely PEM format, from first to last line.
>
> >
> >>
> >> * Though supporting InputStream/OutputStream for reading and writing
> >> makes sense,
> >>   supporting Reader/Writer feels even more essential for robust
> >> support for all character encodings.
> >>   Multi-Byte character encodings won't work with
> InputStream/OutputStream.
> >
> >
> > A Reader will read ahead, buffering the input data.  I saw this when I
> had `decode(Reader)` in the API.  It would return the first PEM object, but
> the read pointer was at the end of the file, missing the remaining PEM
> objects.
>
> An application or a Keystore implementation gets a stream providing
> multiple concatenated PEM objects that are in this keystore object. It
> needs to get all the PEM objects handing in that stream, i.e. in
> essence hand in the input stream and get out a list of pem objects
> containing certificates, keys and their names.
>
> >
> > Is the multi-byte characters for the keystore metadata?  I don’t see how
> this affects Base64.
>
> Try feeding in a UTF-16 encoded PEM file. I guess it won't work with
> InputStream. Because the current implementation expects one character
> in each byte.
>
> >
> >>
> >> * The standard line separator for PEM is "\r\n".
> >>   For PEM files stored in a typical linux file system, "\n" is
> >> typically used, however.
> >>   See the output of openssl, for example.
> >>   Because there are still several command line utilities that do not
> >> work well with "\r\n" line breaks.
> >>   Supporting a different line-separator than "\r\n" is a good idea in
> >> my opinion.
> >>   Base64.getMimeEncoder also supports selecting a custom line separator.
> >
> >
> > The standards I saw says the line separator is “\r\n”.  I realize
> decoders have to be more flexible because many may not follow the line
> separators or cut-n-paste removes them.  I think having a configurable line
> separator for encoding is likely to create more incompatibility rather than
> lessen for cross-platform and using with other applications.
>
> Yes, the PEM standard say"\r\n". But widely used tools like openssl
> produce "\n". In practice, this is just the better solution. It goes
> better with most other tools and version control.
> I would opt for "\r\n" by default, but provide an option for "\n" only.
> There must be a reason why Base64.getMimeEncoder supports custom line
> separators.
>
> >
> >>
> >> * The PEMEncoder encodes the predefined SecurityObjects only.
> >>   There is no way to use it to PEM encoded any other type of object.
> >>   Consider opening a path to generic use.
> >
> >
> > The generic PEM object I mentioned previously I think fits this case.
> It would still be a SecurityObject as I don’t see value in passing in any
> object.
>
> I will have a look at it. Can you provide a link to your implementation?
>
> >
> >>
> >> * If an application has a DER encoded certificate, it has to decode
> >> and parse the certificate before
> >>   it can encode it using PEMEncoder.
> >>   This is inconvenient.
> >
> >
> > Yeah, there isn’t an EncodedKeySpec equivalent.  I’d have to think if
> there is an easy way to do this without causing more problems.  Given the
> purpose is going between Java Objects and PEM, accepting random data isn’t
> a goal.  Maybe something that can be addressed with a generic PEM object.
>
> As a fallback simply allow passing in a byte[] that is base64 encoded,
> i.e. the result of Certificate.getEncoded().
>
> Thank you!
>
> Karl
>
> ...
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20240518/1f4d0c3e/attachment.htm>