RFR: 8277322: Document that setting an invalid property `jdk.serialFilter` disables deserialization

[Roger Riggs]

On Tue, 23 Nov 2021 04:40:04 GMT, Stuart Marks <smarks at openjdk.org> wrote:

>> The effects of an invalid `jdk.serialFilter` property are not completely documented. If the value of the system property jdk.serialFilter is invalid, deserialization should not be possible and it should be clear in the specification. 
>> 
>> Specify an implementation specific exception is thrown in the case where deserialization is invoked after reporting the invalid jdk.serialFilter.
>
> src/java.base/share/classes/java/io/ObjectInputFilter.java line 529:
> 
>> 527:      * if the filter string is invalid, an {@link ExceptionInInitializerError} is thrown
>> 528:      * and the initialization fails; subsequent attempts to use the configuration or
>> 529:      * serialization will fail with an implementation specific exception.
> 
> I'm confused about exactly what happens after `ExceptionInInitializerError`.
> 
>> Subsequent attempts to use the configuration or serialization will fail....
> 
> Which configuration? I thought OIF.Config is a utility class and thus has no instances. If its class initialization fails, then other code cannot use `Config.setSerialFilter` to set a global filter (which might be desirable, but throws NCDFE instead of `IllegalStateException`) and other code can't use `Config.createFilter` to create individual filters. Is that right? It seems like there ought to be a better arrangement than to have the system come up in some dysfunctional way, where any subsequent reference to `OIF.Config` results in NCDFE.
> 
> And surely this affects deserialization, not serialization?

Most configurations of `jdk.serialFilter` and` jdk.serialFilterFactory` will be valid.
If they are not valid, the cause must be clear and useful suggestion made to correct the command line
or security properties.

It must not be possible to do any deserialization if the command line (or security) properties are invalid.
Since the states are held in static fields of `ObjectInputFilter.Config`, the initialization is done in a static initializer.
Exceptions/errors in the static initializer cause `ExceptionInInitializerError` to be thrown.
In most cases, that will be sufficient to report the invalid properties and the program will be terminated.
The message on the error should be sufficient correct the filter

However, there is a chance that `ExceptionInInitializerError` will be caught and ignored.
Ignoring the exception should not allow deserialization.
At the moment, this comes for free because when the initialization of `ObjectInputFilter.Config`
fails any subsequent reference to the class causes `NoClassDefFoundError`.
Neither calls to `ObjectInputFilter.Config` get or set methods will succeed.
Deserialization is prevented because `ObjectInputStream` constructors call `Config.getSerialFilterFactorySingleton` which will fail with `NoClassDefFoundError`.

The question that has been raised is how much of that should be included in the specification
of `ObjectInputFilter.Config`.  The CSR proposes that after `ExceptionInInitializerError` is
thrown, it is specified that an implementation specific exception/error is thrown when attempting to use `Config`.
There is no additional value in being specific about the error that is thrown.

-------------

PR: https://git.openjdk.java.net/jdk/pull/6508