Authorization Header Limitations and Executor Behavior Clarification

[Daniel Fuchs]

Hi,

Thank you for trying the new HTTP client!

I believe both issues described here might have been fixed.
At least JDK 11 should let you specify your own authorization
header, and I have verified that a simple GET request works
with an inline executor.

However there could be some other issues explaining the
behavior you have observed - please continue reading:

1. We do have some tests to verify that it is possible to
    supply custom authorization headers to implement e.g. a
    DIGEST authentication on top of the API. These tests
    are included on our test base and run on regular basis.

    An important point to note is that the
    java.io.IOException: Invalid auth header
    is thrown when the server respond with an Unauthorized code
    (401/407) but does not include the expected
    WWW-Authenticate or Proxy-Authenticate headers.
    It would be helpful to know which headers the server is
    sending back.

    For troubleshooting it's possible to enable logging of
    headers on the java command line with
       -Djdk.httpclient.HttpClient.log=headers

    I realize the 'Invalid auth header' is not a very helpful
    message and I will log a JBS issue to examine if we can
    do better (maybe we should not throw an exception but
    just log it).

2. I have run a simple test to verify that you could supply an
    inline executor. It worked well for GET requests.

    Note however that in that case, you must be very careful of
    what code you trigger in dependent actions you might register
    with the returned CompletableFuture, as those will also execute
    in your inline executor and might therefore wedge the selector
    thread.

If any of these issue still shows up - then please could you try
the same with the java.net.http module in JDK 11?
Then reply to this mail with some more information - in particular
WRT to the header values you're trying to set, and what the server
sends back, and a thread dump to show where the inline executor
is wedged if the problem is still there, together with the
line of code that calls HttpClient::send/sendAsync as it
would be useful to know which BodyHandler was used.

best regards,

-- daniel


On 25/05/2018 19:03, danthonywalker wrote:
> On Java 10.0.1 (build 10.0.1+10), the incubating HttpClient has 2 
> behaviors that I believe are problematic and should either be fixed or 
> documented to better highlight their limitations.
> 
> The first concerns customized Authorization headers. From what I 
> understand, Authorization headers outside of "Basic" are disallowed by 
> the API. For example, when the type of Authorization is set to "Bot" or 
> "Bearer", the following exception is thrown
> 
>     Caused by: java.util.concurrent.CompletionException:
>     java.io.IOException: Invalid auth header
>     at
>     java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
>     at
>     java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
>     at
>     java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1074)
>     at
>     java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>     at
>     java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
>     at
>     java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649)
>     at
>     java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
>     at
>     java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
>     at
>     java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>     at java.base/java.lang.Thread.run(Thread.java:844)
>     Caused by: java.io.IOException: Invalid auth header
>     at
>     jdk.incubator.httpclient/jdk.incubator.http.AuthenticationFilter.lambda$response$1(AuthenticationFilter.java:211)
>     at java.base/java.util.Optional.orElseThrow(Optional.java:397)
>     at
>     jdk.incubator.httpclient/jdk.incubator.http.AuthenticationFilter.response(AuthenticationFilter.java:210)
>     at
>     jdk.incubator.httpclient/jdk.incubator.http.MultiExchange.responseFilters(MultiExchange.java:195)
>     at
>     jdk.incubator.httpclient/jdk.incubator.http.MultiExchange.lambda$responseAsyncImpl$7(MultiExchange.java:273)
>     at
>     java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
>     ... 7 more
> 
> 
> Adding the following hack fixes the issue regarding using a "Bot" type 
> (code is in Kotlin). The solution was taken from here: 
> https://clevertap.com/blog/java-9-httpclient-does-not-allow-custom-authorization-headers/
> 
>     httpRequest.javaClass.getDeclaredMethod("setSystemHeader",
>     String::class.java, String::class.java).apply {
>          isAccessible = true // Allows access to an internal method to
>     add Authorization header for httpRequest
>          invoke(httpRequest, "authorization",
>     requestHeaders["authorization"])
>     }
> 
> 
> However, "Bearer" still fails with the same exception.
> Either this restriction should be lifted (I see no reason to restrict 
> it), or add this behavior to the Javadocs.
> 
> The second issue regards setting an Executor when building a HttpClient. 
> There is problematic behavior regarding it when using an Executor under 
> 1 of 2 scenarios.
> 
> When an Executor uses the current running thread the request never 
> completes and blocks the current thread. I have not done any in-depth 
> debugging to pinpoint the exact cause. The behavior I expect from this 
> is the thread is blocked as it waits for the request to come through, 
> but it should come through eventually; as if it was a sync API.
> 
> When an Executor only uses a single thread (more specifically, if I use 
> Executors.newSingleThreadExecutor), the same behavior occurs; except the 
> behavior in this case should clearly be that the work is offloaded to 
> another thread, allowing the application to continue, but it doesn't, it 
> still gets stuck at the request call.
> 
> I have received an exception before when running on the current thread 
> that relates to some method being invoked on the same thread as another 
> method. However, when retrying these scenarios for this posting, I was 
> unable to replicate this behavior, and thus both Executor scenarios 
> produce the same behavior.
> 
> Either the Executor behavior should be fixed or its limitations should 
> be documented.
> 
> Sorry if this post isn't following a specified format typical for 
> mailing listings, I have never participated in one before, but I hope 
> these concerns can be considered.