A Comparison of Java HTTP Clients

Michael Vitz

Dieser Artikel ist auch auf Deutsch verfügbar

As a result of the tendency in new projects to do without heavyweight infrastructure for the integration of different services, in all my recent projects I have had at least one integration via HTTP, usually in connection with JSON. This raises the question which HTTP library we should be using for these integrations.

Not least due to the longevity of Java, there are now a wide range of options open to us. In this article we will therefore take a look at a selection of libraries. We will highlight differences and at the end hopefully be able to make a qualified choice for our specific use case.

Use Case

The use case under consideration here is the reading out of members of a public list on Twitter. To do this we need to execute two HTTP requests. In the first request we receive an OAuth access token from the programming interface. For this purpose we communicate the static, secret API key and the API key secret known to us with the API by means of Basic-Auth. As a response we then receive an access token. We then need to send this with every further request as an HTTP header.

The second request serves the purpose of querying the members of the selected list. In order to keep the example simple, we only parse the JSON of the response and then display it on the default output. For the same reason we also do without pagination.

In order to be able to parse and use JSON, for example to read the access token, we use org.json implementation. But of course we could also use any other implementation. I will mention below whether the libraries support native JSON or do so by means of another implementation.

With the exception of the final example, the main method from Listing 1 is used for execution. This coordinates the two requests and at the end displays the result.

public static void main(String[] args) {
    var accessToken = getAccessToken(TWITTER_API_KEY, TWITTER_API_KEY_SECRET);
    var listMembers = getListMembers(accessToken, "171867803");
    System.out.println(listMembers.toString(2));
}

HttpURLConnection

Already since Java 1.1 and therefore for almost 25 years Java itself provides an implementation for HTTP requests in the form of HttpURLConnection. HttpsURLConnection for HTTPS connections was added to Java 1.4 in February 2002.

In order to generate such a connection, we have to start with a URL. With this we receive the reference to a connection by means of openConnection. In contrast to what the name might imply, a proper connection has not yet actually been made. This only occurs when we explicitly call up connect on the connection or call up a method, such as getResponseCode, that implicitly requires the request to be executed.

Before the actual request starts, we can use various methods to give additional information such as the HTTP method or header to be used. If we also want to send a request body, we have to give advance notice of this by means of setDoOutput. Then we can establish a connection via connect. If we gave advance notice of sending a request body, we can now write this using getOutputStream. Finally, we inspect the response code and read out the body of the HTTP response using getInputStream and parse it in a JSONObject. This can be seen as code in Listing 2, including minimal logging of request and response.

JSONObject performRequest(String method, String uri,
        Map headers, String body) throws IOException {
    final URL url = new URL(uri);

    final var con = (HttpURLConnection) url.openConnection();

    con.setRequestMethod(method);
    headers.entrySet().forEach(header ->
        con.setRequestProperty(header.getKey(), header.getValue()));

    if (body != null) {
        con.setDoOutput(true);
    }

    System.err.println("Performing request: " +
        con.getRequestMethod() + " " + con.getURL() +
        " with headers: " + con.getRequestProperties());

    con.connect();

    if (body != null) {
        try (var w = new PrintWriter(con.getOutputStream())) {
            w.print(body);
        }
    }

    System.err.println("Got response: " +
        con.getResponseCode() + " " + con.getResponseMessage() +
        " with headers: " + con.getHeaderFields());

    if (con.getResponseCode() != HTTP_OK) {
        throw new IllegalStateException("Error: " +
            con.getResponseCode() + " " + con.getResponseMessage());
    }

    try (var in = connection.getInputStream()) {
        return new JSONObject(new JSONTokener(in));
    }
}

From today’s perspective the code comes across as rather unwieldy. Instead of headers the term request properties is used, and the implicit establishing of the actual connection can lead to surprises, depending on the method used. In addition, the nomenclature of getInputStream and getOutputStream always confuses me as I always think from the perspective of the request, so for me the body that I send is the input and the one I receive the output. From the perspective of a stream this is however of course the other way round and thus correct.

Nonetheless, this method allows us to implement our use case without any major problems, as can be seen in Listing 3. This method only covers the part required for our use case. For more complex scenarios it will certainly be necessary to write additional code. The biggest disadvantage however is that there is no support for HTTP/2. Although this may not be a deal-breaker at the moment, it is likely to develop into one.

JSONObject getListMembers(String accessToken, String listId) {
    return performRequest(
            "GET",
            "https://.../1.1/lists/members.json?list_id=" + listId,
            Map.of("Accept", "...",
                "Authorization", "Bearer " + accessToken),
            null);
}

String getAccessToken(String apiKey, String apiKeySecret) {
    var encodedSecret = Base64.getEncoder()
        .encodeToString((apiKey + ":" + apiKeySecret).getBytes(UTF_8));
    return performRequest(
            "POST",
            "https://.../oauth2/token",
            Map.of("Accept", "...",
                "Authorization", "Basic " + encodedSecret,
                "Content-Type", "..."),
            "grant_type=client_credentials")
        .getString("access_token"); }

java.net.http.HttpClient

Alongside the support of HTTP/2, the growth of non-blocking input and output and asynchronous programming was the driver for a new HTTP client in JDK. This was finally made available with Java 11 and is therefore the currently preferred option for HTTP requests in Java without the use of a third-party library.

As can be seen in Listing 4, this API is based above all on the well-known Builder design pattern. In my opinion this creates compact, but still readable and easily understandable code. In addition to the use of send we could also have switched to the asynchronous version with sendAsync.

JSONObject getListMembers(String accessToken, String listId) {
    var client = HttpClient.newBuilder().build();

    var uri = "https://.../1.1/lists/members.json?list_id=" + listId;
    var request = HttpRequest.newBuilder()
        .GET()
        .uri(URI.create(uri))
        .header("Accept", "...")
        .header("Authorization", "Bearer " + accessToken)
        .build();

    var response = client.send(request, BodyHandlers.ofInputStream());

    ...

    return new JSONObject(new JSONTokener(response.body()));
}

String getAccessToken(String apiKey, String apiKeySecret) {
    var client = HttpClient.newBuilder().build();

    var encodedSecret = Base64.getEncoder()
        .encodeToString((apiKey + ":" + apiKeySecret).getBytes(UTF_8));

    var request = HttpRequest.newBuilder()
        .POST(BodyPublishers.ofString("grant_type=client_credentials"))
        .uri(URI.create("https://.../oauth2/token"))
        .header("Accept", "...")
        .header("Authorization", "Basic " + encodedSecret)
        .header("Content-Type", "...")
        .build();

    var response = client.send(request, BodyHandlers.ofInputStream());

    ...

    return new JSONObject(new JSONTokener(response.body()))
        .getString("access_token");
}

I consider this version better than the one based on HttpURLConnection. However, there are still a few features missing for me. For one thing, there are unfortunately no constants for the common HTTP headers and media types. I therefore have to either write the concrete value each time, and frequently look it up beforehand, or define constants myself in every project. In addition, it would have been great to have had a preprepared BodyPublisher for HTML forms. This can however be added on if required, in addition to publishers for other types. The final thing I’m missing is the possibility to register globally with a HttpClient filter or interceptors. Although this can be done via an external library, it would have been good to have included it directly in the JDK.

The fact that we have to Base64 encode and compile the Authorization header for the first HTTP request is due to a quirk of the Twitter API. Theoretically it is possible to resolve the authorization globally and transparently using the Authenticator class. However, the client sticks very precisely to the specification and only uses this class once the server has communicated by means of status code 401 that authorization is required. In this case however Twitter replies with status code 403, thus preventing the use of the authenticator.

Apache HttpClient

It almost goes without saying that Apache Commons also offers an HTTP client. This hasn’t been around for quite as long as the original implementation in JDK, but is still over 20 years old. The current client has emerged from the Apache Commons project and is a self-contained solution within Apache known as HttpComponents.

This client also uses the Builder pattern in its programming interface, above all in order to generate the desired request. A particular feature, which can be seen in Listing 5, is that some of the generated objects, such as the client itself or the response, can be closed after use by means of the close function. Although using try-with-resources this is easy to write.

JSONObject getListMembers(String accessToken, String listId) {
    try (var client = HttpClients.custom().build()) {
        var request = ClassicRequestBuilder.get()
            .setUri("https://.../1.1/lists/members.json?list_id=" + listId)
            .addHeader(ACCEPT, JSON)
            .addHeader("Authorization", "Bearer " + accessToken)
            .build();

        try (var response = client.execute(request)) {
            if (response.getCode() != HttpStatus.SC_OK) {
                throw new IllegalStateException(...);
            }
            try (var in = response.getEntity().getContent()) {
                return new JSONObject(new JSONTokener(in));
            }
        }
    }
}

String getAccessToken(String apiKey, String apiKeySecret) {
    try (var client = HttpClients.custom()
            .addRequestInterceptorFirst((request, details, ctx) -> {
                System.out.println("Executing request: " + request +
                    " with headers: " + Arrays.asList(request.getHeaders()));
            }).build()) {

        var basicAuth = new BasicScheme();
        basicAuth.initPreemptive(new UsernamePasswordCredentials(
            apiKey, apiKeySecret.toCharArray()));

        var context = HttpClientContext.create();
        context.resetAuthExchange(
            new HttpHost("https", "api.twitter.com", 443), basicAuth);

        var request = ClassicRequestBuilder.post()
            .setUri("https://.../oauth2/token")
            .addHeader(ACCEPT, APPLICATION_JSON.getMimeType())
            .addHeader(CONTENT_TYPE, APPLICATION_FORM_URLENCODED.getMimeType())
            .addParameter("grant_type", "client_credentials")
            .build();

        try (var response = client.execute(request, context)) {
            if (response.getCode() != HttpStatus.SC_OK) {
                throw new IllegalStateException(...);
            }
            try (var in = response.getEntity().getContent()) {
                return new JSONObject(new JSONTokener(in))
                    .getString("access_token");
            }
        }
    }
}

For the validation of the status code, the applied media types, and the headers, Apache HttpClient has many constants in the classes HttpStatus, ContentType, and HttpHeaders that make it easier for us to find the appropriate values. In addition, as can be seen in Listing 5, there is the option of registering request and response interceptors during the generation of a client.

Here too however we hit the problem of the Twitter API and Basic-Auth. In contrast to the new Java client however, here we are given the opportunity to configure preemptive authentication. This ensures that the client sends the Authorization header even if the server has not requested it.

Needless to say, Apache HttpClient now also supports HTTP/2 and also offers us greater configurability. For example, we can swap the way in which network sockets are created; the client supports the remembering of cookies and sends them with the next request, similar to the behavior of a browser; and the incorporation of a cache for responses is possible. There is also an asynchronous programming interface.

Retrofit

All three of the HTTP clients presented above have in common that they are based on a very traditional programming model. We generate a client, specify what type of request we want to execute, implement it, and then process the response.

Retrofit on the other hand decided to take a different path. Here we define the HTTP requests to be executed via an interface, and with additional annotations if required. How this looks for the two requests in our example can be seen in Listing 6.

interface Twitter {

    @POST("/oauth2/token")
    @FormUrlEncoded
    @Headers("Accept: application/json; charset=utf-8")
    Call getAccessToken(
            @Header("Authorization") String authorization,
            @Field("grant_type") String grantType);

    @GET("1.1/lists/members.json?count=5000")
    @Headers("Accept: application/json; charset=utf-8")
    Call getListMembers(
            @Header("Authorization") String authorization,
            @Query("list_id") String listId);
}

JSONObject getListMembers(Twitter client, String accessToken, String listId) {
    var request = client.getListMembers("Bearer " + accessToken, listId);

    var response = request.execute();

    if (!response.isSuccessful()) {
        throw new IllegalStateException(...);
    }
    return response.body();
}

String getAccessToken(Twitter client, String apiKey, String apiKeySecret) {
    var request = client.getAccessToken(
        Credentials.basic(apiKey, apiKeySecret),
        "client_credentials");

    var response = request.execute();
    if (!response.isSuccessful()) {
        throw new IllegalStateException(...);
    }

    return response.body().getString("access_token");
}

The centerpiece here is our own Twitter interface, in which the actual API is depicted. The two methods getListMembers and getAccessToken serve only to organize all examples for these articles in the same way, and otherwise only verify if the request was successful.

As we define the program using an interface and Retrofit dynamically generates an implementation for us at runtime, in contrast to the previous examples here we have to extend our main method with the code in Listing 7. Here it can also be seen that we register a Converter. These converters are used to convert the body of the HTTP response in the return types or method parameters into the body of the HTTP requests or strings for HTTP headers, query parameters, or path parameters. In our case the implementation presented in Listing 8 is relevant in order to be able to handle JSONObject as the return value. Through its own submodules, Retrofit offers several off-the-shelf converters, which support above all libraries with data binding, such as Jackson.

var retrofit = new Retrofit.Builder()
    .baseUrl("https://api.twitter.com")
    .addConverterFactory(new OrgJsonFactory())
    .client(new OkHttpClient.Builder()
        .addNetworkInterceptor(chain -> {
            System.out.println("Executing request: " + chain.request());
            var response = chain.proceed(chain.request());
            System.out.println("Got response: " + response);
            return response;
        })
        .build())
    .build();
var twitter = retrofit.create(Twitter.class);

class OrgJsonFactory extends Converter.Factory {
    @Override
    public Converter responseBodyConverter(
            Type type, Annotation[] annotations, Retrofit retrofit) {
        return (Converter) value -> {
            try (InputStream in = value.byteStream()) {
                return new JSONObject(new JSONTokener(in));
            }
        };
    }
}

For the actual communication via HTTP, Retrofit uses OkHttp. This gives us the option of defining interceptors or undertaking further configurations, such as the use of HTTP proxies or response caches.

In this article we learned about four options for communicating via HTTP in Java projects – HttpURLConnection, java.net.http.HttpClient, Apache HttpClient, and Retrofit. In addition to these four there are many other candidates that we could use. For example, OkHttp, which is used by Retrofit, can also be used alone. Feign offers us a programming model similar to Retrofit, while larger frameworks, such as Spring, include an HTTP client.

The biggest differences between the options are found in the APIs: programmatic or declarative via interface and annotations. But whether the client supports an asynchronous programming model and whether we can define interceptors, for example for request and response logging, can also be relevant. We should also consider whether the client supports HTTP/2.

Personally I prefer the programmatic over the interface-based programming model as I have more direct control, and I generally try to use an existing client, without introducing a new dependency into a project.

The complete example code, including code for the clients not presented in detail here, can be found at https://github.com/mvitz/javaspektrum-http-clients.