The most recent release of Kafka 0.9 with it’s comprehensive security implementation has reached an important milestone. In his blog post Kafka Security 101 Ismael from Confluent describes the security features part of the release very well.
As a part II of the here published post about Kafka Security with Kerberos this post discussed a sample implementation of a Java Kafka producer with authentication. It is part of a mini series of posts discussing secure HDP clients, connecting services to a secured cluster, and kerberizing the HDP Sandbox (Download HDP Sandbox). In this effort at the end of this post we will also create a Kafka Servlet to publish messages to a secured broker.
Kafka provides SSL and Kerberos authentication. Only Kerberos is discussed here.
Kafka from now on supports four different communication protocols between Consumers, Producers, and Brokers. Each protocol considers different security aspects, while PLAINTEXT is the old insecure communication protocol.
- PLAINTEXT (non-authenticated, non-encrypted)
- SSL (SSL authentication, encrypted)
- PLAINTEXT+SASL (authentication, non-encrypted)
- SSL+SASL (encrypted authentication, encrypted transport)
A Kafka client needs to be configured to use the protocol of the corresponding broker. This tells the client to use authentication for communication with the broker:
Making use of Kerberos authentication in Java is provided by the Java Authentication and Authorization Service (JAAS) which is a pluggable authentication method similar to PAM supporting multiple authentication methods. In this case the authentication method being used is GSS-API for Kerberos.
For JAAS a proper configuration of GSS would be needed in addition to being in possession of proper credentials, obviously. Some credentials can be created with MIT Kerberos like this:
The last line is not necessarily needed as it creates us a so called keytab – basically an encrypted password of the user – that can be used for password less authentication for example for automated services. We will make use of that here as well.
First we need to prepare a test topic to publish messages with proper privileges for our kafka-user:
As a sample producer we will use this:
With this setup we can go ahead demonstrating two ways to use a JAAS context to authenticate with the Kafka broker. At first we will configure a context to use the existing privileges possessed by the executing user. Next we use a so called keytab to demonstrate a password-less login for automated producer processes. At last we will look at a Servlet implementation provided here.
Authentication with User Login
To configure a JAAS config with userKeyTab set to false and useTicketCache to true, so that the privileges of the current users are being used.
We store this in a file under
and exeute the broker like this:
Using Keytab to Login
Next we will configure the JAAS context to use a generated keytab file instead of the security context of the executing user. Before we can do this we need to create the keytab storing it also under
The JAAS configuration can now be changed to look like this:
This will use the keytab stored under /home/kafka-user/kafka-user.keytab while the user executing the producer must not be logged in to any security controller:
Kafka Producer Servlet
In a last example we will add a Kafka Servlet to the hdp-web-sampleproject previously described in this post. Our Servlet will get the topic and message as a GET parameter. The Servlet looks as follwoing: