Compressed Kafka Topics

Image by Greg Montani from Pixabay

Text messages may consume quite a lot of space in Kafka topics. However, text may usually be compressed with a high ratio, so sometimes it makes sense to enable topic compression. In this article, I’m going to show how easy it is to set up compression. I’m also going to check out the practical difference between compressed vs. uncompressed topics.

First, let’s generate a 1MB file with some repeating text, like “This is some test data”. It’s very easy using the Linux “yes” command:

> yes This is some test data | head -c 1MB > test.file

Now download Kafka and spin up Zookeeper and Kafka broker, just as described in the documentation, using default settings:

> tar -xzf kafka_2.12-2.5.0.tgz
> cd kafka_2.12-2.5.0
> bin/zookeeper-server-start.sh config/zookeeper.properties
> bin/kafka-server-start.sh config/server.properties

Next, create two topics — test-topic and test-topic-zip. The configuration is the same so far:

> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test-topic
> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test-topic-zip

Update the properties of the second topic to make it use compression. I set the compression.type…