Remember that first time you saw Service Broker and thought of all the great things you could do with it? Remember when you actually tried Service Broker and found all the limitations and pain points? Enter Apache Kafka. Apache Kafka is a distributed publish-subscribe messaging system which can scale out to handle millions of messages per second and support a distributed, microservices-oriented architecture. In this talk, we will cover the basics of this powerful system, including general architectural and design principles. We will also build a .NET project which uses Kafka to produce and consume messages quickly and efficiently.


The slides are available in HTML 5 format. All modern browsers (including tablets and phones) should be able to navigate the slides successfully.

The slides are licensed under Creative Commons Attribution-ShareAlike.

Demo Code

The demonstration code is available on my GitHub repository. This includes all of the console applications and Kafka scripts to work with topics.

To get the flight data we loaded and the airport data we used to enrich our data sets, check out the Data expo '09 contest's set of flight data, including the 2008 flight data and airports CSV file.

The source code is licensed under the terms offered by the GPL. The slides are licensed under Creative Commons Attribution-ShareAlike.

Additional Media

I presented a version of this talk at NDC Oslo in 2018. You can get the recording on the NDC Youtube channel.

Links And Further Information

Kafka Links

Kafka .NET Providers

At this point, I highly recommend using the Confluent .NET client for Kafka due to its official support. Nevertheless, I think it's valuable to have an idea of other Kafka libraries which are available, as until late 2016, there were several viable alternatives.

F# Links