LinkedIn Kafka with Nacho Solis (Repeat)


Originally published October 18, 2019

Apache Kafka was created at LinkedIn. Kafka was open sourced in 2011, when the company was eight years old. By that time, LinkedIn had developed a social network with millions of users. LinkedIn’s engineering team was building a range of externally facing products and internal tools, and many of these tools required a high-throughput system for publishing data and subscribing to topics.

Kafka was born out of this need. Over time, Kafka’s importance within LinkedIn has only grown. Kafka plays a central role for services, log management, data engineering, and compliance. LinkedIn might be the biggest user of Apache Kafka in the entire software industry. Kafka has many use cases, and it is likely that they are almost all on display within LinkedIn.

Nacho Solis is a senior software engineering manager at LinkedIn, where he helps teams build infrastructure for Kafka, as well as Kafka itself. Nacho joins the show to discuss the history of Kafka at LinkedIn, and the challenges of managing such a large deployment of Kafka. We also talk about streaming, data infrastructure, and more general problems in the world of engineering management.

Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Sponsors

Retool gives you a drag and drop interface that makes it super simple to build internal UIs in hours, not days. Companies like DoorDash, Brex, Pinterest, and even Amazon use Retool to build internal tools super fast. Stop wasting your time. Use Retool. Visit retool.com/sedaily.

Triplebyte is a network of 200,000+ Top Engineers. Triplebyte works with more than 400 tech companies including Coinbase, Zoox, Dropbox, and Facebook.  Triplebyte is focused on matching high-quality engineers with great jobs. Let the right roles come to you. Want to know your strengths? Take the Triplebyte quiz and receive your personalized feedback report. Tracks offered: Generalist, Front End Mobile, Machine Learning, DevOps, DataScience, and Entry Level. Visit triplebyte.com/sedaily.