Issue #71
Curated list of blogs, videos, papers, podcasts on programming and distributed systems.
One day, in retrospect, the years of struggle will strike you as the most beautiful.
- Sigmund Freud
Flexibility With Traps in Kafka
Here I will explain what’s going on in a few configurations, how they can impact the entire system and what traps (problems) can wait for us - #websystemer
How LinkedIn customizes Apache Kafka for 7 trillion messages per day
We maintain over 100 Kafka clusters with more than 4,000 brokers, which serve more than 100,000 topics and 7 million partitions. The total number of messages handled by LinkedIn’s Kafka deployments recently surpassed 7 trillion per day. - #engineering #linkedin
Why we decided to go for the Big Rewrite
One of the questions that any software engineer will face at one point in their career is when is the BIG rewrite the right thing to do? And is it ever?
In this post we will try to give a more general framework on how to answer this question for a specific project and we will also tell our story of rewriting the core data processing system that powers Channable. - #channable
Here’s What Makes Apache Flink scale
A glance at the Memory management and Network flow control - #medium #walmartlabs
AWS Elasticsearch: a fundamentally-flawed offering
I’m currently working on a large logging project that was initially implemented using AWS Elasticsearch. Having worked with large-scale mainline Elasticsearch clusters for several years, I’m absolutely stunned at how poor Amazon’s implementation is and I can’t fathom why they’re unable to fix or at least improve it. - #spun
Uber Go Style Guide
The goal of this guide is to manage this complexity by describing in detail the Dos and Don'ts of writing Go code at Uber. These rules exist to keep the code base manageable while still allowing engineers to use Go language features productively. - #github #uber-go
How our security team handle secrets
In the Security team at Monzo, we've been working on a new system to manage secret information, that makes it easier for engineers to work with secrets in a way that's secure, and makes it even harder for attackers to steal information that'd put you at risk. - #monzo #blog
600k concurrent websocket connections on AWS using Node.js
I recently faced the challenge to get as much power as possible out of a AWS EC2 instance at the lowest possible cost using concurrent persistent websockets.
To do this I needed to use a event-driven, non-blocking runtime environment. For this particular purpose Node.js is excellent with its lightweight and fast Chrome V8 engine. - #blog #jayway
The state of ML frameworks in 2019
In 2019, the war for ML frameworks has two remaining main contenders: PyTorch and TensorFlow. My analysis suggests that researchers are abandoning TensorFlow and flocking to PyTorch in droves. Meanwhile in industry, Tensorflow is currently the platform of choice, but that may not be true for long. - #thegradient
Eight Habits of Expert Software Designers: An Illustrated Guide
The best designers employ specific habits, learned practices, and observed principles when they work. Here are a few of them. - #thereader #mitpress
Must read
The Ticket
Sinners, Scalpers and the Search for God: One man’s descent into the underworld of sports - #sbnation
Writing is Thinking: Learning to Write with Confidence
People often ask how I approach writing, so I decided to share this piece to sway the self-conscious writer inside each one of us. I hope it encourages others to develop a practice that enables them to write with confidence, by simply sharing how I’ve designed my own. - #blog #stephsmith
Paper
Towards a Next Generation Data Center Architecture: Scalability and Commoditization
In this paper, we describe Monsoon, a new network architecture, which scales and commoditizes data center networking. Monsoon realizes a simple mesh-like architecture using programmable commodity layer-2 switches and servers. In order to scale to 100,000 servers or more, Monsoon makes modifications to the control plane (e.g., source routing) and to the data plane (e.g., hot-spot free multipath routing via Valiant Load Balancing). - #microsoft
Book
ØMQ - The Guide
ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. - #zeromq
Podcast
Leif Singer on changing from academia to industry, humility and work culture at Automattic, open-source and working remotely
- why he switched from academia to industry,
- how he got hired without knowing the company’s tech stack,
- why he worked for several months at Automattic before being hired,
- how the Automattic creed, influences, and guides company values and mindset,
- Automattic’s software engineering practices,
- and how they fight technical debt during quarterly scheduled clean-up weeks. - #software-engineering-unlocked