Posted on

Gossips in Distributed Systems: Physalia

Reading Time: 6 minutes

I often take notes and jot down observations when I read academic/industry papers.   Thinking of a name for this series ‘Gossips in Distributed Systems’ seemed apt to me, inspired by the gossip protocol with which peers in these systems communicate with each other which mimics the spread of ideas and technologies among practitioners and people alike. The goal of this series would be to do a round-up of any new concepts or papers in computer science (often in distributed systems but not always) and share my thoughts and observations.

Today, we are going to talk about the Physalia paper from AWS: “Millions of Tiny Databases”.  This is inspired by Physalia or  Portuguese man-of-war (pictured), a siphonophore, or a colony of organisms.   Overall, the paper, even though slightly on the longer side, is chock full of details and best practices pertaining to design, architecture, and testing of distributed systems.

Credits: https://en.wikipedia.org/wiki/Portuguese_man_o%27_war

Given the size of the paper and the wide gamut of topics that it touches, we will be discussing only a few aspects of the paper in this post along with some observations. In subsequent sequels, we will go into others in further detail. 

Before proceeding ahead, the present EBS architecture with Physalia has a primary EBS volume (connected to EC2 instance) and a secondary replica, and data flowing from instance to primary and replica in that order. Also, this chain replication is strictly within an Availability Zone (AZ) mainly due to inter-AZ latencies being prohibitive. The pre-Physalia architecture had a similar replication chain but with the control plane also being part of EBS itself rather than a separate database (which we will soon find out was not a good idea).

Credits: Screenshot of figure in the Physalia paper

Raison d’être

All good-to-great systems have a story that necessitated their existence. In this case, it was an outage of the us-east-1 region in 2011 caused by overload and subsequent cascading failure which necessitated a more robust control plane for failure handling. The postmortem of that outage is here, it is quite long and wordy, so I will summarize it here. 

Continue reading “Gossips in Distributed Systems: Physalia”
Posted on

Circuit Breakers: Stock Markets and Distributed Systems

Reading Time: 4 minutes

There are many parallels between the stock markets and the distributed systems in computer science. This post, in particular, is about circuit breakers prevalent in them for better resilience against irrational exuberance and upstream service errors respectively. In particular, this is about exploring breakers in stock markets from a distributed systems perspective. 

Of late, the US stock market (and the non-US ones too) has been quite volatile, to say the least. CBOE VIX index  (also known as the ‘fear index’) which measures implied volatility in the S&P 500 index, has traversed a wide range between highs and lows, fluctuating several points on many of the days in March and April, peaking at 82.69 on 16th March (stock market bottomed out on 23 March, both as of May 10, 2020).  

Continue reading “Circuit Breakers: Stock Markets and Distributed Systems”
Posted on

Haiku and Muffin Top

Reading Time: 3 minutes

My interest in haikus was recently rekindled by James May’s Our Man in Japan series in which he frequently bookends the episodes with a haiku of his own. Accordingly, I started searching for a haiku ebook on Libby (which if you are not using, you should give it a try!) and I found Haiku: Classic Japanese Short Poems.  The book has haikus in Romaji, English, and vertical Japanese. The book is spartan in its use of graphics and has a very simple structure much like the subject matter it deals with. There are about 60-70 haikus in it from the four great haiku masters.

Haiku, for those hearing about it for the first time, is a form of Japanese poetry conforming to certain constraints on syllables (5-7-5) and split over three lines. The book itself does not delve sufficiently enough into the history of haiku, though the Wikipedia page on Haiku proves more than sufficient here. Listing haikus often commence with the famous Matsuo Bashō’s old pond haiku which goes like: 

As it should be evident at this point, they are remarkably short, and often involve nature and seasons. They behoove the reader to invoke their imagination and experience the verses prima facie.  To help with this, the haiku verses are often read slowly, more akin to sipping tea gently. A tabula rasa-esque mindset, for that short period, helps. As a result, the haiku focuses as much on the material as the reader too, a dichotomy similar to an artwork in a gallery and a keen observer. 

Continue reading “Haiku and Muffin Top”