Gossips in Distributed Systems: Physalia

I often take notes and jot down observations when I read academic/industry papers.   Thinking of a name for this series ‘Gossips in Distributed Systems’ seemed apt to me, inspired by the gossip protocol with which peers in these systems communicate with each other which mimics the spread of ideas and technologies among practitioners and people alike. The goal of this series would be to do a round-up of any new concepts or papers in computer science (often in distributed systems but not always) and share my thoughts and observations.

Today, we are going to talk about the Physalia paper from AWS: “Millions of Tiny Databases”.  This is inspired by Physalia or  Portuguese man-of-war (pictured), a siphonophore, or a colony of organisms.   Overall, the paper, even though slightly on the longer side, is chock full of details and best practices pertaining to design, architecture, and testing of distributed systems.

Given the size of the paper and the wide gamut of topics that it touches, we will be discussing only a few aspects of the paper in this post along with some observations. In subsequent sequels, we will go into others in further detail. 

Before proceeding ahead, the present EBS architecture with Physalia has a primary EBS volume (connected to EC2 instance) and a secondary replica, and data flowing from instance to primary and replica in that order. Also, this chain replication is strictly within an Availability Zone (AZ) mainly due to inter-AZ latencies being prohibitive. The pre-Physalia architecture had a similar replication chain but with the control plane also being part of EBS itself rather than a separate database (which we will soon find out was not a good idea).

Credits: Screenshot of figure in the Physalia paper

Raison d’être

All good-to-great systems have a story that necessitated their existence. In this case, it was an outage of the us-east-1 region in 2011 caused by overload and subsequent cascading failure which necessitated a more robust control plane for failure handling. The postmortem of that outage is here, it is quite long and wordy, so I will summarize it here. 

Continue reading “Gossips in Distributed Systems: Physalia”

Slides from PLMCE 2014 breakout session

As many of you already know, PLMCE is an annual MySQL
community conference and Expo organized by Percona in the month of April
(usually). It is a great conference, not only to meet new and eminent people in
MySQL and related database fields, but also to attend interesting talks, and
also to give some.

This year I spoke about synchronous replication at a higher level. The talk was
titled “ACIDic Clusters: Review of current relational databases with synchronous replication”. Having previously given talks with boring titles (but interesting content), this time I decided to go with an interesting title, and it seemed to fit well with topic being discussed.
Continue reading “Slides from PLMCE 2014 breakout session”

My twitter setup

I have been using Twitter for a while under the handle randomsurfer.
I tend to use web interface sometimes, but regularly I use the command-line/ncurses interface.

There are two main clients that I use regularly. One is ttytter
which is a nice command-line client which offers advanced functionality such as
creating/editing lists and scripting. Refer to the linked page for more scoop. I
would say ttytter is one of the best implementations of Twitter API. It allows
for advanced scripting like:

The above invocation can be used to get the list of everyone you follow and
populate the configuration file to allow auto-completion at ttytter prompt. You
can also run ttytter as a daemon.
Continue reading “My twitter setup”

Annoying access keys on web pages

5463604427_8e7998ddcc_b
HTML access keys were introduced to improve web accessibility and
they still seem to serve that purpose. Wikipedia defines them
as “In a web browser, an access key or accesskey allows a computer user to
immediately jump to a specific part of a web page via the keyboard.”
If you
view hover over the link, it should show up in tooltip; for a wikipedia page,
for instance, the ‘view history’ is mapped to ‘alt-shift-h’ by default. In HTML
source, you can see them as accesskey attribute.

Having said that, they can be really annoying many a time. This is true
primarily when you have bound that access key to a mapping in the browser or in
a plugin or even to a global binding (say, with xbindkeys). What makes it even
more worse is that on almost every site it is impossible to disable them.
Continue reading “Annoying access keys on web pages”

MySQL file limit, table cache and max_connections

MySQL variables open_files_limit, table_open_cache and max_connections are
inter-related, and this is for obvious reasons: all deal with file descriptors
one way or another.

If one of the value is provided but others are left out, mysqld calculates
others using a formula and in some cases, emits a warning if not possible.

The whole calculation behind obtaining the final file descriptor limit is a bit
byzantine and is as follows (for Linux):

EDIT: This applies to MySQL 5.5, in 5.6, as Daniël in comments pointed out,
few things have changed, check comment for details. I will probably make a
followup post on the differences.

  1. First it tries to calculate a minimum requirement based on max_connections
    and table_open_cache. As for other variables, if it is not provided in cnf or
    on command line it uses defaults.

    <br />    /* MyISAM requires two file handles per table. */
        wanted_files= 10+max_connections+table_cache_size*2;
    

    Continue reading “MySQL file limit, table cache and max_connections”

Saving form data in firefox

When commenting on sites, I have sometimes, seen that the commenting system just swallows the comment, or there is a browser crash, or a system one. In these cases it would be great if you can recover it somehow, particularly when you typed quite a bit.

There are plugins for firefox and other browsers for this. But since I use Pentadactyl(site), I thought not to use any external plugin but write one for dactyl. But then, I realized a plugin may not be needed.

Following is what I use:
Continue reading “Saving form data in firefox”