Notes about Cutting-Edge Technologies and Everything

The Problem (part 1) Cassandra is a highly-distributable NoSQL database with tunable consistency. What makes it highly distributable makes it also, in part, vulnerable: the whole deployment must run on synchronized clocks. It’s quite surprising that, given how crucial this is, it is not covered sufficiently in literature. And, if it is, it simply refers to installation of a NTP daemon on each node which – if followed blindly – leads to really bad consequences. You will find blog posts by users who got burned by clock drifting. In the first installment of this two part series, it is covered how important clocks are and how bad clocks can be in virtualized systems (like Amazon EC2) today. Details: https://blog.rapid7.com/2014/03/14/synchronizing-clocks-in-a-cassandra-cluster-pt-1-the-problem/ Solutions (part 2) Some disadvantages of off-the-shelf NTP installations, and how to overcome them. Details: https://blog.rapid7.com/2014/03/17/synchronizing-clocks-in-a-cassandra-cluster...

The central idea of the algorithm is to generate candidate tokens, and figure out what would be the effect of adding each of them to the ring as part of the new node. The new token will become primary for part of the range of the next one in the ring, but it will also affect the replication of preceding ones. The algorithm is able to quickly assess the effects thanks to some observations which lead to a simplified but equivalent version of the replication topology2: Replication is defined per datacentre and replicas for data for this datacentre are only picked from local nodes. That is, no matter how we change nodes in other datacentres, this cannot affect what replicates where in the local one. Therefore in analysing the effects of adding a new token to the ring, we can work with a local version of the ring that only contains the tokens belonging to local nodes. If there are no defined racks (or the datacentre is a single rack), data must be replicated in distinct nodes. If racks ...

In Distributed Computing the notion of "when-ness" is fundamental; Lamport's " Time, Clocks, and the. Ordering of Events in a Distributed System"  paper is considered one of the foundational pieces of work. But what about locally? The conclusion is that except for the special case of using nanoTime() in micro benchmarks, you may as well stick to currentTimeMillis() —knowing that it may sporadically jump forwards or backwards. Because if you switched to nanoTime(), you don't get any monotonicity guarantees, it doesn't relate to human time any more —and may be more likely to lead you into writing code which assumes a fast call with consistent, monotonic results. Details: http://steveloughran.blogspot.ca/2015/09/time-on-multi-core-multi-socket-servers.html 

This approach is widely used in both multi-leader replication and leaderless databases such as Cassandra. Details: https://aphyr.com/posts/294-jepsen-cassandra

Service discovery is a key component of most distributed systems and service oriented architectures. The problem seems simple at first: How do clients determine the IP and port for a service that exist on multiple hosts? This problem has been addressed in many different ways and is continuing to evolve. Let's look at some open-source or openly-discussed solutions to this problem to understand how they work. Specifically, let's look at how each solution uses strong or weakly consistent storage, runtime dependencies, client integration options and what the tradeoffs of those features might be.  We’ll start with some strongly consistent projects such as Zookeeper , Doozer and Etcd which are typically used as coordination services but are also used for service registries as well. We’ll then look at some interesting solutions specifically designed for service registration and discovery. We’ll examine Airbnb’s SmartStack , Netflix’s Eureka , Bitly’s NSQ , Serf , Spotify and DNS an...