Notes about Cutting-Edge Technologies and Everything

To gather insights on the current and future state of open source software (OSS), we talked to 31 executives. This is nearly double the number we speak to for a research guide and believe this reiterates the popularity of, acceptance of, and demand for OSS. We began by asking, "What Open Source software do you use?" As you would expect, most respondents are using several versions of open source software. Here's what they told us: Apache Apache Cassandra, Elassandra  (ElasticSearch + Cassandra) , Spark, and Kafka  (as the core tech we provide through our managed service) are the big ones for us. We find that the governance arrangements and independence of the Apache Foundation make a great foundation for strong open source projects. 95% of what we do with big data is open source. We use  Apache Hadoop  and contribute back to grow skills and expertise. We use so much that it would be impossible to list. The core of our software is based on  Apache So...

What happens if the data on disk is corrupted, or the data is wiped out due to hardware error or misconfiguration? Here is the problem that losing disk state really induces in an ensemble of ZooKeeper servers: https://fpj.me/2015/05/28/dude-wheres-my-metadata/

The leader election and failure detection mechanisms are fairly mature, and typically just work… until they don’t. Four different bugs resulting in random cluster-wide lockups. Two of those bugs laid in ZooKeeper, and the other two were lurking in the Linux kernel. This is the story. https://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-poison-packet/

Service discovery is a key component of most distributed systems and service oriented architectures. The problem seems simple at first: How do clients determine the IP and port for a service that exist on multiple hosts? This problem has been addressed in many different ways and is continuing to evolve. Let's look at some open-source or openly-discussed solutions to this problem to understand how they work. Specifically, let's look at how each solution uses strong or weakly consistent storage, runtime dependencies, client integration options and what the tradeoffs of those features might be.  We’ll start with some strongly consistent projects such as Zookeeper , Doozer and Etcd which are typically used as coordination services but are also used for service registries as well. We’ll then look at some interesting solutions specifically designed for service registration and discovery. We’ll examine Airbnb’s SmartStack , Netflix’s Eureka , Bitly’s NSQ , Serf , Spotify and DNS an...