Notes about Cutting-Edge Technologies and Everything

Here’s what’s NEW: Up to 100x performance improvement for a wide range of query workloads, using Apache Arrow’s new kernel – Gandiva. Gandiva performs just-in-time compilation of SQL queries to machine code to get the fastest possible performance. (Our blog post explains more about how.) Support for Teradata, Azure Data Lake Store, AWS S3 GovCloud, and the latest version of Elasticsearch. Expect more soon! We’ve got a new connector framework that improves performance, stability, and development velocity for all data sources. Cluster Workload Manager, which lets you deploy diverse workloads on a single operational cluster while ensuring critical SLAs for performance and availability. More data catalog features, including wikis and tags for your data sets. That makes it even easier to discover, organize, curate, and share datasets from all your data sources.  Improved security and governance controls, like end-to-end encryption over TLS, and integration with Apache Ranger, ...

Kubernetes is really cool because managing services as flocks of little containers is a really cool way to make computing happen. We can get away from the idea that the computer will run the program and get into the idea that a service happens because a lot of little computing just happens. This idea is crucial to making reliable services that don’t require a ton of heroism to stand up or keep running. But there is a dark side here. Containers want to be agile because that is the point of containers in the first place. We want containers because we want to make computing more like a gas made up of indistinguishable atoms instead of like a few billiard balls with colors and numbers on their sides. Stopping or restarting containers should be cheap so we can push flocks of containers around easily and upgrade processes incrementally. If ever a container becomes heavy enough that we start thinking about that specific container, the whole metaphor kind of dissolves. So that metap...