Notes about Cutting-Edge Technologies and Everything

This is a major release that includes many new features, performance improvements, and hundreds of stability enhancements - see the highlights and more details below. • Elasticsearch 6.  Dremio now supports the latest versions of Elasticsearch. Enjoy full SQL support, including JOINs, Window functions, and accelerated analytics through any BI tool, including Tableau and Power BI. We also added support for compressing Elasticsearch responses to minimize network traffic.  • Approximate count distinct acceleration.  Dremio now supports accelerating count distinct queries based on an approximation-based algorithm (HyperLogLog). This provides a faster and more memory efficient way of providing distinct counts and is especially useful in high cardinality scenarios with very large datasets.  • Faster ORC performance.  Data encoded in ORC is now significantly faster to access and more memory efficient for ORC managed in Hive sources.  • Support for AWS GovClou...

Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN. Livy provides the following features:     Interactive Scala, Python, and R shells     Batch submissions in Scala, Java, Python     Multiple users can share the same server (impersonation support)     Can be used for submitting jobs from anywhere with REST     Does not require any code change to your programs     Support Spark1/ Spark2, Scala 2.10/2.11 within one build. Architecture Core Functionalities Livy offers three modes to run Spark jobs:     Using programmatic API     Running interactive statements through REST API     Submitting batch applications with REST API In the following sections, I will provide the details of these 3 modes. Details:  https://hortonworks.c...