Apache Livy - A REST Interface for Apache Spark

Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN. Livy provides the following features:

•     Interactive Scala, Python, and R shells
•     Batch submissions in Scala, Java, Python
•     Multiple users can share the same server (impersonation support)
•     Can be used for submitting jobs from anywhere with REST
•     Does not require any code change to your programs
•     Support Spark1/ Spark2, Scala 2.10/2.11 within one build.

Architecture


Core Functionalities

Livy offers three modes to run Spark jobs:

•     Using programmatic API
•     Running interactive statements through REST API
•     Submitting batch applications with REST API

In the following sections, I will provide the details of these 3 modes.

Details: https://hortonworks.com/blog/livy-a-rest-interface-for-apache-spark/
Guide: https://dzone.com/articles/quick-start-with-apache-livy