What’s new in Hortonworks DataFlow 3.3?

With the upcoming HDP 3.1 release, we also bring about some exciting innovations to enhance our Kafka offering –

• New Hive Kafka Storage Handler (for SQL Analytics) – View Kafka topics as tables and execute SQL via Hive with full SQL Support for joins, windowing, aggregations, etc.
• New Druid Kafka Indexing Service (for OLAP Analytics) – View Kafka topics as cubes and perform OLAP style analytics on streaming events in Kafka using Druid.

HDF 3.3 includes the following major innovations and enhancements:

Core HDF Enhancements

• Support for Kafka 2.0, the latest Kafka release in the Apache community, with lots of enhancements into security, reliability and performance.
• Support for Kafka 2.0 NiFi processors
• NiFi Connection load balancing – This feature allows for bottleneck connections in the NiFi workflow to spread the queued-up flow files across the NiFi cluster and increase the processing speed and therefore lessen the effect of the bottleneck.
• MQTT performance improvements including handling a higher velocity of messages streaming from field IoT devices

Enhanced Streaming Support

• Kafka Streams Support
• Kafka Streams is now an officially supported component
• Integration with Schema Registry, Ranger and Streams Messaging Manager (SMM)
• Supports fully Kerberized/Rangerized Kafka clusters

Cross Platform Integrations

• Kafka 2.0 – Ambari and Ranger
• Ambari support to install, configure, manage Kafka 2.0 multi-node secure clusters.
• Ranger support for new ACL like topic symmetric create/delete level permissions
• Kafka Streams – Schema Registry and Ranger
• Manage security policies of streams apps in Ranger
• Schema Registry Serializer/Deserializer support for Streams Apps
• KNOX SSO Support
• Knox SSO support for Schema Registry and SAM

Enhanced Operations/Administrative Features

• Site to Site (S2S) Reporting Task – Component name filtering
• Allow administrators to efficiently capture the desired scope of provenance data from the running systems.
• Decommission NiFi nodes in a cluster – New API feature that allows for an administrator to completely remove a NiFi node from a cluster. This feature doesn’t just shut down the node but rather ensures that all data (content, and flowfile) is offloaded from the NiFi node, pausing flow execution until this is complete. Once all of the data is offloaded from the node the framework will then gracefully remove the node from the cluster. This guarantees that no data is left on the local storage of the NiFi node that is removed from the cluster.
• Ambari support for Express and Rolling upgrades from HDF 3.2
• Smartsense can be used in an HDF 3.3 installation without requiring HDP.
• Explicit fine-grained Ranger permissions for starting/stopping processor groups within the NiFi canvas
• Ranger integration with NiFi Registry

Original article >>>