Lightbend Fast Data Platform 1.2 Released

By Craig Blitz (@cblitz) August 2, 2018

Lightbend is happy to announce the release of version 1.2 of Fast Data Platform. Lightbend Fast Data Platform includes the Fast Data Platform Manager application (shown below) for full lifecycle management and monitoring of Fast Data Platform Clusters. Fast Data Platform also includes one-stop-shop vendor support for Kafka, Spark, Akka Streams, Kafka Streams, HDFS, the Lightbend Enterprise Suite, integration with OpsClarity for monitoring, and support tools.

The release also includes a growing suite of sample applications for learning how to build streaming applications. A preview release of Hive on Spark is included, along with “extra” tools for installing third-party “certified” components, including Yahoo! Kafka Manager and Quantifind Kafka Offset Monitor (with enhancements from Morningstar) for production management and monitoring of Kafka, the streaming engine Flink, and to support our sample applications, the notebook environment Zeppelin, time-series database InfluxDB, and the dashboard tool Grafana.

There are several new features in this release. They include updates to several component versions, a greatly simplified process for deploying the sample apps, improvements to Spark and Kubernetes integration on Mesos, and improved customization options for components during installation. Further details are provided in the documentation link below.

For production use, Lightbend Fast Data Platform recommends Mesosphere DC/OS v1.11.1 or newer, Enterprise Edition. DC/OS Open Source can be used for non-production scenarios, such as development and testing. This release of DC/OS introduces commercial support for Kubernetes integrated with DC/OS. Kubernetes is the preferred way to deploy Lightbend Reactive Platform applications.. For customers who don’t need Kubernetes, release 1.10.4 can also be used. The documentation for this release can be found here: https://developer.lightbend.com/docs/fast-data-platform/1.2.0/

Differences from 1.1.0

Fast Data Platform 1.2.0 includes a number of improvements.

Updates to Several Component Versions

Spark 2.2 with Patches

The Spark build in this release includes several patches to the integration between Spark, Hive, and Kerberos, when running on Mesos environments. The patches are bundled into a custom Lightbend build lightbend-2.3.1-2.2.1-2-hadoop-2.6.5-01 and lightbend-2.3.1-2.2.1-2-hadoop-2.7.3-01 of the corresponding Mesosphere DC/OS Spark builds (2.3.1-2.2.1-2-*). They are based on Spark 2.2.1. Details of these patches can be found in our release notes.

Hive on Spark Preview

Customers looking to migrate Hive-based, batch workloads to hybrid batch-streaming workloads based on SparkSQL can use our preview support for Hive with Spark as the execution engine, running in Fast Data Platform on DC/OS, first introduced in our 1.1 release.

While still in preview, this release upgrades Hive to its 3.0.0 GA release and the underlying Spark engine to the 2.2 build with the patches discussed above. See Installing Hive on Spark for more details.

Sample Applications

We have revamped the way we deploy pre-built sample apps. Instead of one, large Docker image with all the apps, each sample app is now available in its own Docker image. Instructions are included in the source code distribution for running these Docker images. You can also modify, build, and deploy the applications yourself.

Fast Data Platform Manager

We continue to improve customization options for components during installation, including the following:

  • It’s now possible to assign custom service ids and to run platform services under service groups. This supports large clusters where different teams need to run different instances of Fast Data Platform services with different access controls.
  • A new command-line tool makes it possible to install all of the Fast Data Platform components with one command, where the details are driven using a JSON configuration file. This supports organizations with large clusters who want to install platform instances from a “template”, for example in a DevOps pipeline. Fast Data Platform Manager must still be installed first, but this is also done using a command-line tool.

Flink

We’ve upgraded Flink support from preview to certified, which means we consider Flink on DC/OS to be suitable for production workloads, but we currently don’t offer production support for Flink. We are working with data Artisans, the company behind Flink, on a plan to offer production support.

DC/OS 1.11

Lightbend Fast Data Platform now supports Mesosphere DC/OS v1.11.1 or newer, Enterprise Edition, for production use. DC/OS Open Source can be used for non-production scenarios, such as development and testing. DC/OS v1.10.4 is still supported.

Looking Forward

As Mesosphere support for Kubernetes continues to improve, look for Lightbend to leverage that support. We are currently building Kubernetes-based functionality to provide new developer and devops tooling that makes it easier to create and manage applications that span multiple streaming data components.

Contact Lightbend for information about obtaining Fast Data Platform.