The importance of machine learning has not gone unnoticed, with 64 percent of the 2015 Spark Survey respondents using Spark for advanced analytics and 44 percent creating recommendation systems.

In concert with the shift to DataFrames, most applications today are using the Spark SQL engine, including many data science applications developed in Python and Scala languages. The Spark SQL engine ...

For instance, with Apache Spark having been written in Scala and optimized for running Scala or Java programs, this often left R and Python developers out in the cold.