The Apache Spark community has improved support for Python to such a great degree over the past few years that Python is now a “first-class” language, and no longer a “clunky” add-on as it once was, ...

In concert with the shift to DataFrames, most applications today are using the Spark SQL engine, including many data science applications developed in Python and Scala languages. The Spark SQL engine ...