Introduction to PySpark

Shagun Sodhani

Agenda

• Introduction to Spark.
• How to use Spark with Python.
  • PySpark Shell
  • IPython/Jupyter Notebooks
  • Python Scripts

What is Apache Spark?

• Apache Spark™ is a fast and general engine for large-scale data processing
• Originally developed in AMPLab at UC Berkely (2009), open-sourced in 2010, transferred to Apache 2013
• Claims to run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.

Advantages Over MapReduce

• Speed.
• Ease of Use - Scala, Java, Python, R.
• Generality - Supports different use cases.
• Runs Everywhere - Hadoop, Mesos, standalone ...

Spark Components

• Spark SQL and DataFrames
• Spark Machine Learning (MLlib)
• GraphX
• Spark Streaming

Talk is cheap. Show me the code.

Linus Torvalds

http://bit.ly/PySparkDemo

Resources

• http://spark.apache.org/documentation.html
• https://databricks.com/spark/developer-resources
• http://spark-packages.org

Shagun Sodhani

https://twitter.com/shagunsodhani/

https://github.com/shagunsodhani/

Thank You