PySpark Tutorial for Beginners Spark with Python
In this PySpark Tutorial, Spark with Python with Examples, you will learn what is PySpark
its features, advantages, modules, passages, and how to use RDD and Dataframe with sample
examples in Python code. All Spark examples provided in this PySpark, Spark with Python Tutorial are basic, simple
and easy to practice for beginners who are enthusiastic to learn PySpark and advance
their careers in big data and machine learning. Note, in case you can't find the PySpark examples you are looking for on this tutorial page
I would recommend using the search option from the menu bar to find your tutorial and
sample example code. There are hundreds of tutorials in Spark, Scala, PySpark, and Python on this website
you can learn from. If you are working with a smaller dataset and don't have a Spark cluster, but still
you wanted to get benefits similar to Spark Dataframe, you can use Python Pandas Dataframes
The main difference is Pandas Dataframe is not distributed and run on a single node
Before we jump into the PySpark Tutorial first, let's understand what is PySpark and how it
is related to Python, who uses PySpark, and its advantages. PySpark is a Spark library written in Python to run Python applications using Apache Spark capabilities
Using PySpark we can run applications parallely on the distributed cluster, multiple nodes
In other words, PySpark is a Python API for Apache Spark. Apache Spark is an analytical processing engine for large-scale powerful distributed data
processing and machine learning applications. Spark, basically written in Scala, and later on due to its industry adaptation, its API
PySpark released to Python using Py4j. Py4j is a Java library that is integrated within PySpark and allows Python to dynamically
interface with JVM objects. Hence, to run PySpark you also need Java to be installed along with Python and Apache Spark
Additionally, for the development, you can use Anaconda distribution, widely used in
the machine learning community, which comes with a lot of useful tools like Spyder IDE
GFETA notebooks, to run PySpark applications. In real time, PySpark is used a lot in the machine learning and data sciences community
thanks to VASP Python machine learning libraries. Spark runs operations on billions and trillions of data on distributed clusters, 100 times
faster than the traditional Python applications. PySpark is very well used in data science and machine learning communities, as there
are many widely used data science libraries written in Python, including NumPy, TensorFlow
Also used due to its efficient processing of large datasets, PySpark has been used by many
organizations like Walmart, Trivago, Sanofi, Runtastic, and many more. Following are the main features of PySpark, in memory computation, distributed processing
using Parallelize can be used with many cluster managers, Spark, Yarn, Mesos, ETC, Fault Tolerance
Immutable, Lazy Evaluation, Cache and Persistence, In-Build Optimization when using DataFrame support and PySQL
Advantages of PySpark, PySpark is a general purpose in memory distributed processing engine
that allows you to process data efficiently in a distributed fashion. Applications running on PySpark are 100x faster than traditional systems
You will get great benefits using PySpark for data ingestion pipelines. Using PySpark, we can process data from Hadoop, HDFS, AWSS3 and many file systems
PySpark also is used to process real-time data using streaming and Kafka
Using PySpark streaming, you can also stream files from the file system and also stream from the socket
PySpark natively has machine learning and graph libraries. Apache Spark works in a master-slave architecture where the master is called driver and slaves are called workers
When you run a Spark application, Spark driver creates a context that is an entry point to your application
and all operations, transformations and actions are executed on worker nodes and the resources are managed by cluster manager
As advised in this Spark with Python PySpark tutorial, Spark supports below cluster managers
Standalone, a simple cluster manager included with Spark that makes it easy to set up a cluster
Apache Matos Mesons is a cluster manager that can also run Hadoop macrodute and PySpark applications
Hadoop will arm the resource manager in Hadoop 2. This is mostly used cluster manager
Kubernetes, an open-source system for automating deployment, scaling and management of containerized applications
Local, which is not really a cluster manager but still I wanted to mention as we use local for master
in order to run Spark on your laptop computer. PySpark RDD, PySpark RDD, PySpark Datafone and SQL, PySpark SQL, PySpark Streaming, PySpark Streaming
PySpark MLib, PySpark ML, PySpark MLib, PySpark Graphome, GraphFrame, PySpark UGOR, PySpark Resource
It's new in PySpark 3.0. Besides these, if you wanted to use third-party libraries, you can find them at https://packages.org/
This page is kind of a repository of all Spark third-party libraries
In order to run PySpark examples mentioned in this tutorial, you need to have Python, Spark
and its needed tools to be installed on your computer. Since most developers use Windows for development, I will explain how to install PySpark on Windows
Download and install either Python from python.org or Anaconda Distribution, which includes Python, Spyder IDE and Jupyter Notebook
I would recommend using Anaconda as it's popular and used by the machine learning and data science community
Follow instructions to install Anaconda Distribution and Jupyter Notebook. To run PySpark application, you would need Java 8 or later version hence download the Java version from Oracle
and install it on your system. Post installation, set Java underscore home and path variable
Download Apache Spark by accessing Spark download page and select the link from Download Spark.3
If you wanted to use a different version of Spark and Hadoop, select the one you wanted from dropdowns
and the link on.3 changes to the selected version and provides you with an updated link to download
After download, untar the binary using 7-zip and copy the underlying folder Spark 3.0 0-bin-hadoop-2 7 to c.app
Now set the following environment variables. Download the neutrals.hd file from the neutrals and copy it too
When the materials are different for each Hadoop version, hence download the right version from
Now open the command prompt and type PySpark command to run the PySpark shell
Spark shell also creates a Spark context web UI and by default it can access from http://localhost.4041
Apache Spark provides a suite of web UIs, jobs, stages, tasks, storage, environment, executors and SQL
to monitor the status of your Spark application, resource consumption of Spark cluster and Spark configurations
On Spark web UI, you can see how the operations are executed
Spark history servers. Keep a log of all Spark applications you submit by Spark submit Spark shell
Before you start, first you need to set the below config on sparkdefaults.conf
Now start the Spark history server on Linux or Mac by running
If you are running Spark on Windows, you can start the history server by starting the below command
By clicking on each app ID, you will get the details of the application in PySpark web UI
To write PySpark applications, you would need an IDE. There are tens of IDEs to work with and I choose to use Spyder IDE and Jupyter Notebook
If you have not installed Spyder IDE and Jupyter Notebook along with Anaconda distribution, install these before you proceed
Now, set the following environment variables. Now open Spyder IDE and create a new file with the below simple PySpark program and run it
You should see five in output. Now let's start the Jupyter Notebook
In this section of the PySpark tutorial, I will introduce the RDD and explain how to create them
and use its transformation and action operations as examples. Here is the full article on PySpark RDD in case if you wanted to learn more of and get your fundamentals strong
PySpark RDD, Resilient Distributed Dataset, is a fundamental data structure of PySpark
that is fault tolerant in mutable distributed collections of objects, which means once you create an RDD, you cannot change it
Each dataset in RDD is divided into logical partitions, which can be computed on different nodes of the cluster
In order to create an RDD first, you need to create a Spark Session, which is an entry point to the PySpark application
Spark Sessions can be created using a builder or new Session method of the Spark Session
Spark Session internally creates a Spark Context variable of Spark Context. You can create multiple Spark Session objects, but only one Spark Context per JVM
In case if you want to create another new Spark Context, you should stop existing Spark Context using Stop before creating a new one
Spark Context has several functions to use with RDDs. For example, its Parallelize method is used to create an RDD from a list
RDD can also be created from a text file using TextFile, function of the Spark Context
Once you have an RDD, you can perform transformation and action operations
Any operation you perform on RDD runs in parallel. On PySpark RDD, you can perform two kinds of operations, RDD transformations
Transformations are lazy operations. When you run a transformation, for example, Update, instead of updating a current RDD, these operations return another RDD
RDD actions, operations that trigger computation and return RDD values to the driver
Transformations on Spark RDD returns another RDD, and transformations are lazy, meaning they don't execute until you call an action on RDD
Some transformations on RDDs are FlatMap, Map, Reduce by Key, Filter, Sort by Key, and return new RDD instead of updating the current
RDD action operation returns the value from an RDD to a driver node
In other words, any RDD function that returns non-RDD key is considered as an action
Some actions on RDDs are Count, Collect First, Max, Reduce, and more
DataFoam definition is very well explained by Databricks, hence I do not want to define it again and confuse you
Below is the definition I took it from Databricks. If you are coming from a Python background, I would assume you already know what Pandas DataFoam is
PySpark DataFoam is mostly similar to Pandas DataFoam with the exception PySpark DataFoams are distributed in the cluster
meaning the data in DataFoams are stored in different machines in a cluster
And any operations in PySpark execute in parallel on all machines, whereas Pandas DataFoam stores and operates on a single machine
If you have no Python background, I would recommend you learn some basics on Python before you proceed in this Spark tutorial
For now, just know that data in PySpark DataFoams are stored in different machines in a cluster
Due to parallel execution on all cores on multiple machines, PySpark runs operations faster than Pandas
In other words, Pandas DataFoams run operations on a single node, whereas PySpark runs on multiple machines
To know more, read up Pandas DataFoam versus PySpark differences with examples
The simplest way to create a DataFoam is from a Python list of data
DataFoam can also be created from an RDD and by reading files from several sources
By using createDataFoam function of the Spark session, you can create a DataFoam
Since DataFoams are structure format which contains names and columns, you can get the schema of the DataFoam using df.printschema, df.showShowFor20 elements from the DataFoam
Like RDD, DataFoam also has operations like transformations and actions. In real-time applications, DataFoams are created from external sources like files from the local system
HDFS, S3 Azure, HBase, MISQL table etc. Below is an example of how to read a CSV file from a local system
DataFoam has a rich set of API which supports reading and writing several file formats
CSV, Text, Avro, Parquet, CSV, XML and many more. PySpark SQL is one of the most used PySpark modules which is used for processing structured columnar data format
Once you have a DataFoam created, you can interact with the data by using SQL syntax
In other words, Spark SQL brings native raw SQL calls on Spark, meaning you can run traditional anti-SQL on Spark DataFoam
In the later section of this PySpark SQL tutorial, you will learn in detail using SQL select where group by, join, union etc
In order to use SQL first, create a temporary table on DataFoam using create a replace temp view function
Once created, this table can be accessed throughout the Spark session using SQL
and it will be dropped along with your Spark context termination. Use SQL, method of the Spark session object to run the query and this method returns a new DataFoam
Similarly, you can run any traditional SQL queries on DataFoams using PySpark SQL
PySpark Streaming is a scalable, high throughput, fault tolerant streaming processing system that supports both batch and streaming workloads
It is used to process real-time data from sources like FileSystem folder, TCP socket, S3, Kafka, Plume, Twitter and Amazon Kinesis to name a few
The processed data can be pushed to databases, Kafka, live dashboards, etc
Use readstream.formatsocket from Spark session object to read data from the socket and provide options host and port where you want to stream data from
Spark reads the data from the socket and represents it in a value column of DataFoam
Do off print streamer output. After processing, you can stream the DataFoam to console
In real-time, you ideally stream it to either Kafka, database etc. Using Spark Streaming, you can read from Kafka topic and write to Kafka topic in text, CSV, Avro and JSON formats
Below PySpark example, write message to another topic in Kafka using write stream
PySpark GraphFoams are introduced in Spark 3.0 version to support GraphFoam DataFoams
In this section, I will cover PySpark examples by using Mellib library
Prior to 3.0, Spark has GraphX library which ideally runs on RDD and uses all DataFoam capabilities
GraphFoams is a package for Apache Spark which provides DataFoam based graph
It provides high-level APIs in Scala, Java and Pulsum. It aims to provide both the functionality of GraphX and extended functionality taking advantage of Spark DataFoams
This extended functionality includes motif finding, DataFoam based serialization and highly expressive graph queries
GraphX works on RDDs whereas GraphFoams works with DataFoams. Every sample example explained here is tested in our development environment and is available at PySpark examples GitHub project for reference