Back to blog

Pyspark Tutorial: Setup, Key Concepts, and MLlib Quick Start

Bar Yochai Shaya

Director of Solution Engineering & Technical Sales, Intel Granulate

What is PySpark? 

PySpark is a library that lets you work with Apache Spark in Python. Apache Spark is an open-source distributed general-purpose cluster-computing framework. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

PySpark combines the power of Spark’s distributed computing capabilities with the simplicity and elegance of the Python programming language, making it a great tool for handling big data. Unlike traditional systems, PySpark enables data scientists to perform data processing tasks on large datasets that would otherwise be too big to handle on a single machine.

Get the Big Data Optimization Guide

In addition to its data processing capabilities, PySpark also supports SQL queries, streaming data, and complex analytics such as machine learning and graph algorithms.

In this article:

PySpark Tutorial: Setting up PySpark 

Now that we have a better understanding of what PySpark is, let’s move on to the practical part of this PySpark tutorial – setting up PySpark.

Step 1: System Requirements and Prerequisites

Before we can start installing PySpark, there are a few system requirements and prerequisites. First of all, PySpark requires a 64-bit operating system. It supports various operating systems including Linux, Mac OS, and Windows.

In terms of hardware, PySpark doesn’t require any special hardware. As long as your machine has a decent amount of memory (4 GB or more is recommended), you should be good to go.

As for software prerequisites, you’ll need to have Java installed on your machine, as Spark is written in Scala, which runs on the Java Virtual Machine (JVM). You’ll also need to have Python installed – we’ll cover this in more detail in the next section.

Step 2: Installing Python

As PySpark is a Python library, we’ll need to have Python installed on our machine. If you don’t already have Python installed, you can download it from the official Python website. It’s recommended to install the latest version of Python 3, as Python 2 is no longer supported.

After downloading the Python installer, run it and follow the prompts to install Python. Make sure to check the box that says “Add Python to PATH” during the installation process – this will make it easier to run Python from the command line.

Once Python is installed, you can confirm the installation by opening a command prompt (or terminal on Mac or Linux) and typing python –version. This should display the version of Python that you installed.

Get the Big Data Optimization Guide

Step 3: Installing PySpark

With Python installed, we can now move on to installing PySpark. The easiest way to install PySpark is using pip, Python’s package installer. Open a command prompt or terminal and type: 

pip install pyspark

This will download and install PySpark and its dependencies.

If you encounter any issues during the installation, it’s likely that they’re due to pip not being updated. You can update pip by typing: 

pip install --upgrade pip

After installing PySpark, you can confirm the installation by typing:

pyspark --version

This should display the version of PySpark that you installed.

Step 4: Configuring PySpark Environment

After installing PySpark, the next step in this PySpark tutorial is to configure our PySpark environment. This involves setting a few environment variables that PySpark needs in order to run correctly.

First, we need to set the SPARK_HOME environment variable to the path where Spark is installed. This is typically the spark folder in your home directory.

Next, we need to add the PySpark library to the Python path. This can be done by adding the following line to your .bashrc file (or .bash_profile file on Mac): 


With these environment variables set, you should now be able to run PySpark from the command line by simply typing pyspark. If everything is set up correctly, this should start the PySpark shell.

Data Structures in PySpark 

Now that we have PySpark installed and configured, let’s move on to exploring some of the main data structures in PySpark.

Working with RDDs

In PySpark, one of the fundamental data structures is the Resilient Distributed Dataset (RDD). An RDD is a collection of elements that can be processed in parallel. In other words, RDDs are the building blocks of any PySpark program.

Creating an RDD in PySpark is simple. You can create an RDD from a Python list using the parallelize() function, or from a file using the textFile() function. Once you have an RDD, you can perform actions on it (like counting the number of elements) or transformations (like filtering elements).

One of the key features of RDDs is their ability to recover from node failures. If a node in your cluster fails while processing an RDD, PySpark can automatically recover the lost data and continue processing.

Understanding PySpark DataFrames

In addition to RDDs, PySpark also supports DataFrames. A DataFrame in PySpark is a distributed collection of data organized into named columns. It’s conceptually equivalent to a table in a relational database or a data frame in R or Python, but with optimizations for big data processing.

DataFrames can be created from a variety of sources including structured data files, Hive tables, external databases, or existing RDDs. They provide a domain-specific language for structured data manipulation in Scala, Java, Python, and R.

One of the main advantages of DataFrames over RDDs is their ability to leverage the Spark SQL engine for optimization. This can lead to significantly faster execution times for certain types of queries.

Learn how to work with PySpark DataFrames in the official documentation.

Get the Big Data Optimization Guide

Understanding PySpark Operations 

Before we dive into the specifics, it’s essential to understand PySpark’s fundamental operations. PySpark’s operations are divided into two main categories – transformations and actions, and they form the backbone of any PySpark program.

Transformations and Actions

Transformations are operations in PySpark that produce an RDD (Resilient Distributed Dataset). They are lazily evaluated, which means the execution doesn’t start right away. They are remembered and are only executed when an action is triggered. Examples of transformations include map, filter, and reduceByKey.

Actions, on the other hand, are operations that return a final value to the driver program or write data to an external system. Actions trigger the execution of transformations. Some common actions include count, collect, first, and take.

Pair RDD Operations

Pair RDD is a special type of RDD in PySpark where each element is a pair tuple (key-value pair). Pair RDDs are a useful building block in many programs, as they expose operations that allow you to act on each key in parallel or regroup data across the network. Common pair RDD operations include reduceByKey, groupByKey, sortByKey, and join.

These operations are fundamental to working with data in PySpark and are the first step towards mastering big data processing.

Key-Value Pair Operations

Key-Value pair operations are a subset of pair RDD operations that specifically operate on RDDs where the data is organized as key-value pairs. They provide a convenient way to aggregate data and perform computations on specific data groups.

For example, you can use the reduceByKey operation to sum all values with the same key or use the groupByKey operation to group all values with the same key. Understanding key-value pair operations is crucial for efficient data processing and manipulation with PySpark.

Learn more in our detailed guide to PySpark optimization techniques 

Machine Learning with PySpark MLlib 

Once you’ve grasped the basics of PySpark operations, another exciting area to explore is machine learning with PySpark’s MLlib.

Introduction to MLlib

MLlib stands for Machine Learning Library. As the name suggests, MLlib is PySpark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, and much more.

One of the significant advantages of using MLlib is that it comes built-in with PySpark, which means you can enjoy the power of machine learning on a large scale without having to resort to additional libraries.

Data Preprocessing with MLlib

Before building machine learning models, it’s essential to preprocess the data to make it suitable for the model. With PySpark’s MLlib, you can perform various preprocessing tasks such as tokenizing text, converting categorical data into numerical data, normalizing numerical data, and more.

MLlib’s feature transformers help in transforming one DataFrame into another, typically by appending one or more columns. For example, you can use the Binarizer to threshold numerical features to binary (0/1) features.

Get the Big Data Optimization Guide

Building Machine Learning Models with MLlib

Once the data is preprocessed, you can start building machine learning models. MLlib supports several popular machine learning algorithms for classification, regression, clustering, and collaborative filtering, as well as supporting lower-level optimization primitives.

These algorithms include logistic regression, decision trees, random forests, gradient-boosted trees, k-means, and many others. You can train these models with your preprocessed data and start making predictions.

Model Evaluation and Tuning

After building the models, it’s crucial to evaluate their performance and tune them if necessary. MLlib provides several tools for model evaluation and tuning, such as CrossValidator, TrainValidationSplit, and several metric functions in the evaluation module.

The CrossValidator allows for hyperparameter tuning by fitting the pipeline on different parameter combinations and selecting the best model. The TrainValidationSplit, similarly, also allows for hyperparameter tuning with a simpler interface.

Related content: Read our guide to pyspark tutorial

Optimize application performance.

Save on cloud costs.

Start Now
Back to blog