Description
Big Data Processing with Apache Spark is a technical training program that covers the core components and application development methods of Apache Spark technology, which enables fast and effective processing of large-scale data on distributed systems.
The training examines Apache Spark architecture, distributed data processing principles, PySpark, RDD and DataFrame API, Spark SQL, data reading and writing operations, data transformations, and basic performance optimization topics through hands-on practice.
Participants will gain skills in reading data from different data sources, cleaning, transforming, analyzing data, and developing basic ETL processes using Apache Spark.
To enable participants to learn the Apache Spark architecture and fundamental operating principles, and to develop data processing and analytics applications on large-scale data using PySpark and Spark SQL.
Audience
- Data Engineer
- Big Data Developer
- Data Analyst
- ETL Developer
- Software Developer
- Technical teams working with data processing and big data technologies
Outline
1. Introduction to Big Data & Apache Spark
- Big Data concept and fundamental characteristics
- Overview of big data processing technologies
- Comparison of Apache Hadoop and Apache Spark
- Use cases of Apache Spark
- Apache Spark ecosystem
- Overview of Spark Core, Spark SQL, Structured Streaming, and MLlib components
- Advantages and usage scenarios of Apache Spark
- Open Data Table (Delta Lake, Iceberg)
2. Apache Spark Architecture
- Apache Spark architecture
- Driver and Executor concepts
- Cluster Manager
- Spark Application lifecycle
- Jobs, Stages, and Tasks
- Distributed Computing
- Lazy Evaluation
- Transformations and Actions
- SparkSession and SparkContext
3. Apache Spark Development Environment
- Apache Spark installation and configuration
- Python and PySpark development environment
- Creating SparkSession
- Running Spark applications
- Local Mode and Cluster Mode
- Using Spark Shell and PySpark
- Developing the first Apache Spark application
4. Resilient Distributed Datasets (RDD)
- RDD concept
- RDD creation methods
- RDD Transformations
- RDD Actions
- Map, Filter, and Reduce operations
- Key-Value RDD
- RDD Persistence and Caching
- RDD and DataFrame comparison
- Hands-on practice: Data processing with RDD
5. PySpark & DataFrame API
- DataFrame concept and architecture
- Creating DataFrames
- Schema and Data Types
- Selecting and filtering data
- Column Operations
- Data transformation operations
- GroupBy and Aggregations
- Join operations
- Missing Values management
- Duplicate Records management
- Hands-on practice: PySpark DataFrame operations
6. Spark SQL
- Spark SQL architecture
- Relationship between SQL and DataFrame API
- Creating Temporary Views
- Spark SQL queries
- SELECT, WHERE, GROUP BY, and ORDER BY
- JOIN operations
- Aggregate Functions
- Window Functions
- Data analysis with SQL
- Hands-on practice: Querying data with Spark SQL
7. Data Sources & File Formats
- Reading and writing data with Spark
- CSV data format
- JSON data format
- Parquet data format
- Overview of ORC data format
- Schema Inference
- Defining Explicit Schema
- Database connections through JDBC
- Partitioned Data
- Data reading and writing options
- Hands-on practice: Working with different data sources
8. Data Processing & ETL with Spark
- ETL concept and data processing processes
- Batch Data Processing
- Data cleaning
- Data transformation
- Data integration
- Data validation
- Aggregation operations
- Creating an ETL Pipeline
- Hands-on practice: Developing a basic ETL process with PySpark
9. Spark Performance Fundamentals
- Performance concept in Spark applications
- Partitioning
- Repartition and Coalesce
- Caching and Persistence
- Shuffle Operations
- Broadcast Variables
- Basic Join optimization
- Using Spark UI
- Execution Plan review
- Basic performance improvement methods
10. Introduction to Structured Streaming
- Batch and Streaming data processing
- Structured Streaming concept
- Streaming DataFrame
- Streaming Sources and Sinks
- Micro-Batch Processing
- Running Streaming queries
- Checkpointing concept
- Hands-on practice: Simple Structured Streaming application
11. Hands-on Workshop: Apache Spark Data Processing
- Preparing the sample data set
- Reading data with PySpark
- Creating DataFrames
- Data cleaning and transformation
- Data analysis with Spark SQL
- Integrating different data sources
- Developing a basic ETL Pipeline
- Saving results in Parquet format
- Application review through Spark UI
Prerequisites
- Basic Python programming knowledge
- SQL querying knowledge
- Basic knowledge of databases and data processing processes
- Prior knowledge of big data technologies is recommended
