Day 1: Introduction to Apache Kafka and Producer/Consumer API
Introduction to Apache Kafka
- Fundamentals of Kafka
- Kafka’s Working Model (Pub/Sub)
- Kafka Components (Broker, Topic, Partition, Zookeeper)
Kafka Terminology
- Topic, Partition, Offset, Producer, Consumer
- Concepts of Consumer Group and Consumer Lag
First Application with Kafka (Java)
- Sending Messages with Kafka Producer API
- Synchronous and Asynchronous Message Delivery
- Error Handling with Callback
- Reading Messages with Kafka Consumer API
- Parallel Data Processing with Consumer Group
First Application with Kafka (Python)
- KafkaProducer and KafkaConsumer Libraries
- Simple Producer and Consumer Application
- Monitoring Consumer Lag and Managing Offsets
- Error Handling with Python
Day 2: Advanced API Usage and Data Serialization
Advanced Kafka Applications with Java
- Writing a Custom Partitioner
- Using Custom Serializer/Deserializer
- Message Serialization with Avro and Protobuf
- Producer and Consumer Configurations
- Performance Tuning for High Throughput
Advanced Kafka Applications with Python
- Writing Custom Serializer/Deserializer
- Message Serialization (JSON, Avro, Protobuf)
- Offset Management and Manual Commit
- Parallel Processing for Python Consumers
Kafka Streams API (Java)
- Real-Time Data Processing with Kafka Streams API
- Examples of Stream-Table Joins
- Stateless and Stateful Transformations
- Windowing Concepts
Kafka-Python Streams Usage
- Simple Stream Processing Applications
- Examples of Stream Aggregation and Filtering
Day 3: Application Development and Deployment
Real-Time Data Processing Project (Java)
- Reading and Processing Data from Multiple Topics
- Writing Processed Data to a Different Topic
- Using Transactional Producer
Real-Time Data Processing Project (Python)
- Reading Data from Different Sources
- Data Transformation and Writing to a New Topic
- Application Logging and Error Handling
Deploying Kafka Applications with Docker
- Docker Compose for Kafka, Zookeeper, and Applications
- Dockerizing Java and Python Applications
- Testing Scenarios on a Local Cluster
CI/CD for Kafka Applications
- Configuring Pipelines
- Testing Kafka-Based Applications
- Deployment Strategies
Security and Authorization
- Encryption with SSL/TLS
- Authentication with SASL/PLAIN
- Authorization with ACL
