Description
The Apache Kafka Admin training is a comprehensive program that covers all administrative processes of Kafka, from installation to security. This training is designed to help system administrators, DevOps engineers, and platform managers effectively set up, configure, monitor, and optimize Kafka clusters. Participants will specialize in creating high-performance and scalable systems by understanding Kafka’s behavior under heavy workloads.
While the training focuses on understanding how Kafka works behind the scenes, it also equips participants with practical skills such as troubleshooting common issues in real-world applications and securing the system.
Audience
- System Administrators: Those responsible for the daily maintenance and management of Kafka clusters.
- DevOps Engineers: Professionals who want to integrate Kafka clusters into CI/CD processes and ensure operational efficiency.
- Platform Managers: Team members working with big data infrastructure and responsible for Kafka management.
- Beginners: Technical professionals looking to start their journey with Kafka administration.
Outline
Day 1: Apache Kafka Basics and Installation
Introduction to Apache Kafka
- Kafka’s Role and Importance
- Kafka’s Architectural Components (Broker, Producer, Consumer, Zookeeper)
- Messaging Models: Pub/Sub and Queue Models
The Role and Installation of Zookeeper
- Zookeeper’s Responsibilities
- Single Node and Multi Node Zookeeper Installation
- Zookeeper Configuration and Management
Apache Kafka Installation
- Single Node Kafka Installation
- Multi Node (Cluster) Kafka Installation
- Broker Configuration: Log Segment, Retention Policy, Number of Replicas
Creating and Configuring Kafka Cluster
- Broker ID and Broker Configurations
- Cluster Balancing and Partition Distribution
- Replica Management and ISR (In-Sync Replica) Concept
Day 2: Management and Monitoring
Kafka Broker Management
- Broker Health Check
- Broker Restart and Rebalancing Operations
- Adding and Removing Brokers
Topic Management
- Creating, Listing, and Deleting Topics
- Topic Configuration Parameters
- Defining Retention Policy and Cleanup Policy
- Changing the Number of Partitions
Offset and Consumer Group Management
- Offset Management and Monitoring Consumer Lag
- Resetting Offsets
- Consumer Group Balancing and Coordination
Apache Kafka Monitoring
- Metric Collection and Monitoring Tools (JMX, Prometheus, Grafana)
- Monitoring Broker, Zookeeper, and Consumer Metrics
- Log Monitoring and Analysis
- Kafka Health Check Scripts
Day 3: Advanced Management and Optimization
Security
- Encryption with SSL/TLS
- Authentication with SASL
- Authorization with ACL (Access Control List)
Efficiency and Performance Optimization
- Producer and Consumer Performance Tuning
- Optimizing Partition Distribution
- Tuning Replica Count and ISR Settings
- Batch Size and Linger.ms Tuning
Disaster Recovery and Backup
- Cluster Backup Strategies
- Topic and Offset Backup
- Data Recovery Scenarios
Management with Kafka Tools
- Using Kafka Manager
- Managing with Confluent Control Center
- Management Operations with Kafka CLI Commands
Advanced Configuration
- Reviewing Broker Configurations
- Reviewing Zookeeper Configurations
- JVM Settings and Garbage Collection Optimization
Prerequisites
Basic Java programming knowledge is required.