Apache Kafka Admin

Learn via : Virtual Classroom / Online
Duration : 3 Days
Download PDF
  1. Home
  3. Apache Kafka Admin

Description

The Apache Kafka Admin training is a comprehensive program that covers all administrative processes of Kafka, from installation to security. This training is designed to help system administrators, DevOps engineers, and platform managers effectively set up, configure, monitor, and optimize Kafka clusters. Participants will specialize in creating high-performance and scalable systems by understanding Kafka’s behavior under heavy workloads.

While the training focuses on understanding how Kafka works behind the scenes, it also equips participants with practical skills such as troubleshooting common issues in real-world applications and securing the system.

Audience

  • System Administrators: Those responsible for the daily maintenance and management of Kafka clusters.
  • DevOps Engineers: Professionals who want to integrate Kafka clusters into CI/CD processes and ensure operational efficiency.
  • Platform Managers: Team members working with big data infrastructure and responsible for Kafka management.
  • Beginners: Technical professionals looking to start their journey with Kafka administration.

Outline

Day 1: Apache Kafka Basics and Installation

Introduction to Apache Kafka

  • Kafka’s Role and Importance
  • Kafka’s Architectural Components (Broker, Producer, Consumer, Zookeeper)
  • Messaging Models: Pub/Sub and Queue Models

The Role and Installation of Zookeeper

  • Zookeeper’s Responsibilities
  • Single Node and Multi Node Zookeeper Installation
  • Zookeeper Configuration and Management

Apache Kafka Installation

  • Single Node Kafka Installation
  • Multi Node (Cluster) Kafka Installation
  • Broker Configuration: Log Segment, Retention Policy, Number of Replicas

Creating and Configuring Kafka Cluster

  • Broker ID and Broker Configurations
  • Cluster Balancing and Partition Distribution
  • Replica Management and ISR (In-Sync Replica) Concept

Day 2: Management and Monitoring

Kafka Broker Management

  • Broker Health Check
  • Broker Restart and Rebalancing Operations
  • Adding and Removing Brokers

Topic Management

  • Creating, Listing, and Deleting Topics
  • Topic Configuration Parameters
  • Defining Retention Policy and Cleanup Policy
  • Changing the Number of Partitions

Offset and Consumer Group Management

  • Offset Management and Monitoring Consumer Lag
  • Resetting Offsets
  • Consumer Group Balancing and Coordination

Apache Kafka Monitoring

  • Metric Collection and Monitoring Tools (JMX, Prometheus, Grafana)
  • Monitoring Broker, Zookeeper, and Consumer Metrics
  • Log Monitoring and Analysis
  • Kafka Health Check Scripts

Day 3: Advanced Management and Optimization

Security

  • Encryption with SSL/TLS
  • Authentication with SASL
  • Authorization with ACL (Access Control List)

Efficiency and Performance Optimization

  • Producer and Consumer Performance Tuning
  • Optimizing Partition Distribution
  • Tuning Replica Count and ISR Settings
  • Batch Size and Linger.ms Tuning

Disaster Recovery and Backup

  • Cluster Backup Strategies
  • Topic and Offset Backup
  • Data Recovery Scenarios

Management with Kafka Tools

  • Using Kafka Manager
  • Managing with Confluent Control Center
  • Management Operations with Kafka CLI Commands

Advanced Configuration

  • Reviewing Broker Configurations
  • Reviewing Zookeeper Configurations
  • JVM Settings and Garbage Collection Optimization

Prerequisites

Basic Java programming knowledge is required.