Apache Kafka Series – Learn Apache Kafka for Beginners v2

Apache Kafka Series – Learn Apache Kafka for Beginners v2

English | MP4 | AVC 1280×720 | AAC 48KHz 2ch | 123 lectures (7h 32m) | 2.80 GB

START HERE: Learn Apache Kafka 2.0 Ecosystem, Core Concepts, Real World Java Producers/Consumers & Big Data Architecture

Apache Kafka has become the leading distributed data streaming enterprise big data technology. Kafka is used in production by over 33% of the Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart and LinkedIn.

To learn Kafka easily, step-by-step, you have come to the right place! No prior Kafka knowledge is required

If you look at the documentation, you can see that Apache Kafka is not easy to learn…

Thanks to my several years of experience in Kafka and Big Data, I wanted to make learning Kafka accessible to everyone.

We’ll take a step-by-step approach to learn all the fundamentals of Apache Kafka.
At the end of this course, you’ll be productive and you’ll know the following:

  • The Apache Kafka Ecosystem Architecture
  • The Kafka Core Concepts: Topics, Partitions, Brokers, Replicas, Producers, Consumers, and more!
  • Launch your own Kafka cluster in no time using native Kafka binaries – Windows / MacOS X / Linux
  • Learn and Practice using the Kafka Command Line Interface (CLI)
  • Code Producer and Consumers using the Java API
  • Real world project using Twitter as a source of data for a producer and ElasticSearch as a sink for our consumer
  • Overview of Advanced APIs (Kafka Connect, Kafka Streams)
  • Real World Case Studies and Big Use cases
  • Overview of Advanced Kafka for Administrators
  • Advanced Topic Configurations
  • Annexes (starting a Kafka cluster locally, using Docker, etc…)

Note: The hands-on section is based on Java, which is the native Kafka programming language. But, good news! Your learning in Java will be completely applicable to other programming languages, such as Python, C#, Node.js or Scala, and Big Data frameworks such as Spark, NiFi or Akka

What you’ll learn

  • Understand Apache Kafka Ecosystem, Architecture, Core Concepts and Operations
  • Master Concepts such as Topics, Partitions, Brokers, Producers, Consumers
  • Start a personal Kafka development environment
  • Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
  • Create your Producers and Consumers in Java to interact with Kafka
  • Program a Real World Twitter Producer & ElasticSearch Consumer
  • Extended APIs Overview (Kafka Connect, Kafka Streams), Case Studies and Big Data Architecture
  • Practice and Understand Log Compaction
Table of Contents

Kafka Introduction
1 Welcome!
2 Apache Kafka in 5 minutes
3 Course Objectives
4 Important Message

Code Download
5 Code Download

====== Kafka Fundamentals ======
6 Kafka Fundamentals

Kafka Theory
7 Kafka Theory Overview
8 Topics, Partitions and Offsets
9 Brokers and Topics
10 Topic Replication
11 Producers and Message Keys
12 Consumers & Consumer Groups
13 Consumer Offsets & Delivery Semantics
14 Kafka Broker Discovery
15 Zookeeper
16 Kafka Guarantees
17 Theory Roundup

Starting Kafka
18 Important Starting Kafka
19 FAQ for Setup Problems
20 Mac OS X – Download and Setup Kafka in PATH
21 Mac OS X – Using brew
22 Mac OS X – Start Zookeeper and Kafka
23 Mac OS X – Summary
24 Linux – Download and Setup Kafka in PATH
25 Linux – Start Zookeeper and Kafka
26 Linux – Summary
27 Windows – Download Kafka and PATH Setup
28 Windows – Start Zookeeper & Kafka
29 Windows – Summary

CLI (Command Line Interface) 101
30 CLI Introduction
32 Kafka Topics CLI
33 Kafka Console Producer CLI
34 Kafka Console Consumer CLI
35 Kafka Consumers in Group
36 Kafka Consumer Groups CLI
37 Resetting Offsets
38 CLI Options that are good to know
39 What about UIs Conduktor
40 KafkaCat as a replacement for Kafka CLI

Kafka Java Programming 101
41 Intro to Kafka Programming
42 Installing Java 8 & IntelliJ Community Edition
43 Creating Kafka Project
44 Java Producer
45 Java Producer Callbacks
46 Java Producer with Keys
47 Java Consumer
48 Java Consumer inside Consumer Group
49 Java Consumer with Threads
50 Java Consumer Seek and Assign
51 Client Bi-Directional Compatibility
52 Configuring Producers and Consumers

===== Kafka Real World Project =====
53 Real World Project Overview
54 Real World Exercise

Kafka Twitter Producer & Advanced Configurations
55 Producer and Advanced Configurations Overview
56 Twitter Setup
57 Producer Part 1 – Writing Twitter Client
58 Producer Part 2 – Writing the Kafka Producer
59 Producer Configurations Introduction
60 acks & min.insync.replicas
61 retries, delivery.timeout.ms & max.in.flight.requests.per.connection
62 Idempotent Producer
63 Producer Part 3 – Safe Producer
64 Producer Compression
65 Producer Batching
66 Producer Part 4 – High Throughput Producer
67 Producer Default Partitions and Key Hashing
68 [Advanced] max.block.ms and buffer.memory
69 Refactoring the Project

Kafka ElasticSearch Consumer & Advanced Configurations
70 Consumer and Advanced Configuration Overview
71 Setting up ElasticSearch in the Cloud
72 ElasticSearch 101
73 Consumer Part 1 – Setup Project
74 Consumer Part 2 – Write the Consumer & Send to ElasticSearch
75 Delivery Semantics for Consumers
76 Consumer Part 3 – Idempotence
77 Consumer Poll Behaviour
78 Consumer Offset Commit Strategies
79 Consumer Part 4 – Manual Commit of Offsets
80 Consumer Part 5 – Performance Improvement using Batching
81 Consumer Offsets Reset Behaviour
82 Consumer Part 6 – Replaying Data
83 Consumer Internal Threads

===== Kafka Ecosystem & Real-World Architectures =====
84 Kafka in the Real World

Kafka Extended APIs for Developers
85 Kafka Connect Introduction
86 Kafka Connect Twitter Hands-On
87 Kafka Streams Introduction
88 Kafka Streams Hands-On
89 Kafka Schema Registry Introduction
90 Which Kafka API should I use

Real World Insights and Case Studies (Big Data Fast Data)
91 Choosing Partition Count & Replication Factor
92 Kafka Topics Naming Convention
93 Case Study – MovieFlix
94 Case Study – GetTaxi
95 Case Study – MySocialMedia
96 Case Study – MyBank
97 Case Study – Big Data Ingestion
98 Case Study – Logging and Metrics Aggregation

Kafka in the Enterprise for Admins
99 Kafka Cluster Setup High Level Architecture Overview
100 Kafka Monitoring & Operations
101 Kafka Security
102 Kafka Multi Cluster & MirrorMaker

===== Advanced Kafka =====
103 Advanced Kafka

Advanced Topics Configurations
104 Changing a Topic Configuration
105 Segment and Indexes
106 Log Cleanup Policies
107 Log Cleanup Delete
108 Log Compaction Theory
109 Log Compaction Practice
110 min.insync.replicas reminder
111 Unclean Leader Election

===== Annexes =====
112 What are annexes

Annex – Starting Kafka Differently
113 Annex 1 – Overview
114 Starting Kafka with the Confluent CLI
115 Starting a multi broker Kafka Cluster using Binaries
116 Note on Docker Setup
117 Start Kafka Development environment using Docker
118 Starting a multi broker Kafka Cluster using Docker
119 Kafka Advertised Host Setting
120 Starting Kafka on a Remote Machine

===== Next Steps =====
121 What’s Next
123 Bonus Lecture Student Special Coupons for my Other courses