8295  Reviews star_rate star_rate star_rate star_rate star_half

Kafka for Application Developers

In modern applications, real-time information is continuously generated by applications (publishers/producers) and routed to other applications (subscribers/consumers). Apache Kafka is an open...

Read More
By Request
$1,460 USD
Duration 2 days
Course Code WA2708
Available Formats Classroom

Overview

In modern applications, real-time information is continuously generated by applications (publishers/producers) and routed to other applications (subscribers/consumers). Apache Kafka is an open source, distributed publish-subscribe messaging system. Kafka has high-throughput and is built to scale-out in a distributed model on multiple servers. Kafka persists messages on disk and can be used for batched consumption as well as real-time applications.

  • Introduction to Kafka
  • Using Apache Kafka
  • Building Data Pipelines
  • Integrating Kafka with Other Systems
  • Kafka and Schema Management
  • Kafka Streams and KSQL
  • KSQL UDF and Deployment

Skills Gained

  • Understand the use of Kafka for high performance messaging
  • Identify the usages for Kafka in Microservices
  • Explain the benefits of Kafka patterns
  • Differentiate between messaging and message brokers
  • Describe Kafka messaging environments
  • Develop producers and consumers for Kafka
  • Recognize how Kafka enables Cloud-native applications
  • Summarize characteristics and architecture for Kafka
  • Demonstrate how to process messages with Kafka
  • Design distributed high throughput systems based on Kafka
  • Describe the built-in partitioning, replication and inherent fault-tolerance of Kafka

Who Can Benefit

This is a general introduction course for developers, architects, system integrators, security administrators, network administrators, software engineers, technical support individuals, technology leaders & managers, and consultants who are responsible for elements of messaging for data collection, transformation, and integration for your organization supporting Application Modernization, Cloud-Native Development, and Digital Data Supply Chain (Big Data/IoT/AI/Machine Learning/Advanced Analytics/Business Intelligence).

Prerequisites

Basic understanding of messaging, cloud, development, architecture and virtualization would be beneficial.

Course Details

Outline

Chapter 1 - Introduction to Kafka

  • Messaging Architectures – What is Messaging?
  • Messaging Architectures – Steps to Messaging
  • Messaging Architectures – Messaging Models
  • What is Kafka?
  • What is Kafka? (Contd.)
  • Kafka Overview
  • Kafka Overview (Contd.)
  • Need for Kafka
  • When to Use Kafka?
  • Kafka Architecture
  • Core concepts in Kafka
  • Kafka Topic
  • Kafka Partitions
  • Kafka Producer
  • Kafka Consumer
  • Kafka Broker
  • Kafka Cluster
  • Why Kafka Cluster?
  • Sample Multi-Broker Cluster
  • Overview of ZooKeeper
  • Kafka Cluster & ZooKeeper
  • Who Uses Kafka?
  • Summary

Chapter 2 - Using Apache Kafka

  • Installing Apache Kafka
  • Configuration Files
  • Starting Kafka
  • Using Kafka Command Line Client Tools
  • Setting up a Multi-Broker Cluster
  • Using Multi-Broker Cluster
  • Kafka Cluster Planning
  • Kafka Cluster Planning – Producer/Consumer Throughput
  • Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
  • Kafka Cluster Planning – Sizing for Topics and Partitions
  • Kafka Cluster Planning – Sizing for Storage
  • Kafka Connect
  • Kafka Connect – Configuration Files
  • Using Kafka Connect to Import/Export Data
  • Creating a Spring Boot Producer
  • Adding Kafka dependency to pom.xml
  • Defining a Spring Boot Service to Send Message(s)
  • Defining a Spring Boot Controller
  • Testing the Spring Boot Producer
  • Creating a Nodejs Consumer
  • Summary

Chapter 3 - Building Data Pipelines

  • Building Data Pipelines
  • What to Consider When Building Data Pipelines
  • Timeliness
  • Reliability
  • High and Varying Throughput
  • High and Varying Throughput (Contd.)
  • Data Formats
  • Data Formats (Contd.)
  • Transformations
  • Transformations - ELT
  • Security
  • Failure Handling
  • Agility and Coupling
  • Ad-hoc Pipelines
  • Metadata Loss
  • Extreme Processing
  • Kafka Connect vs. Producer and Consumer
  • Kafka Connect vs. Producer and Consumer (Contd.)
  • Summary

Chapter 4 - Integrating Kafka with Other Systems

  • Introduction to Kafka Integration
  • Kafka Connect
  • Kafka Connect (Contd.)
  • Running Kafka Connect Operating Modes
  • Key Configurations for Connect workers:
  • Kafka Connect API
  • Kafka Connect Example – File Source
  • Kafka Connect Example – File Sink
  • Kafka Connector Example – MySQL to Elasticsearch
  • Kafka Connector Example – MySQL to Elasticsearch (Contd.)
  • Write the data to Elasticsearch
  • Building Custom Connectors
  • Kafka Connect – Connectors
  • Kafka Connect - Tasks
  • Kafka Connect - Workers
  • Kafka Connect - Offset management
  • Alternatives to Kafka Connect
  • Introduction to Storm
  • Integrating Storm with Kafka
  • Integrating Storm with Kafka – Sample Code
  • Integrating Storm with Kafka
  • Integrating Hadoop with Kafka
  • Hadoop Consumers
  • Hadoop Consumers (Contd.)
  • Hadoop Consumers – Produce Topic
  • Hadoop Consumers – Fetch Generated Topic
  • Kafka at Uber
  • Kafka at Uber (Contd.)
  • Kafka at LinkedIn
  • Kafka at LinkedIn – Core Kafka Services
  • Kafka at LinkedIn – Core Kafka Services (Contd.)
  • Kafka at LinkedIn – Libraries
  • Kafka at LinkedIn – Monitoring and Stream Processing
  • Summary

Chapter 5 - Kafka and Schema Management

  • Evolving Schema
  • Protobuf (Protocol Buffers) Overview
  • Avro Overview
  • Managing Data Evolution Using Schemas
  • Confluent Platform
  • Confluent Schema Registry
  • Schema Change and Backward Compatibility
  • Collaborating over Schema Change
  • Handling Unreadable Messages
  • Deleting Data
  • Segregating Public and Private Topics
  • Summary

Chapter 6 - Kafka Streams and KSQL

  • What Kafka can be used for?
  • What Kafka can be used for? (Contd.)
  • What Exactly is Kafka?
  • The APIs for Stream Processing
  • Kafka: A Streaming Platform
  • What is KSQL?
  • What is KSQL? (Contd.)
  • Starting KSQL
  • Using the KSQL CLI
  • KSQL Data Types
  • Review the Structure of an Existing STREAM
  • Query the STREAM
  • KSQL Functions
  • Writing to a Topic
  • KSQL Table vs. Stream
  • KSQL JOIN
  • Windows in KSQL Queries
  • Miscellaneous KSQL Commands
  • Summary

Chapter 7 - KSQL UDF and Deployment

  • KSQL Custom Functions
  • KSQL UDF/UDAF
  • Implement a Custom Function
  • Creating UDF and UDAF
  • Creating UDF and UDAF (Contd.)
  • UDFs and Null Handling
  • UDFs and Null Handling (Contd.)
  • Sample UDF Class
  • Build Engine
  • UDAF
  • UDAF Sample Class
  • Supported Types
  • Deploying Custom Functions
  • Using Custom Functions
  • Summary

Lab Exercises

  • Lab 1. Kafka Basics
  • Lab 2. Kafka Multiple Brokers and Import/Export Messages
  • Lab 3. Apache Kafka with Java
  • Lab 4. Apache Kafka with Node.js
  • Lab 5. Kafka Integration With Spark
  • Lab 6. KSQL Basics
  • Lab 7. KSQL Create and Deploy UDF
Read Less

Schedule

Upcoming Course Dates

There are currently no scheduled dates for this course. If you are interested in this course, contact us with the form below.

Contact Information
Address

FAQ

Does the course schedule include a Lunchbreak?

Classes typically include a 1-hour lunch break around midday. However, the exact break times and duration can vary depending on the specific class. Your instructor will provide detailed information at the start of the course.

What languages are used to deliver training?

Most courses are conducted in English, unless otherwise specified. Some courses will have the word "FRENCH" marked in red beside the scheduled date(s) indicating the language of instruction.

What does GTR stand for?

GTR stands for Guaranteed to Run; if you see a course with this status, it means this event is confirmed to run. View our GTR page to see our full list of Guaranteed to Run courses.

Does Ascendient Learning deliver group training?

Yes, we provide training for groups, individuals and private on sites. View our group training page for more information.

What does vendor-authorized training mean?

As a vendor-authorized training partner, we offer a curriculum that our partners have vetted. We use the same course materials and facilitate the same labs as our vendor-delivered training. These courses are considered the gold standard and, as such, are priced accordingly.

Is the training too basic, or will you go deep into technology?

It depends on your requirements, your role in your company, and your depth of knowledge. The good news about many of our learning paths, you can start from the fundamentals to highly specialized training.

How up-to-date are your courses and support materials?

We continuously work with our vendors to evaluate and refresh course material to reflect the latest training courses and best practices.

Are your instructors seasoned trainers who have deep knowledge of the training topic?

Ascendient Learning instructors have an average of 27 years of practical IT experience and have also served as consultants for an average of 15 years. To stay current, instructors spend at least 25 percent of their time learning new, emerging technologies and courses.

Do you provide hands-on training and exercises in an actual lab environment?

Lab access is dependent on the vendor and the type of training you sign up for. However, many of our top vendors will provide lab access to students to test and practice. The course description will specify lab access.

Will you customize the training for our company’s specific needs and goals?

We will work with you to identify training needs and areas of growth.  We offer a variety of training methods, such as private group training, on-site of your choice, and virtually. We provide courses and certifications that are aligned with your business goals.

How do I get started with certification?

Getting started on a certification pathway depends on your goals and the vendor you choose to get certified in. Many vendors offer entry-level IT certification to advanced IT certification that can boost your career. To get access to certification vouchers and discounts, please contact info@ascendientlearning.com.

Will I get access to content after I complete a course?

You will get access to the PDF of course books and guides, but access to the recording and slides will depend on the vendor and type of training you receive.

How do I request a W9 for Ascendient Learning?

View our filing status and how to request a W9.

Reviews

The class and material is good. I think some of the software needs to be updated.

The instructor was thorough and they also provided hands-on demonstrations with labs.

Quick to sign-up to course, and was able to garner some information from the course.

Good training material and good instruction. More time needs to be provided for the lab work.

this class was informative, made me think about certifying for the suse manager cert.