8351  Reviews star_rate star_rate star_rate star_rate star_half

Data Engineering with Python

This intensive hands-on Data Engineering training course teaches the students how to apply Python to the practical aspects of data engineering and introduces the students to the popular Python...

Read More
By Request
$2,090 USD
Duration 3 days
Course Code WA2905
Available Formats Classroom

Overview

This intensive hands-on Data Engineering training course teaches the students how to apply Python to the practical aspects of data engineering and introduces the students to the popular Python libraries used in the field, including NumPy, pandas, Matplotlib, scikit-learn, and Apache Spark.

Skills Gained

  • Data engineering practice
  • High-octane introduction to Python
  • Technical reviews of NumPy, pandas, and other Python libraries and data processing systems
  • Data visualization and exploratory data analysis
  • Data repairing and normalization
  • Understanding the data needs and requirements of Machine Learning and Data Science projects
  • Python in the Cloud
  • Python on Hadoop and Spark (PySpark)

Who Can Benefit

Developers, Software Engineers, Data Scientists, and IT Architects.

Prerequisites

Participants must have practical experience coding in one or more modern programming languages. Knowledge of Python is desirable but not necessary. Students are expected to be able to learn the new material quickly, reinforce their knowledge by doing programming exercises (labs) and then apply their knowledge in data engineering mini-projects.

Course Details

Outline

Chapter 1. Defining Data Engineering

  • Data is King
  • Translating Data into Operational and Business Insights
  • What is Data Engineering
  • The Data-Related Roles
  • The Data Science Skill Sets
  • The Data Engineer Role
  • Core Skills and Competencies
  • An Example of a Data Product
  • What is Data Wrangling (Munging)?
  • The Data Exchange Interoperability Options
  • Summary

Chapter 2. Distributed Computing Concepts for Data Engineers

  • The Traditional Client–Server Processing Pattern
  • Enter Distributed Computing
  • Data Physics
  • Data Locality (Distributed Computing Economics)
  • The CAP Theorem
  • Mechanisms to Guarantee a Single CAP Property
  • Eventual Consistency
  • Summary

Chapter 3. Data Processing Phases

  • Typical Data Processing Pipeline
  • Data Discovery Phase
  • Data Harvesting Phase
  • Data Priming Phase
  • Exploratory Data Analysis
  • Model Planning Phase
  • Model Building Phase
  • Communicating the Results
  • Production Roll-out
  • Data Logistics and Data Governance
  • Data Processing Workflow Engines
  • Apache Airflow
  • Data Lineage and Provenance
  • Apache NiFi
  • Summary

Chapter 4. Quick Introduction to Python for Data Engineers

  • What is Python?
  • Additional Documentation
  • Which version of Python am I running?
  • Python Dev Tools and REPLs
  • IPython
  • Jupyter
  • Jupyter Operation Modes
  • Jupyter Common Commands
  • Anaconda
  • Python Variables and Basic Syntax
  • Variable Scopes
  • PEP8
  • The Python Programs
  • Getting Help
  • Variable Types
  • Assigning Multiple Values to Multiple Variables
  • Null (None)
  • Strings
  • Finding Index of a Substring
  • String Splitting
  • Triple-Delimited String Literals
  • Raw String Literals
  • String Formatting and Interpolation
  • Boolean
  • Boolean Operators
  • Numbers
  • Looking Up the Runtime Type of a Variable
  • Divisions
  • Assignment-with-Operation
  • Dates and Times
  • Comments:
  • Relational Operators
  • The if-elif-else Triad
  • An if-elif-else Example
  • Conditional Expressions (a.k.a. Ternary Operator)
  • The While-Break-Continue Triad
  • The for Loop
  • try-except-finally
  • Lists
  • Main List Methods
  • Dictionaries
  • Working with Dictionaries
  • Sets
  • Common Set Operations
  • Set Operations Examples
  • Finding Unique Elements in a List
  • Enumerate
  • Tuples
  • Unpacking Tuples
  • Functions
  • Dealing with Arbitrary Number of Parameters
  • Keyword Function Parameters
  • The range Object
  • Random Numbers
  • Python Modules
  • Importing Modules
  • Installing Modules
  • Listing Methods in a Module
  • Creating Your Own Modules
  • Creating a Runnable Application
  • List Comprehension
  • Zipping Lists
  • Working with Files
  • Reading and Writing Files
  • Reading Command-Line Parameters
  • Accessing Environment Variables
  • What is Functional Programming (FP)?
  • Terminology: Higher-Order Functions
  • Lambda Functions in Python
  • Example: Lambdas in the Sorted Function
  • Other Examples of Using Lambdas
  • Regular Expressions
  • Using Regular Expressions Examples
  • Python Data Science-Centric Libraries
  • Summary

Chapter 5. Practical Introduction to NumPy

  • SciPy
  • NumPy
  • The First Take on NumPy Arrays
  • Getting Help
  • Understanding Axes
  • Indexing Elements in a NumPy Array
  • NumPy Arrays
  • Understanding Types
  • Re-Shaping
  • Commonly Used Array Metrics
  • Commonly Used Aggregate Functions
  • Sorting Arrays
  • Vectorization
  • Broadcasting
  • Filtering
  • Array Arithmetic Operations
  • Array Slicing
  • 2-D Array Slicing
  • The Linear Algebra Functions
  • Summary

Chapter 6. Practical Introduction to Pandas

  • What is pandas?
  • The Series Object
  • Accessing Values and Indexes in Series
  • Setting Up Your Own Index
  • Using the Series Index as a Lookup Key
  • Can I Pack a Python Dictionary into a Series?
  • The DataFrame Object
  • The DataFrame's Value Proposition
  • Creating a pandas DataFrame
  • Getting DataFrame Metrics
  • Accessing DataFrame Columns
  • Accessing DataFrame Rows
  • Accessing DataFrame Cells
  • Using iloc
  • Using loc
  • Examples of Using loc
  • DataFrames are Mutable via Object Reference!
  • Deleting Rows and Columns
  • Adding a New Column to a DataFrame
  • Appending / Concatenating DataFrame and Series Objects
  • Example of Appending / Concatenating DataFrames
  • Re-indexing Series and DataFrames
  • Getting Descriptive Statistics of DataFrame Columns
  • Getting Descriptive Statistics of DataFrames
  • Applying a Function
  • Sorting DataFrames
  • Reading From CSV Files
  • Writing to the System Clipboard
  • Writing to a CSV File
  • Fine-Tuning the Column Data Types
  • Changing the Type of a Column
  • What May Go Wrong with Type Conversion
  • Summary

Chapter 7. Descriptive Statistics Computing Features in Python

  • Descriptive Statistics
  • Non-uniformity of a Probability Distribution
  • Using NumPy for Calculating Descriptive Statistics Measures
  • Finding Min and Max in NumPy
  • Using pandas for Calculating Descriptive Statistics Measures
  • Correlation
  • Regression and Correlation
  • Covariance
  • Getting Pairwise Correlation and Covariance Measures
  • Finding Min and Max in pandas DataFrame
  • Summary

Chapter 8. Data Grouping and Aggregation with pandas

  • Data Aggregation and Grouping
  • Sample Data Set
  • The pandas.core.groupby.SeriesGroupBy Object
  • Grouping by Two or More Columns
  • Emulating SQL's WHERE Clause
  • The Pivot Tables
  • Cross-Tabulation
  • Summary

Chapter 9. Repairing and Normalizing Data

  • Repairing and Normalizing Data
  • Dealing with the Missing Data
  • Sample Data Set
  • Getting Info on Null Data
  • Dropping a Column
  • Interpolating Missing Data in pandas
  • Replacing the Missing Values with the Mean Value
  • Scaling (Normalizing) the Data
  • Data Preprocessing with scikit-learn
  • Scaling with the scale() Function
  • The MinMaxScaler Object
  • Summary

Chapter 10. Data Visualization in Python using matplotlib

  • Data Visualization
  • What is matplotlib?
  • Getting Started with matplotlib
  • The matplotlib.pyplot.plot() Function
  • The matplotlib.pyplot.scatter() Function
  • Labels and Titles
  • Styles
  • The matplotlib.pyplot.bar() Function
  • The matplotlib.pyplot.hist () Function
  • The matplotlib.pyplot.pie () Function
  • The Figure Object
  • The matplotlib.pyplot.subplot() Function
  • Selecting a Grid Cell
  • Saving Figures to a File
  • Summary

Chapter 11. Parallel Data Processing with PySpark

  • What is Apache Spark
  • The Spark Platform
  • Languages Supported by Spark
  • Running Spark on a Cluster
  • The Spark Shell
  • The High-Level Execution Flow in Stand-alone Spark Cluster
  • The Spark Application Architecture
  • The Resilient Distributed Dataset (RDD)
  • The Lineage Concept
  • Datasets and DataFrames
  • Data Partitioning
  • Data Partitioning Diagram
  • Finding the Most Frequently Used Words in PySpark
  • Summary

Chapter 12. Python as a Cloud Scripting Language

  • Python's Value
  • Python on AWS
  • AWS SDK For Python (boto3)
  • What is Serverless Computing?
  • How Functions Work
  • The AWS Lambda Event Handler
  • What is AWS Glue?
  • PySpark on Glue - Sample Script
  • Summary

Lab Exercises

  • Lab 1. A/B Testing Data Engineering Tasks Project
  • Lab 2. Data Availability and Consistency
  • Lab 3. Using Jupyter Notebook
  • Lab 4. Understanding Python
  • Lab 5. Data Engineering Project
  • Lab 6. Understanding NumPy
  • Lab 7. A NumPy Project
  • Lab 8. Understanding pandas
  • Lab 9. Working with Files in pandas
  • Lab 10. Data Grouping and Aggregation
  • Lab 11. Repairing and Normalizing Data
  • Lab 12. Data Visualization in Jupyter Notebooks using matplotlib
  • Lab 13. Exploratory Data Analysis (EDA)
  • Lab 14. Correlating Cause and Effect
  • Lab 15. Using the Parquet Data Format
Read Less

Schedule

Upcoming Course Dates

There are currently no scheduled dates for this course. If you are interested in this course, contact us with the form below.

Contact Information

FAQ

Does the course schedule include a Lunchbreak?

Classes typically include a 1-hour lunch break around midday. However, the exact break times and duration can vary depending on the specific class. Your instructor will provide detailed information at the start of the course.

What languages are used to deliver training?

Most courses are conducted in English, unless otherwise specified. Some courses will have the word "FRENCH" marked in red beside the scheduled date(s) indicating the language of instruction.

What does GTR stand for?

GTR stands for Guaranteed to Run; if you see a course with this status, it means this event is confirmed to run. View our GTR page to see our full list of Guaranteed to Run courses.

Does Ascendient Learning deliver group training?

Yes, we provide training for groups, individuals and private on sites. View our group training page for more information.

What does vendor-authorized training mean?

As a vendor-authorized training partner, we offer a curriculum that our partners have vetted. We use the same course materials and facilitate the same labs as our vendor-delivered training. These courses are considered the gold standard and, as such, are priced accordingly.

Is the training too basic, or will you go deep into technology?

It depends on your requirements, your role in your company, and your depth of knowledge. The good news about many of our learning paths, you can start from the fundamentals to highly specialized training.

How up-to-date are your courses and support materials?

We continuously work with our vendors to evaluate and refresh course material to reflect the latest training courses and best practices.

Are your instructors seasoned trainers who have deep knowledge of the training topic?

Ascendient Learning instructors have an average of 27 years of practical IT experience and have also served as consultants for an average of 15 years. To stay current, instructors spend at least 25 percent of their time learning new, emerging technologies and courses.

Do you provide hands-on training and exercises in an actual lab environment?

Lab access is dependent on the vendor and the type of training you sign up for. However, many of our top vendors will provide lab access to students to test and practice. The course description will specify lab access.

Will you customize the training for our company’s specific needs and goals?

We will work with you to identify training needs and areas of growth.  We offer a variety of training methods, such as private group training, on-site of your choice, and virtually. We provide courses and certifications that are aligned with your business goals.

How do I get started with certification?

Getting started on a certification pathway depends on your goals and the vendor you choose to get certified in. Many vendors offer entry-level IT certification to advanced IT certification that can boost your career. To get access to certification vouchers and discounts, please contact info@ascendientlearning.com.

Will I get access to content after I complete a course?

You will get access to the PDF of course books and guides, but access to the recording and slides will depend on the vendor and type of training you receive.

How do I request a W9 for Ascendient Learning?

View our filing status and how to request a W9.

Reviews

Some Labs are very good but some steps it ask to update but its already updated, but overall its very good training.

Overall ExitCertified is a great training provider and the remote learning is as effective as in person.

Good training. A lot to take in for the short amount of time we have though

Provided good amount of material and a great instructor to teach the material.

Both course material and instructor demonstrated a sound foundation on Maximo material