Beyond BI: A Data Scientist's Guide to Microsoft's Modern Data Stack [Webinar]

IT Training and Development Resources
Webinars
Beyond BI: A Data Scientist's Guide to Microsoft's Modern Data Stack [Webinar]

Are you a data professional stuck in the past? In this free webinar, you will go on a practical journey from the legacy Microsoft BI stack to the cutting-edge of modern data engineering.

Remember the days of Business Intelligence Development Studio (BIDS), SQL Server Data Tools (SSDT), SSIS packages, SSAS cubes, SSAS Data Mining, and SSRS reports?

We will explore tools like Power BI, Microsoft Fabric, Azure Databricks, and more. Watch this session to bridge the gap between what you know and the innovative solutions that are shaping the future of data.

If you prefer to read instead of watch, we have a blog article based on this webinar.

Microsoft Modern Data Stack webinar

Recorded January 8, 2026

What You'll Experience

Whether you're a BI professional looking to modernize your skill set or a data engineer/data scientist curious about Microsoft’s expanding ecosystem, this session will bridge the gap between legacy tools and today’s innovations specifically through Microsoft Fabric, Power BI, Azure Machine Learning, Azure Databricks, and Azure Data Factory. Join us to learn practical skills, including:

Understanding how tools like Power BI, Microsoft Fabric, Azure Databricks, and Azure Data Factory are reshaping the data landscape
Migrating from SSIS to cloud-native pipelines and dataflows
Leveraging Jupyter Notebooks and machine learning within Microsoft Fabric
Building smarter dashboards with real-time data and semantic models with KQL
Reimagining SSAS Data Mining through Azure Machine Learning

Audience

This session is a must-attend for BI professionals looking to modernize their skills and data engineers/data scientists curious about Microsoft’s evolving data ecosystem.

About the Presenter

Faheem Javed began his IT career as a Visual C++ programmer in the 1990s and has since cultivated a deep and diverse expertise across the Microsoft technology stack. His journey has spanned early development in .NET, source control and lifecycle management with TFS, and modern deployment practices through DevOps. Faheem has closely followed Microsoft's Business Intelligence evolution—witnessing the transition from third-party tools like Proclarity Analytics to enterprise-grade platforms such as SSIS, SSAS, SSRS, SharePoint PerformancePoint Services, Polybase, Power BI, Azure Synapse, Azure Databricks, and Microsoft Fabric.

He holds a wide array of certifications, including Azure Solutions Architect, Azure Data Engineer, Azure AI Engineer, Microsoft Fabric Data Engineer, Microsoft Fabric Analytics Engineer, and Azure Data Science, reflecting his broad capabilities in cloud architecture, data engineering, machine learning, and analytics. Faheem has delivered both standard and customized training on these technologies and continues to share his expertise as an instructor for Microsoft Fabric courses through Ascendient Learning, helping professionals stay ahead in the evolving landscape of data and AI.

WEBVTT

00:01:03.000 --> 00:01:07.000
a data scientist's guide to Microsoft's modern data stack.

00:01:07.000 --> 00:01:17.000
My name's Alex, and I'm actually going to be the MC for today. Uh, before we begin, I just wanted to go over some housekeeping rules first.

00:01:17.000 --> 00:01:35.000
The chat below is open, like I'd mentioned before, so is the Q&A, so please feel free to utilize it throughout the session, uh, just however, you know, as Fahim is instructing, we might not be able to read all of the current questions, so just be, you know, patient with us, and we'll ensure that we save some time at the end.

00:01:35.000 --> 00:01:39.000
to answering any of your remaining comments or questions.

00:01:39.000 --> 00:01:44.000
Lastly, this session is actually recording, and I'm going to be sending a follow-up email.

00:01:44.000 --> 00:01:49.000
to the attendees. with the recording for today.

00:01:49.000 --> 00:02:05.000
With that said, I just want to shift gears momentarily and reintroduce Ascendient Learning. I know most of you might be familiar with us, but… some changes. So, last year, Ascendient Learning became a part of Accenture under a new business called LearnVantage.

00:02:05.000 --> 00:02:14.000
So Accenture LearnVantage, it's a growing learning service and platform with expanded capabilities through investments such as Ascendient learning.

00:02:14.000 --> 00:02:20.000
So with Ascendient Learning's instructor-led training and certification services.

00:02:20.000 --> 00:02:28.000
We can actually help organizations receive customizable suite of offerings when it comes to programs like upscaling and reskilling.

00:02:28.000 --> 00:02:36.000
And as Ascendia Earning, we still remain very much committed to providing standard and tailored training to our customers, of course.

00:02:36.000 --> 00:02:42.000
with a catalog of over 850 courses on IT training topics and vendors.

00:02:42.000 --> 00:02:52.000
You know, such as AWS, AI, machine learning, SAP, we have a really great partnership with them, cloud, and many, many more.

00:02:52.000 --> 00:03:05.000
If you're interested, you can visit ascendientlearning.com to browse our training catalog, or you can also submit a request for more information on private training. I'll share some links in the chat for you to do so.

00:03:05.000 --> 00:03:12.000
With that, I want to segue into introducing our instructor for today, Fahim Javed, excuse me.

00:03:12.000 --> 00:03:27.000
So, Fahim's IT journey actually started as a visual C++ programmer, and has actually grown into deep expertise across the Microsoft technology stack. Over the years, he's worked on early .NET development, mastered source control and life management with TFS.

00:03:27.000 --> 00:03:45.000
and has embraced modern deployment practices through DevOps. The game also has an impressive range of certifications under his belt. As you can see, we have, uh… He has. Azure Solutions Architect, Azure Data Engineer, Azure AI Engineer, Microsoft Fabric Data Engineer, and…

00:03:45.000 --> 00:03:56.000
many more. Behind his technical skills, he's actually passionate about sharing the knowledge, much like today, delivering both standard and customized training on these technologies. So for today's.

00:03:56.000 --> 00:04:06.000
session, he's actually going to, um, discuss some of these topics, and he's still very much teaches Microsoft Fabric courses through Ascendia Learning.

00:04:06.000 --> 00:04:15.000
Helping professionals stay ahead in the fast-changing world of data and AI. So enough about me, I'm going to pass it over to Fahim.

00:04:15.000 --> 00:04:19.000
Thank you, Alex. I really appreciate the great intro, and greetings, everyone.

00:04:19.000 --> 00:04:25.000
Um, okay, so let me quickly share my screen. Where is my screen?

00:04:25.000 --> 00:04:34.000
Uh, okay, so, uh, you can see, uh, the screen, hopefully. If yes, you can give me a thumbs up, or leave a message in the chat window.

00:04:34.000 --> 00:04:42.000
And, uh, I'll definitely let you ask questions as well. You can post them in the chat window, and then towards the end of the session, I'll answer your questions.

00:04:42.000 --> 00:04:58.000
Okay, awesome, so you can see my screen. Uh, okay, so today's session is about, um… what is beyond business intelligence? Like, uh, if you have been using business intelligence technologies, especially from Microsoft.

00:04:58.000 --> 00:05:09.000
what's new, so the old BI… Era is, uh, dying or almost dead, uh, instead we have lots of new technologies available.

00:05:09.000 --> 00:05:15.000
And, uh, the purpose of today's brief session is to give you some exposure to what's new.

00:05:15.000 --> 00:05:22.000
Uh, so that you're aware of it, and you can, uh… maybe upscale yourself and your organization and whatnot.

00:05:22.000 --> 00:05:33.000
Okay, so today we will talk about, uh. First of all, what exactly BI is, uh, the classic BI, uh, what are the drawbacks of that classic BI?

00:05:33.000 --> 00:05:39.000
And then what's the modern shift to data engineering and data science sort of roles?

00:05:39.000 --> 00:05:46.000
like, back in the day, we used to have BEI developers, but now we have, uh, data engineers, data scientists.

00:05:46.000 --> 00:05:51.000
Obviously, we have AI engineers, etc, as well, and whatnot, but it's not the main scope of the session.

00:05:51.000 --> 00:05:56.000
We will mostly focus on data engineering slash data science in this session.

00:05:56.000 --> 00:06:02.000
And then, uh, technology stack-wise, we will discuss Microsoft Fabric.

00:06:02.000 --> 00:06:09.000
Azure Databricks, Azure ML, or machine learning, and then we will also briefly discuss, uh.

00:06:09.000 --> 00:06:18.000
the role of ML Apps. in the context of data science, end-to-end kind of, uh, lifecycle sort of projects and whatnot.

00:06:18.000 --> 00:06:24.000
Which is somewhat similar to DevOps. Uh, and then we will also touch on data governance as well.

00:06:24.000 --> 00:06:34.000
Okay, so, uh, let's first of all talk about the classic Microsoft BI, uh… sort of error that we used to have.

00:06:34.000 --> 00:06:46.000
Um, so back in the day. like, uh… my personal, uh, forced exposure to this area was back in, uh, the, uh, 1999 slash 2000, uh.

00:06:46.000 --> 00:06:53.000
when Microsoft introduced DTS, Data Transformation Services. Along with all lab cubes.

00:06:53.000 --> 00:07:01.000
And then later on in the year 2005. Microsoft ended up, uh, creating this consolidated stack called.

00:07:01.000 --> 00:07:09.000
BI stack, where we had SQL Server, uh, then there was a tool called Business Intelligence Development Studio.

00:07:09.000 --> 00:07:13.000
which was later renamed to SQL Server Data Tools.

00:07:13.000 --> 00:07:20.000
And it allowed us to create end-to-end business intelligence solutions.

00:07:20.000 --> 00:07:25.000
And overall, business intelligence used to be, uh, going from raw data.

00:07:25.000 --> 00:07:32.000
to be able to make well-informed decisions. So go from data to decisions, basically.

00:07:32.000 --> 00:07:41.000
And for this purpose, we used to have a SQL Server integration services, SQL Server Analysis Services, and SQL Server.

00:07:41.000 --> 00:07:51.000
reporting services. Um… If anyone has used any of these technologies before, please give me a thumbs up, or leave a message in the chat window.

00:07:51.000 --> 00:08:04.000
If yes, that's great, if not, don't worry about it. It's, uh, sort of the old technology stack now. Very few companies, yes, they are still using it, and ideally they should switch over to the modern stack, which we will cover.

00:08:04.000 --> 00:08:13.000
Uh, in this session in just a bit. And if none of this makes sense, what's the purpose of SSIS, SSAS, SSRS?

00:08:13.000 --> 00:08:20.000
Uh, maybe because you have never used this before, uh, you have never heard of this before, you never encountered this before.

00:08:20.000 --> 00:08:25.000
Let me give you a quick bird's-eye view of what it used to look like.

00:08:25.000 --> 00:08:30.000
So, um, our goal used to go from raw data.

00:08:30.000 --> 00:08:38.000
Uh, which could exist in your relational databases, or in the form of files, like CSV, Excel.

00:08:38.000 --> 00:08:43.000
and various other files can be there as well. So go from raw data.

00:08:43.000 --> 00:08:49.000
to, uh… reports, dashboards, so that you can make decisions.

00:08:49.000 --> 00:08:55.000
And, uh, this, uh, is the classic business intelligence.

00:08:55.000 --> 00:09:01.000
end-to-end process. So we used to, uh… utilize this tool.

00:09:01.000 --> 00:09:07.000
Actually, some companies, they are still using it, and it's called SQL Server Integration Services.

00:09:07.000 --> 00:09:12.000
So SSIS used to be a drag-and-drop kind of a tool.

00:09:12.000 --> 00:09:20.000
where you were able to write code in SQL, as well as in .NET and various other languages.

00:09:20.000 --> 00:09:25.000
Uh, but primarily .NET and SQL was used as the coding language.

00:09:25.000 --> 00:09:28.000
And you were able to create your data pipelines.

00:09:28.000 --> 00:09:31.000
So you were able to connect to your raw data.

00:09:31.000 --> 00:09:38.000
clean it up, transform it. uh, like, uh, remove duplicates, uh, do something about missing values.

00:09:38.000 --> 00:09:42.000
fix data types and whatnot. So make sure your data is.

00:09:42.000 --> 00:09:50.000
cleaned up. before you start making decisions based on your messy data, make sure you clean it up first.

00:09:50.000 --> 00:09:54.000
So SSIS used to allow us to create these, uh.

00:09:54.000 --> 00:10:04.000
data pipelines, ATL pipelines, extract, transform, load pipelines. And then, uh, once we, uh, have cleaned up the data.

00:10:04.000 --> 00:10:12.000
After that, we used to, uh, store it. And, uh, sort of a single source of truth.

00:10:12.000 --> 00:10:20.000
A single central repository. where, uh, all of the necessary data is organized, it's stored.

00:10:20.000 --> 00:10:30.000
So this is basically your data warehouse concept. So, data warehouse is like, let's say, some retail store, such as maybe Costco, Walmart.

00:10:30.000 --> 00:10:35.000
So you go to just a single store. And, uh, look at whatever data you want.

00:10:35.000 --> 00:10:44.000
Rather than going to, uh, tens of… lots of dozens of, let's say, files and database servers and whatnot.

00:10:44.000 --> 00:10:46.000
So, Databbed House used to allow us to create a.

00:10:46.000 --> 00:10:52.000
Central, uh, repository, where we used to have slicing and dicing of the data.

00:10:52.000 --> 00:10:58.000
We used to use the term cube, where data is basically, uh, consolidated, and, uh.

00:10:58.000 --> 00:11:06.000
It's aggregated plus all the breakdowns are. evaluated as well, and uh… it's basically a very fast way of accessing data.

00:11:06.000 --> 00:11:12.000
So, Databbed House, uh, we mostly designed these warehouses by using.

00:11:12.000 --> 00:11:20.000
SQL Server analysis services. And then, um… Uh, on top of it.

00:11:20.000 --> 00:11:25.000
If we had any need for. predictive analysis.

00:11:25.000 --> 00:11:31.000
So, predict data, forecast data. future forecasting, uh, projection, basically.

00:11:31.000 --> 00:11:38.000
So even back then. We used to have data mining available as part of SQL Server analysis services.

00:11:38.000 --> 00:11:44.000
So we used to create data mining structures, we had all of the… or most of the common, uh.

00:11:44.000 --> 00:11:56.000
algorithms like, uh, neural network, we have clustering, uh… association, uh, and whatnot. So we used navbase as well, so we used to have several algorithms.

00:11:56.000 --> 00:12:02.000
So we could predict data. Like, it could be, let's say, uh, for, uh, sales kind of scenario.

00:12:02.000 --> 00:12:09.000
So, based on your current sales performance. Predict how many items we will sell.

00:12:09.000 --> 00:12:15.000
Uh, in the coming year, in the specific quarter of the coming year, and whatnot. So we were able to.

00:12:15.000 --> 00:12:19.000
predict, forecast. data when using data mining.

00:12:19.000 --> 00:12:29.000
So even back then, before data science, the modern data science came out, we still had data mining available for future forecasting, prediction, and whatnot.

00:12:29.000 --> 00:12:34.000
So we used to utilize, uh, SQL Server analysis services and create our data mining structures.

00:12:34.000 --> 00:12:39.000
based on different algorithms, so we could predict our data and so on.

00:12:39.000 --> 00:12:47.000
And, uh, once we, uh, had these warehouses and mining structures in place, they were hosted on some server.

00:12:47.000 --> 00:12:53.000
And then after that, we were able to use different visualization technologies, such as.

00:12:53.000 --> 00:12:58.000
SQL Server Reporting Services. It used to allow us to create reports.

00:12:58.000 --> 00:13:08.000
uh… Excel. Which has been available since forever, so even Excel can connect to these warehouses, as well as mining structures.

00:13:08.000 --> 00:13:13.000
Um, and, uh, we also had SharePoint performance point services.

00:13:13.000 --> 00:13:18.000
to create dashboards. and our SharePoint portal.

00:13:18.000 --> 00:13:21.000
And this actually was the foundation of Power BI.

00:13:21.000 --> 00:13:33.000
So the modern technology called Power BI. Back in the day, it used to be a bunch of separate technologies, and one of them used to be SharePoint performance point services dashboards.

00:13:33.000 --> 00:13:36.000
So we were able to connect to these warehouses and, uh.

00:13:36.000 --> 00:13:42.000
predictive data mining structures. to create our reports, forecast data, and whatnot.

00:13:42.000 --> 00:13:47.000
So this is the classic. Microsoft Business Intelligence.

00:13:47.000 --> 00:13:55.000
BI stack. And, uh, if you're still using it, no, ideally you should start switching over to.

00:13:55.000 --> 00:14:01.000
the modern technologies, which we will discuss here in just a bit.

00:14:01.000 --> 00:14:09.000
Okay, so, um, like I briefly discussed that, uh, we used to have ETL data integration.

00:14:09.000 --> 00:14:21.000
pipeline sort of functionalities available in SSIS. Uh, we were able to create, uh… SSAS Analysis Services-based warehouses, multidimensional cubes, Tableau data models.

00:14:21.000 --> 00:14:36.000
Uh, they used to act as data warehouses. We were able to model our data, like, uh, configure relationship between tables and whatnot, and then… data mining. So before the modern data science concept came out.

00:14:36.000 --> 00:14:41.000
We used to have data mining. where we had to be aware of different algorithms.

00:14:41.000 --> 00:14:47.000
Like, uh, decision trees, clustering, neural networks. Uh, association algorithm.

00:14:47.000 --> 00:14:54.000
Naivia's algorithm and whatnot. So we were able to use these different algorithms based on our requirements.

00:14:54.000 --> 00:14:59.000
project our data, forecast our data. predict our data and whatnot.

00:14:59.000 --> 00:15:06.000
And then, uh, last but not the least, we were able to visualize our data in the form of reports and create dashboards and, uh.

00:15:06.000 --> 00:15:14.000
Let our executives make tough decisions. And, uh, profit, uh, earned profit for the company and whatnot.

00:15:14.000 --> 00:15:20.000
So, this is the classic BI stack. And, uh, next.

00:15:20.000 --> 00:15:26.000
Let's talk about the limitations of the old stack before we look at the new stack.

00:15:26.000 --> 00:15:32.000
So the old stack, it was meant for simpler scenarios.

00:15:32.000 --> 00:15:39.000
smaller data sets. So, uh, maybe you just have maybe a few, uh.

00:15:39.000 --> 00:15:44.000
gigabytes, or, uh, tops just a few dozens of gigabytes of data.

00:15:44.000 --> 00:15:54.000
Definitely not hundreds of gigabytes of data. And, uh, you don't have enough variety of data sources, you just have maybe some simple file formats.

00:15:54.000 --> 00:16:01.000
Some, uh, traditional SQL Server articles titled database servers, so that is where it used to be.

00:16:01.000 --> 00:16:06.000
Good. But, uh, in just a bit, we would see that the modern technologies, they are meant.

00:16:06.000 --> 00:16:14.000
for the modern problems. So now that we have different types of problems, and this is no longer sufficient.

00:16:14.000 --> 00:16:20.000
But in general, for simpler scenarios. to create our ATL pipelines.

00:16:20.000 --> 00:16:26.000
to create order warehouses. implement data modeling, also known as semantic modeling.

00:16:26.000 --> 00:16:31.000
Uh, and then I create reports. It used to be okay.

00:16:31.000 --> 00:16:36.000
But no, this is, uh, pretty much… dead every year now.

00:16:36.000 --> 00:16:42.000
If you're still using any of these technologies. Considered using the modern trend.

00:16:42.000 --> 00:16:51.000
So, you will otherwise still get stuck with lots of, uh, bottlenecks, like, uh, your data size.

00:16:51.000 --> 00:16:55.000
if it ever grows. Out of proportions.

00:16:55.000 --> 00:17:02.000
it becomes gigantic, you will not be able to handle insane amount of data load by using your two different technologies.

00:17:02.000 --> 00:17:07.000
If you have, uh, machine learning requirements, where you want to predict your data.

00:17:07.000 --> 00:17:17.000
Especially, uh, big data, large data. where you have a high volume, or large volume, high velocity.

00:17:17.000 --> 00:17:21.000
And, uh, variety of data available, then machine learning will not be.

00:17:21.000 --> 00:17:27.000
enough, uh, based on the tools that you're using as part of your classic BI stack.

00:17:27.000 --> 00:17:33.000
And, uh, we also didn't have any support for AI, generative AI in the older tools.

00:17:33.000 --> 00:17:41.000
And there's also no, uh, streaming support live. data streaming. Support in the classic tools.

00:17:41.000 --> 00:17:48.000
So think of, let's say, Uber. 911 sort of services, where you want to get data in real time.

00:17:48.000 --> 00:17:50.000
and you want to process it in real time.

00:17:50.000 --> 00:17:54.000
So the classic BI stack is not curable enough.

00:17:54.000 --> 00:18:01.000
doing several, uh, things that are typically expected these days. So, most companies usually have, uh.

00:18:01.000 --> 00:18:08.000
Uh, the requirement to process huge volume of data, or have live real-time streaming of data.

00:18:08.000 --> 00:18:14.000
Or to implement machine learning, or in fact, even deep learning, on complex data sets.

00:18:14.000 --> 00:18:21.000
That can take several, uh… hours or even days at times to process that data.

00:18:21.000 --> 00:18:29.000
So, uh, what's the solution? So the solution is to shift to modern data engineering and data science roles.

00:18:29.000 --> 00:18:34.000
So we, uh, do not use the term. BI developer.

00:18:34.000 --> 00:18:43.000
business intelligence developer these days. Instead, we say that someone is a data engineer, someone is a data scientist, someone is a data analyst.

00:18:43.000 --> 00:18:52.000
AI engineer, and so on. So why do we use these modern, uh… terms now, it has to do with.

00:18:52.000 --> 00:19:01.000
the scale. So we no longer have just simple requirements. We, at times, have complex requirements.

00:19:01.000 --> 00:19:08.000
Uh, not even, uh… hundreds of gigabytes, but uh… hundreds of terabytes.

00:19:08.000 --> 00:19:16.000
or petabytes of data could be there. Do you have, uh, very high volume.

00:19:16.000 --> 00:19:21.000
And your datasets are just out of proportions these days.

00:19:21.000 --> 00:19:25.000
And we also have modern file formats available these days.

00:19:25.000 --> 00:19:30.000
Uh, not that we will practically see any of these, uh, in depth in this very brief session.

00:19:30.000 --> 00:19:40.000
But know that, uh, in order of. in-depth courses, we usually talk about all of the stuff practically as well, but in general, we have modern open data formats.

00:19:40.000 --> 00:19:45.000
So rather than using CSV. We use parking.

00:19:45.000 --> 00:19:51.000
So CSV, Thomas Edward Value Files. They are typically stored row by row.

00:19:51.000 --> 00:20:00.000
they are slow to read data. Parquet, it's stored column by column. So it's a column nerd.

00:20:00.000 --> 00:20:05.000
storage format. highly compact and very, very fast.

00:20:05.000 --> 00:20:11.000
So, whenever we have to store data, rather than preferring CSV format, we prefer using Parquet.

00:20:11.000 --> 00:20:17.000
And likewise, on top of parquet, we have this delta table.

00:20:17.000 --> 00:20:21.000
uh, slash Delta Lake sort of format available to us.

00:20:21.000 --> 00:20:28.000
Which adds transactional capabilities to Parquet. So Parkey, it's a highly fast.

00:20:28.000 --> 00:20:36.000
contrast file format. and to add support for transactions, like inserts, updates, deletes, and even time travel.

00:20:36.000 --> 00:20:46.000
going back to a certain point in time, what data you had a few days ago, a few months ago, a few… weeks ago, etc, we are able to use Delta Lakes, cabalities, and so on.

00:20:46.000 --> 00:20:51.000
So we have these modern file formats available, and as data engineer, we have to be aware of it.

00:20:51.000 --> 00:20:56.000
as data scientists, we have to be aware of these concepts.

00:20:56.000 --> 00:20:59.000
And then, uh, we also have, uh, something called Spark.

00:20:59.000 --> 00:21:06.000
Which is a massive. Parallel processing infrastructure slash framework.

00:21:06.000 --> 00:21:12.000
Where we have multiple nodes running in parallel. And, uh, you're able to write some.

00:21:12.000 --> 00:21:17.000
code in, let's say, Python, in SQL, in our sort of languages.

00:21:17.000 --> 00:21:26.000
And, uh, it will basically end up using all of these nodes in parallel, and process your insane amount of data in a very fast time.

00:21:26.000 --> 00:21:36.000
So Spark is basically this framework slash infrastructure. Which, uh, heavily gets utilized by Python, R.

00:21:36.000 --> 00:21:47.000
even PySpark sort of developers and whatnot. And you can use it to analyze the data, visualize the data, and even as a data engineer, as a data scientist, you can benefit from it.

00:21:47.000 --> 00:21:52.000
And then I'll talk about this lake house in more detail in just a bit, but in short.

00:21:52.000 --> 00:21:58.000
These file formats, storage formats, they are utilized internally.

00:21:58.000 --> 00:22:02.000
by this lake house. architecture.

00:22:02.000 --> 00:22:07.000
So, back in the day, like I mentioned, we used to have warehouses.

00:22:07.000 --> 00:22:16.000
We're, uh, slicing and dicing of data used to be done, and uh… we had dimensional tables and fact tables, etc, stored in a specific format.

00:22:16.000 --> 00:22:20.000
These days, we mostly prefer using this format called Lake House.

00:22:20.000 --> 00:22:25.000
Which internally makes use of these file formats or concepts.

00:22:25.000 --> 00:22:31.000
And then, uh, we will also talk about deep learning, which is basically a more specialized form of machine learning.

00:22:31.000 --> 00:22:36.000
Which is meant for more complex kind of machine learning scenarios.

00:22:36.000 --> 00:22:45.000
Uh, where you might have image recognition, uh, more complex kind of, uh, uh, pattern recognition, uh, requirements and whatnot.

00:22:45.000 --> 00:22:49.000
And it supports natural language processing, etc, as well, and whatnot.

00:22:49.000 --> 00:23:00.000
And, uh, as a data scientist, you also usually, uh, sort of get your feet wet in this Gen AI area as well. Uh, AI slash gen AI area.

00:23:00.000 --> 00:23:06.000
which I'll briefly touch on later on as well. But in short, as a data scientist, as a data engineer.

00:23:06.000 --> 00:23:13.000
Uh, we have lots of new concepts available to us that we have to be aware of.

00:23:13.000 --> 00:23:20.000
So, if this used to be the old.

00:23:20.000 --> 00:23:35.000
end-to-end architecture. What's the new end-to-end architecture? So let me actually, uh, be lazy, grab a screenshot… Second, later on… show you this for comparison.

00:23:35.000 --> 00:23:39.000
Okay, so here, if I switch over to the modern trend.

00:23:39.000 --> 00:23:51.000
Okay, so at this point, it might not even make sense what's exactly happening. We will discuss each of these sections in detail in just a bit, but in short, you can see the modern.

00:23:51.000 --> 00:24:03.000
uh, end-to-end architecture involves lots of options. And, like, the old one, where it was fairly simple, there was mostly just one way of doing things.

00:24:03.000 --> 00:24:07.000
But now we have multiple options available to us.

00:24:07.000 --> 00:24:12.000
So you could use, uh, maybe just this part. Maybe you could use just this one. You could, uh, use the combination of these.

00:24:12.000 --> 00:24:16.000
So you have lots of options available to us.

00:24:16.000 --> 00:24:23.000
So let's try to dissect this gigantic end-to-end architecture and try to make sense of, uh.

00:24:23.000 --> 00:24:29.000
These. Okay, so, uh, first thing first.

00:24:29.000 --> 00:24:36.000
Regardless of your role. Whether you are a data scientist, you are a data engineer, you are a data analyst.

00:24:36.000 --> 00:24:41.000
your end-to-end process always starts off with getting data.

00:24:41.000 --> 00:24:50.000
from different data sources. And, uh, in this section, we will see two ways of obtaining the data.

00:24:50.000 --> 00:24:56.000
So you can choose to do things by yourself, uh, self-service, DIY.

00:24:56.000 --> 00:25:03.000
If you are a non-technical person. and you don't want to ask your IT department.

00:25:03.000 --> 00:25:09.000
To, uh, implement the solution on your behalf. You can choose to do things by yourself.

00:25:09.000 --> 00:25:14.000
It's also known as self-service or DIY. option. Or, we have the more.

00:25:14.000 --> 00:25:20.000
Enterprise way of doing things. where data engineering role is involved.

00:25:20.000 --> 00:25:27.000
Okay, what is it? exactly mean. So, whenever you have to get data.

00:25:27.000 --> 00:25:33.000
We have 1, 2, 3, 4 options. that are provided by Microsoft.

00:25:33.000 --> 00:25:38.000
Actually, we have more than 4, but these are the 4 commonly used options.

00:25:38.000 --> 00:25:43.000
So the fourth option is, if you want to create your reports, dashboards, etc.

00:25:43.000 --> 00:25:46.000
And you want to connect to your data, clean it up.

00:25:46.000 --> 00:25:54.000
transform it, shape it, remove duplicates and whatnot. Do it by yourself. Don't ask your IT department.

00:25:54.000 --> 00:25:58.000
So you can choose to utilize Excel. And the modern Excel.

00:25:58.000 --> 00:26:03.000
uh, version contains this. feature called Power Queen.

00:26:03.000 --> 00:26:15.000
Which is a no-code slash very low-code option. In fact, in most cases, you don't need to write any code at all, and you're able to connect to different data sources, like SQL Server, Oracle.

00:26:15.000 --> 00:26:20.000
CSV, or even websites. You can perform web scrapping as well.

00:26:20.000 --> 00:26:26.000
So, get data from, uh, different data sources by using just Excel, or by using Power BI Desktop.

00:26:26.000 --> 00:26:34.000
And both of them, they have the same Power Query editor sort of UI. You can get the data, clean it up by yourself.

00:26:34.000 --> 00:26:41.000
So if you have used, uh, any of these options before, like Excel with Power Query or Power BI, Desktop with Power Query, awesome.

00:26:41.000 --> 00:26:49.000
Um, this is the self-service DIY option. Uh, the next option that's available to us.

00:26:49.000 --> 00:26:57.000
is Azure Data Factory. So if you say that, no, we want to do things at a more enterprise scale.

00:26:57.000 --> 00:27:01.000
We want to sort of discourage people, uh, so that they don't.

00:27:01.000 --> 00:27:08.000
always have to clean up the data by themselves. We want to clean up the data at the enterprise level.

00:27:08.000 --> 00:27:12.000
and create our warehouses, uh, lake houses, etc. So for that.

00:27:12.000 --> 00:27:21.000
we have this option called Azure Data Factory. Which is actually, uh, sort of the new… version of SSIS.

00:27:21.000 --> 00:27:27.000
Actually, it's not really just the new version, it's just the new, modern way of doing things on cloud.

00:27:27.000 --> 00:27:33.000
So we have Azure, Data Factory, which allows you to create your data pipelines.

00:27:33.000 --> 00:27:41.000
And if I briefly open up my, let's say, browser here, so I can, uh… show you what, uh, it actually looks like.

00:27:41.000 --> 00:27:49.000
So here, um, I have my Azure account. And I already have my Azure Data Factory instance created.

00:27:49.000 --> 00:27:53.000
If I open up this Azure Data Factory Studio.

00:27:53.000 --> 00:27:57.000
You will notice that here, as a data engineer.

00:27:57.000 --> 00:28:01.000
as a data engineer, you will be able to create your.

00:28:01.000 --> 00:28:08.000
uh, data pipelines. If I, let's say, open up my pipeline, or maybe a slightly more detailed one.

00:28:08.000 --> 00:28:13.000
probably just open up this one. So you'll be able to see that we have.

00:28:13.000 --> 00:28:18.000
Uh, E… extract data from different data sources.

00:28:18.000 --> 00:28:28.000
T, transformations. and load. They can choose to conditionally split your data, you can say you want to write data to multiple destinations.

00:28:28.000 --> 00:28:36.000
based on some criteria, like maybe based on department name, you'll want to make sure that certain data, after cleaning it up.

00:28:36.000 --> 00:28:40.000
We, uh, write it to a different destination of our choice and whatnot.

00:28:40.000 --> 00:28:48.000
So you can use this Azure Data Factory, which is a graphical way of creating your data pipelines, but it does support.

00:28:48.000 --> 00:28:53.000
coding and SQL, Python, and other languages as well.

00:28:53.000 --> 00:28:59.000
So that is another option available to you. Which is suitable if you mostly want to rely on.

00:28:59.000 --> 00:29:05.000
Azure-specific services. So maybe you already have the Azure Cloud Platform.

00:29:05.000 --> 00:29:10.000
So in that case, you can use this Azure Data Factory option.

00:29:10.000 --> 00:29:16.000
Or, you can say that we probably don't have Azure, we don't want to use Azure.

00:29:16.000 --> 00:29:19.000
Uh, because it does require a bit of administration.

00:29:19.000 --> 00:29:22.000
So your admins will have to set it up.

00:29:22.000 --> 00:29:28.000
And, uh, it does take a bit of effort, you need a delegated admin to do this thing for you.

00:29:28.000 --> 00:29:36.000
So that is where you can choose to say that, no, we want, uh, something that doesn't require too much administration.

00:29:36.000 --> 00:29:42.000
We do want to clean up the data, extract the data, transform it, and write it to some warehouse.

00:29:42.000 --> 00:29:48.000
So that is where you can use fabric. So Microsoft Fabric.

00:29:48.000 --> 00:29:54.000
If you have been using Power BI. as your reporting solution?

00:29:54.000 --> 00:30:02.000
We have been creating your reports, your dashboards. Then know that, uh, Power BI has been expanded.

00:30:02.000 --> 00:30:08.000
a lot. So now we have this much larger technology called Microsoft Fabric.

00:30:08.000 --> 00:30:16.000
Which is an end-to-end. data analytics platform, which can be used by data scientists.

00:30:16.000 --> 00:30:19.000
Which I'll show you in just a bit. Data engineers.

00:30:19.000 --> 00:30:29.000
data analysts, any role can benefit from it. The fabric, uh, lets you create your data pipelines in multiple ways.

00:30:29.000 --> 00:30:34.000
So, uh, Microsoft realizes that not everyone will be tech-savvy.

00:30:34.000 --> 00:30:38.000
So that is why we have multiple options available to us in Fabric.

00:30:38.000 --> 00:30:44.000
So the simplest option in Fabric to create your data pipelines.

00:30:44.000 --> 00:30:49.000
is known as Dataflow. And it's usually no code.

00:30:49.000 --> 00:30:53.000
Or very little code. In fact, in most cases, it's no code.

00:30:53.000 --> 00:30:59.000
And, uh, in fact, this is the exact same thing as Power Query.

00:30:59.000 --> 00:31:03.000
Which is available both in Excel as well as Power BI Desktop.

00:31:03.000 --> 00:31:10.000
So there's absolutely no difference here. The only difference, uh, perhaps, is it's available on, uh.

00:31:10.000 --> 00:31:14.000
cloud as part of this fabric, which I'll briefly show you in just a bit.

00:31:14.000 --> 00:31:21.000
But this is the no-code option. If you say that, no, I'm a more technical person.

00:31:21.000 --> 00:31:26.000
I don't want to have a drag-and-drop sort of option. I need some more advanced capabilities.

00:31:26.000 --> 00:31:29.000
then in that case, you can choose to look into Fabric pipeline.

00:31:29.000 --> 00:31:39.000
Which also, uh… very much like Azure Data Factory lets you drag and drop things, but you can still choose to run some Python code.

00:31:39.000 --> 00:31:47.000
SQL code as well, and so on. And if you say that, no, I'm a hardcore programmer.

00:31:47.000 --> 00:31:54.000
I'm a Python programmer, I know R, I know various other languages. So that is where you can use this notebook option.

00:31:54.000 --> 00:31:59.000
write code in Python. in art, or various other languages.

00:31:59.000 --> 00:32:04.000
and perform your ETL. operations to create your data pipelines.

00:32:04.000 --> 00:32:09.000
So you can choose to go with this pure programming-based approach as well.

00:32:09.000 --> 00:32:15.000
As part of Fabric. So fabric is an end-to-end analytics platform.

00:32:15.000 --> 00:32:25.000
And as data engineers. Later on, I'll talk about data scientist role as well, but as data engineers, you can benefit from it as well, very much like, uh.

00:32:25.000 --> 00:32:31.000
data scientists can benefit from it. Uh, and if I briefly show you this fabric, uh, here.

00:32:31.000 --> 00:32:38.000
Where is my fabric? Okay, so fabric is available actually in my same browser.

00:32:38.000 --> 00:32:48.000
Uh, this one. So, uh… Even though it's PowerBI.com, but in reality, it is Power BI as well as Fabric. So Fabric is the end-to-end analytics platform.

00:32:48.000 --> 00:32:55.000
and you're able to create your data pipelines. Uh, if I, let's say, briefly show you one small thing here.

00:32:55.000 --> 00:32:59.000
But now I'll be lazy and just open up my existing.

00:32:59.000 --> 00:33:07.000
Workspace, and you can see that you can create various options here. So, for example, you can choose to create your pipelines directly here.

00:33:07.000 --> 00:33:12.000
Where you can choose to run your code in Python and various other languages and so on.

00:33:12.000 --> 00:33:22.000
The fabric overall is an end-to-end analytics platform. And, uh, you might want to consider this option if you're already using Power BI.

00:33:22.000 --> 00:33:25.000
And you don't want to have a full-time admin.

00:33:25.000 --> 00:33:32.000
who should be responsible for the administration. Which is typically required in case of Azure Data Factory.

00:33:32.000 --> 00:33:37.000
So Azure Data Factory is suitable if you are using Azure as the cloud platform.

00:33:37.000 --> 00:33:42.000
If you don't have a full-time admin, go with Fabric. Fabric.

00:33:42.000 --> 00:33:47.000
uh, already has all of the necessary tools, services.

00:33:47.000 --> 00:33:51.000
for data engineers, for data scientists, for data analysts, and so on.

00:33:51.000 --> 00:33:57.000
And in case if you say that, uh, okay, fabric, uh, yes, it's good to know it exists.

00:33:57.000 --> 00:34:05.000
But our architects, or company, uh, has been contributing using Databricks.

00:34:05.000 --> 00:34:14.000
So Databricks, it's, uh, typically a third-party service. Which was designed or created by the same folks who created our.

00:34:14.000 --> 00:34:24.000
something called Apache Spark. And all of these technologies, like Fabric, Databricks, etc, they all have the same spark.

00:34:24.000 --> 00:34:29.000
massive parallel processing infrastructures, large framework built into them.

00:34:29.000 --> 00:34:39.000
So if you want to go with Databricks. your architects like Databricks, they have been hearing about that a lot, they have used it maybe back in the day. You can use.

00:34:39.000 --> 00:34:45.000
Databricks on Azure. And it's known as Azure Databricks.

00:34:45.000 --> 00:34:48.000
And even there, you can write the code in Python.

00:34:48.000 --> 00:34:52.000
very much like I briefly mentioned that you can have your code in a fabric.

00:34:52.000 --> 00:34:58.000
And I'll show you some of the stuff practically as well afterward, but in short, you can have your Databricks, uh.

00:34:58.000 --> 00:35:02.000
utilized with the Python code, uh, and you can also letron create your.

00:35:02.000 --> 00:35:11.000
pipelines in sort of a more graphical manner, where you can drag and drop your notebooks, you can say you want to populate your sales data, then you want to populate your.

00:35:11.000 --> 00:35:16.000
Uh, human resources, marketing sort of data, so you can create your data pipelines.

00:35:16.000 --> 00:35:24.000
Um, by combining different notebooks and so on. So, in short, uh, data engineering role.

00:35:24.000 --> 00:35:34.000
has, uh, evolved. Because, uh, we have lots of options, we have lots of technologies, we have different data types.

00:35:34.000 --> 00:35:47.000
on a variety of data sets available to us, like we have streaming data, we have, uh… file formats such as Parkey, we have Delta, etc, and so on.

00:35:47.000 --> 00:35:52.000
And if you say that, okay, it's good to know that data engineering role is available here.

00:35:52.000 --> 00:35:59.000
Uh, what's the next step? in our overall… end-to-end analytics, uh, sort of, uh.

00:35:59.000 --> 00:36:05.000
life cycle and whatnot. So the next step after you have cleaned up the data.

00:36:05.000 --> 00:36:12.000
Make sure you store it. in either a warehouse or lake house.

00:36:12.000 --> 00:36:22.000
So, again, fabric lets you store data in sort of a centralized manner, and let me just show you directly here on this slide.

00:36:22.000 --> 00:36:26.000
So after you have cleaned up the data. You can store your data.

00:36:26.000 --> 00:36:35.000
either in a warehouse, which is just. Structured tables, rows and columns sort of data.

00:36:35.000 --> 00:36:43.000
So we just have structured tables, rows and column sort of data, and data is stored in the delta table format, which is capable of, uh.

00:36:43.000 --> 00:36:58.000
Uh, using the time travel concept, you can go back to a certain point in time, find out what data you had a few days ago, a few months ago, yesterday, just an hour ago. So there's basically support for data, or, uh, record versioning available here.

00:36:58.000 --> 00:37:04.000
And, uh, this is mostly suitable if you, uh, mostly have SQL.

00:37:04.000 --> 00:37:11.000
quoting sort of requirements. Or, if you have more complex requirements.

00:37:11.000 --> 00:37:16.000
You want to act as data scientists, you want to utilize.

00:37:16.000 --> 00:37:23.000
files as well, raw files. And you still want to create your structured tables.

00:37:23.000 --> 00:37:26.000
then in that case, we use something called lacos.

00:37:26.000 --> 00:37:33.000
The lake house… It's basically, uh… warehouse plus.

00:37:33.000 --> 00:37:47.000
data lake. So, data lake usually means, um. A storage ADO where you can dump all of your files, any of your files, and it's a very fast storage area, you can process your data in a very fast manner.

00:37:47.000 --> 00:37:52.000
So the combination of data leak. Plus, warehouse is known as lake house.

00:37:52.000 --> 00:38:00.000
Where you'll have your… raw files, as well as your structured tables.

00:38:00.000 --> 00:38:15.000
And, uh, Fabric also lets you store streaming kind of data, which, uh… will be live in real time, data will be streamed, and for this purpose, Fabric supports something called EventHouse, or KQL Database Concept.

00:38:15.000 --> 00:38:26.000
And then, uh, typically, in Fabric, we also create this data model, or semantic model, so that Power BI, Tableau, Excel users can connect to this, uh, data model and create reports.

00:38:26.000 --> 00:38:31.000
And, uh… even as a data scientist.

00:38:31.000 --> 00:38:37.000
you want to be aware of these concepts so that, uh, as a data scientist, you can connect your lake house.

00:38:37.000 --> 00:38:42.000
and manipulate your data, which I'll practically show you as well in just a bit.

00:38:42.000 --> 00:38:47.000
And again, if you say that, uh, okay, it's good to know that fabric can be used.

00:38:47.000 --> 00:38:51.000
For this purpose, to store our data in a centralized manner.

00:38:51.000 --> 00:38:56.000
Uh, but again, our architects, uh, they have been considering using Datafix.

00:38:56.000 --> 00:39:03.000
So, you can use Azure Databricks. And there, you can utilize something called Unity Catalog.

00:39:03.000 --> 00:39:07.000
Uh, let me actually show this a bit practically on my side.

00:39:07.000 --> 00:39:18.000
Uh, where is my UD card log? Databricks is available here in this, uh, tab. Okay, so you can see that I have, uh, Databricks.

00:39:18.000 --> 00:39:26.000
Which can be used on Azure. It's known as Azure Databricks. And, uh, you'll be able to utilize something called Unity Catalog.

00:39:26.000 --> 00:39:32.000
where you can create your schemas or databases. Uh, you can store your structured data.

00:39:32.000 --> 00:39:39.000
Which can be used by even, uh, non-technical people, such as Power BI users and so on.

00:39:39.000 --> 00:39:43.000
Uh, or if you are a hardcore data scientist, data engineer.

00:39:43.000 --> 00:39:53.000
Uh, you might want to utilize raw files. Currently, I have .csv format utilized here, but you can also utilize .park format.

00:39:53.000 --> 00:39:58.000
which is a highly compressed file format. So, in short, Databricks.

00:39:58.000 --> 00:40:14.000
can, uh, to some extent, act as kind of a warehouse. It's not exactly a warehouse, it's just a… data lick, or a sort of a format where, uh, where we have… or delta Lake format, where we have structured tables, as well as raw fonts as well.

00:40:14.000 --> 00:40:22.000
Um, and it does support, uh, SQL warehouses, if you want to create just data warehouses for non-technical people, it does support that as well.

00:40:22.000 --> 00:40:30.000
Otherwise, even this tables area can be used by Power BI users, Tableau users, Excel users can use it as well.

00:40:30.000 --> 00:40:35.000
And data scientists can definitely benefit both from tables as well as.

00:40:35.000 --> 00:40:51.000
raw data as well. Okay, so in short, uh, we have, uh… different options. Do you want to use fabric as your end-to-end platform? Do you want to use Databix as your end-to-end platform?

00:40:51.000 --> 00:40:57.000
Uh, and then before we talk about the data, uh, or machine learning concept as well.

00:40:57.000 --> 00:41:07.000
Let's also quickly touch on data visualization. So we have, uh, ATL pipelines, or data pipelines.

00:41:07.000 --> 00:41:11.000
Because we can also have ELT pipelines as well.

00:41:11.000 --> 00:41:17.000
So ETL versus ELT, it's a huge topic, which we will not go into depth of in this brief session.

00:41:17.000 --> 00:41:22.000
But in short, we can clean up your data, store it in your lake house.

00:41:22.000 --> 00:41:27.000
And then… depending on your role.

00:41:27.000 --> 00:41:32.000
you can, uh, visualize your data. Uh, self-service DIY way of.

00:41:32.000 --> 00:41:38.000
visualizing data would be… you use Excel or Power BI.

00:41:38.000 --> 00:41:45.000
Or, if you say that I need more advanced data science capabilities to visualize data.

00:41:45.000 --> 00:41:52.000
I need to use visuals that do not… exist out of the box as part of Power BI. So here, I have Power BI open.

00:41:52.000 --> 00:42:00.000
So if you have used Power BI before, you might have noticed that we have these visuals available, but if you want to use more advanced visuals.

00:42:00.000 --> 00:42:08.000
Then, uh, you could go with the marketplace option, get more visuals, but you might not see every visual there.

00:42:08.000 --> 00:42:12.000
And that is where you can use this Python option in Power BI.

00:42:12.000 --> 00:42:19.000
So data scientists can use Python visual. I already actually have some visuals added here.

00:42:19.000 --> 00:42:22.000
So this is, uh, not available out of the box.

00:42:22.000 --> 00:42:29.000
This is a box plot, uh, the box plot visual, and actually, it's, uh, this Python visual.

00:42:29.000 --> 00:42:32.000
And you can see that we have code in Python here.

00:42:32.000 --> 00:42:40.000
The Power BI lets you visualize data. as a data scientist, by writing code in Python.

00:42:40.000 --> 00:42:46.000
So you get to use, uh, this regular Python-based library, such as matplotlib, CBON.

00:42:46.000 --> 00:42:55.000
And whatnot. You can even use Panda's data frame, NumPy for fast number crunching, etc. As well. So we have different, uh, visualizations.

00:42:55.000 --> 00:42:59.000
And none of these exist out of the box as part of Power BI.

00:42:59.000 --> 00:43:06.000
We even have this, let's say, radar visual as well, which also doesn't exist as part of the standard, uh.

00:43:06.000 --> 00:43:12.000
Power BI experience. Uh, in fact, you can also clean up your data as well.

00:43:12.000 --> 00:43:19.000
So if I, let's say, open up a Power Query Editor here, and if you have the need for more advanced data cleanup operations.

00:43:19.000 --> 00:43:23.000
And in that case, you can, um, write code in Python.

00:43:23.000 --> 00:43:35.000
Uh, so you can see that I have. Python code here, and you're able to write code in Python to clean up your data, etc, and so on, in Power BI.

00:43:35.000 --> 00:43:40.000
So, in short, as a data scientist. This is one of the options.

00:43:40.000 --> 00:43:46.000
to visualize your data directly in. Power BI, where you use some standard visuals and some.

00:43:46.000 --> 00:43:54.000
Python-based visuals. Or, you can say that, no, I don't even use Power BI.

00:43:54.000 --> 00:43:58.000
I'm a data scientist, I do want to visualize data.

00:43:58.000 --> 00:44:06.000
Um, in a more powerful manner. And I want to write code in Python, R, Scala, or various other languages. So you can use.

00:44:06.000 --> 00:44:15.000
Notebooks. So Jupyter notebooks, they are available in Azure Databricks.

00:44:15.000 --> 00:44:22.000
fabric, and later on, I'll discuss that they exist even in Azure Machine Learning Studio as well.

00:44:22.000 --> 00:44:25.000
So for now, if I open up Fabric again.

00:44:25.000 --> 00:44:33.000
And, uh, this is actually my, uh, Databricks. Even here, you can see that you are able to add your, uh, notebooks.

00:44:33.000 --> 00:44:39.000
So I can go to my workspace, uh, you can open up your, uh, notebook here, write code in, uh.

00:44:39.000 --> 00:44:43.000
Python, let's see, I'll just open up maybe this one for now.

00:44:43.000 --> 00:44:46.000
So you're able to write your code in Python.

00:44:46.000 --> 00:44:52.000
visualize your data, uh… as a data scientist and whatnot.

00:44:52.000 --> 00:45:00.000
And, uh, the same concept is available in fabric as well, so if I also show you a bit of fabric here. So even in fabric.

00:45:00.000 --> 00:45:05.000
If I, let's say, open up my notebook, so you can see that, uh, you're able to write code in Python.

00:45:05.000 --> 00:45:09.000
And I'll just open up this one for now.

00:45:09.000 --> 00:45:12.000
So you can see that, uh, overall, you're able to write code in Python.

00:45:12.000 --> 00:45:19.000
For more complex, advanced, uh… Uh, data processing, data visualization, and in fact.

00:45:19.000 --> 00:45:30.000
I'll also touch on machine learning as well in just a bit, but in short, overall, you're able to write code in Python to, uh… connecting your data, clean it up, uh, visualize it.

00:45:30.000 --> 00:45:44.000
Uh, and whatnot. And, um, if I go back to my slide deck here… Okay, so we have, uh… This one.

00:45:44.000 --> 00:45:53.000
And then, uh, typically after you have your reports created, you just create your dashboards as well, which you can create.

00:45:53.000 --> 00:45:58.000
and your notebook, uh, you can do so in your Databricks environment as well.

00:45:58.000 --> 00:46:01.000
Or you can do it in Power BI as well.

00:46:01.000 --> 00:46:08.000
And they're not machine learning. So, as a data scientist, this is what that will make you sort of stand out.

00:46:08.000 --> 00:46:14.000
Compared to other roles. So, the data scientist role.

00:46:14.000 --> 00:46:22.000
ideally should be aware of how to get data, clean it up, transform it, even, uh… visualize it.

00:46:22.000 --> 00:46:35.000
and also machine learning as well. You could argue that, uh, no, uh, I'm mostly a data… engineer, so I just want to focus on data cleanup, etc, and so on.

00:46:35.000 --> 00:46:43.000
I'm a data analyst, I want to focus on visualization, but a true data scientist should be.

00:46:43.000 --> 00:46:49.000
a jack-of-all-trades. The data scientist role should be aware of how to get the data, clean it up.

00:46:49.000 --> 00:46:56.000
Shepard, transform it, visualize it. And, very importantly, machine learning.

00:46:56.000 --> 00:47:03.000
Where you have to, uh, maybe protect data. maybe you have to classify your data.

00:47:03.000 --> 00:47:09.000
So you have maybe a bunch of customers. So you have, let's say, uh, customers who buy lots of products.

00:47:09.000 --> 00:47:15.000
customers who buy less products, so you want to basically classify your customers.

00:47:15.000 --> 00:47:20.000
Or it could be, let's say, uh, maybe diabetes kind of scenario, so is a.

00:47:20.000 --> 00:47:24.000
Patient, a diabetic or not, uh, diabetic based on.

00:47:24.000 --> 00:47:29.000
the dataset we have for blood glucose readings, etc, and whatnot.

00:47:29.000 --> 00:47:35.000
So, based on your existing data, predict. whether the customer will buy.

00:47:35.000 --> 00:47:41.000
from us next year, or not buy the product from us next year? Classify then.

00:47:41.000 --> 00:47:49.000
Regression, uh, predicting, uh, let's say. sales amount, order quantity for the next year.

00:47:49.000 --> 00:47:54.000
Um, you can utilize, actually, uh, forecasting for this purpose.

00:47:54.000 --> 00:48:00.000
As well, if you want. And then, uh, we also have various other concepts available here, but in general.

00:48:00.000 --> 00:48:07.000
Machine learning is about predicting something, forecasting something, identifying patterns in our data.

00:48:07.000 --> 00:48:19.000
I've defined data patterns, basically. And, um, it's a huge area so usually we have lots of algorithms, we have extreme gradient boost, XG boost, we have neural networks, etc.

00:48:19.000 --> 00:48:24.000
We have lots of, uh, other, uh, neural network and.

00:48:24.000 --> 00:48:29.000
other sea men is another algorithm that we utilize, and so on, so we have lots of algorithms available.

00:48:29.000 --> 00:48:35.000
That, uh, we can utilize. But in short, as a data scientist, you will end up defining.

00:48:35.000 --> 00:48:41.000
your, uh, problem? Do we have to predict something, forecast something based on time?

00:48:41.000 --> 00:48:56.000
Or, uh, just classify. uh, the existing records as a buyer, not a buyer, kind of, um… Classification can be created. So, uh, usually you define the problem, which means you identify the algorithm.

00:48:56.000 --> 00:48:59.000
You get the data, you clean it up, transform it.

00:48:59.000 --> 00:49:09.000
Then you create your model, machine learning model. which will be capable of predicting your data based on your current data.

00:49:09.000 --> 00:49:16.000
Uh, so usually we also have to get into other, uh, technologies, uh, or frameworks as well, like scikit-learn, PyTorch.

00:49:16.000 --> 00:49:20.000
or after you get into Synapse ML as well.

00:49:20.000 --> 00:49:26.000
Uh, KRAS is the year, so we have lots of options available. So you usually create your predictive models.

00:49:26.000 --> 00:49:33.000
by selecting your right algorithm. You trained your model based on your training data.

00:49:33.000 --> 00:49:41.000
And your test data, you trained your model, you deploy the model, after that, you're able to generate insights based on that.

00:49:41.000 --> 00:49:44.000
So, uh, how do you implement, uh, machine learning?

00:49:44.000 --> 00:49:51.000
what options are available. as far as, uh… these Microsoft technologies are concerned.

00:49:51.000 --> 00:49:57.000
Uh, you can use fabric. with Python, sort of, code.

00:49:57.000 --> 00:50:03.000
Or, I'll show you… you can also use this no-code option as well.

00:50:03.000 --> 00:50:07.000
So we have less time left, I'll just open up my fabric environment.

00:50:07.000 --> 00:50:16.000
Which is actually this one. Okay, so I can go to my fabric environment, and I can open up my notebook.

00:50:16.000 --> 00:50:21.000
linear regression. Okay, so, uh, here, uh, we have a bunch of lines of code.

00:50:21.000 --> 00:50:24.000
We don't need to focus too much on the code for now.

00:50:24.000 --> 00:50:27.000
But know that, uh, we have this simple scenario.

00:50:27.000 --> 00:50:33.000
Where we have, uh, to predict home prices. To protect home prices.

00:50:33.000 --> 00:50:42.000
Based on this data, so we have a number of bedrooms, we have size available, we have location, quality available.

00:50:42.000 --> 00:50:46.000
Uh, and we have, uh, estimate price for each, uh.

00:50:46.000 --> 00:50:50.000
home and sun. So if this is your current data.

00:50:50.000 --> 00:50:56.000
then, uh, you're able to write code here. So you can predict home prices.

00:50:56.000 --> 00:51:04.000
So I won't bother going to, uh, or going through the entire code, but overall, we are using this linear regression algorithm in this case.

00:51:04.000 --> 00:51:13.000
to create a linear regression-based predictive model. I'm printing my model, I'm splitting my data into test data and training data.

00:51:13.000 --> 00:51:16.000
I'm training my model, and then after that, I'm able to.

00:51:16.000 --> 00:51:31.000
uh, predict… home prices. So if you assume that, uh, some home has, let's say, two bedrooms, this is the size, this is the… location quality, then predict the price for that, so it's saying that based on the current data.

00:51:31.000 --> 00:51:34.000
the price for this home will be approximately this much.

00:51:34.000 --> 00:51:44.000
So you create your model. You later on use MLflow, machine learning flow, which is basically a model lifecycle management.

00:51:44.000 --> 00:51:51.000
sort of framework. Do you create your model, and then you… deploy or publish your model.

00:51:51.000 --> 00:51:57.000
So here, uh, I have the event data visualized as well, and I should have the model published here.

00:51:57.000 --> 00:52:10.000
If I briefly open up my model here. It should be in my… root folder. Okay, so here I can open up, let's see, this, uh, experiment for now.

00:52:10.000 --> 00:52:17.000
And overall, it will show you, uh, the, uh… experiment. Experiment means you're.

00:52:17.000 --> 00:52:25.000
trying what-if possibilities with your algorithms. And then after that, you can save it as a machine learning model, and you can start consuming it.

00:52:25.000 --> 00:52:34.000
Or, you can also use something called. AutoML, which lets you create your machine learning models.

00:52:34.000 --> 00:52:39.000
in a graphical manner, without writing. And in code.

00:52:39.000 --> 00:52:46.000
So I won't, uh, bother going through the entire process here, but overall, you're able to create your machine learning models.

00:52:46.000 --> 00:52:56.000
Even by using, there's no-code approach as well. And this is known as AutoML, which is basically an automatic, easy, designer-based way of creating your.

00:52:56.000 --> 00:53:02.000
machine learning model. And again, we have different options available here. Do you want to forecast something?

00:53:02.000 --> 00:53:09.000
time series kind of thing, or you want to… have a classification kind of a scenario involved and so on.

00:53:09.000 --> 00:53:15.000
But in short, you have your machine learning model available as part of Fabric.

00:53:15.000 --> 00:53:20.000
And, uh, you can also choose to do the same thing in Databricks as well.

00:53:20.000 --> 00:53:28.000
Which is, uh, somewhat similar. Um, I'll just briefly show you here, since we have very less time left.

00:53:28.000 --> 00:53:33.000
Uh, okay, so where does my Databricks? So even in Databricks.

00:53:33.000 --> 00:53:42.000
We have this AI ML option, experiments. And this is where you can create your experiments. Again, you can specify whether you want to have, uh.

00:53:42.000 --> 00:53:47.000
time series-based forecasting, and we have lots of options here. It will end up creating your, uh.

00:53:47.000 --> 00:53:51.000
ML model, and you can now start predicting the values and so on.

00:53:51.000 --> 00:54:02.000
So if you are a fan of writing code, awesome. Write code in Python, like I briefly showed you, and uh… fabric. Or you can use this, uh, designer-based approach as well.

00:54:02.000 --> 00:54:07.000
Which actually also exists. And one more place.

00:54:07.000 --> 00:54:14.000
And that is called Azure Machine Learning. So, as a data scientist.

00:54:14.000 --> 00:54:22.000
You can consider using Fabric as your end-to-end. Platform, or… Databricks as your end-to-end platform.

00:54:22.000 --> 00:54:28.000
Or, again, if you are currently using Azure. without fabric, without Databricks.

00:54:28.000 --> 00:54:38.000
you're using bits and pieces of… Azure Technologies, like Azure Data Factory, Azure SQL Server, Azure Datalink Storage.

00:54:38.000 --> 00:54:41.000
you can use Azure Machine Learning as a data scientist.

00:54:41.000 --> 00:54:48.000
And again, uh, here you will see that we have 3 options. We have the… no code option.

00:54:48.000 --> 00:54:54.000
So without writing any line of code, you can train your machine learning model.

00:54:54.000 --> 00:55:00.000
specify the, uh, business need. Do you want to predict something? Do you want to classify something?

00:55:00.000 --> 00:55:07.000
Uh, and it will be able to create the model, train it, and you can later on start using it to predict the values and so on.

00:55:07.000 --> 00:55:12.000
Or, if you want to have more control over the process, use this.

00:55:12.000 --> 00:55:19.000
designer-based approach. So, the designer-based approach lets you drag and drop more complex activities.

00:55:19.000 --> 00:55:29.000
fine-tune the activities. Um, in a more… Uh, sub-segated manner. Or, if you are a programmer, then in that case, write your code.

00:55:29.000 --> 00:55:37.000
in the form of Python notebooks. And that is definitely the most advanced way of, uh, creating your machine learning models.

00:55:37.000 --> 00:55:49.000
And, uh, if you also have the need for, uh, generative AI, you want to call LLM models, large language models, like OpenAI and Gemini and Cloud, and, uh.

00:55:49.000 --> 00:55:53.000
whatnot, you can implement something called prompt flow as well.

00:55:53.000 --> 00:55:56.000
So I'll just briefly point out that in Azure.

00:55:56.000 --> 00:56:06.000
We have, uh, something called… Um… okay, so here I can open up Azure Machine Learning Studio in my Azure environment.

00:56:06.000 --> 00:56:12.000
And you can see that here we are able to use this automated ML. If you want to use the no-code option.

00:56:12.000 --> 00:56:22.000
This is the, uh, minimal effort kind of. scenario, where you're able to simply just, uh, specify exactly what type of problem you want to, uh, solve.

00:56:22.000 --> 00:56:29.000
Do you want to have time series-based forecasting, or… classify data as yes, no.

00:56:29.000 --> 00:56:38.000
true, false kind of thing. Uh, customer… Or, um, not a customer buyer, not a buyer kind of scenario, you can use these options.

00:56:38.000 --> 00:56:49.000
This is the no-code option. And if you want to have a bit of more control over things, then in that case, you can use this designer-based option, where you can drag and drop activities, connect them in a specific manner.

00:56:49.000 --> 00:56:54.000
For a code-based approach, use the notebooks option, where you can write code in Python.

00:56:54.000 --> 00:57:01.000
Uh, and uh, if you want to also call your LLM models, large language models, like OpenAI.

00:57:01.000 --> 00:57:08.000
Gemini, Claude, etc, then in that case, you can use the prompt flow option as part of Azure machine learning.

00:57:08.000 --> 00:57:14.000
And moving forward, Azure machine learning will eventually become a part of Fabric.

00:57:14.000 --> 00:57:21.000
So everything will be integrated just in Fabric. The fabric will be, uh, a single stop.

00:57:21.000 --> 00:57:25.000
one-stop solution that contains everything, or will contain everything.

00:57:25.000 --> 00:57:36.000
Eventually. Okay, and then I'll just briefly mention that there's also the lifecycle management concept, where you can choose to implement, uh.

00:57:36.000 --> 00:57:46.000
The whole process and automate it, schedule it. So anytime you source data changes, you want to say that I want to create my new machine learning model.

00:57:46.000 --> 00:57:51.000
deployed in an automated manner, so we do have the ML apps available.

00:57:51.000 --> 00:57:56.000
or support available to you. So think of ML Apps as, like, DevOps.

00:57:56.000 --> 00:58:01.000
But this is meant for machine learning scenarios, where your machine learning models will be deployed.

00:58:01.000 --> 00:58:06.000
Periodically, in an automated manner. whenever your source data has changed.

00:58:06.000 --> 00:58:14.000
So we have different, uh, tools available for this purpose. We don't have any time, plus it's not the scope of this brief session to talk about the tools.

00:58:14.000 --> 00:58:18.000
But we have lots of tools available to implement this automated.

00:58:18.000 --> 00:58:28.000
Machine learning model deployment concept and whatnot. And last but not the least, uh… We have lots of data involved here.

00:58:28.000 --> 00:58:33.000
So you might want to secure your data. Who has access to what?

00:58:33.000 --> 00:58:39.000
data scientist roles, uh, data engineer, data analyst role, uh, they all have to.

00:58:39.000 --> 00:58:45.000
only access what is really meant for them, so that is where we use something called data governance.

00:58:45.000 --> 00:58:52.000
In technology, we have something called Purview, which lets you secure your data sources and whatnot.

00:58:52.000 --> 00:59:02.000
It's a whole different area, it's. Where you can implement data governance, who has access to what, uh, mask… data masking, and whatnot.

00:59:02.000 --> 00:59:12.000
Alrighty, so, uh… Any questions regarding, uh, this brief session?

00:59:12.000 --> 00:59:17.000
If yes, please type the message in the chat window, and I'll let Alex take over.

00:59:17.000 --> 00:59:21.000
Thanks, Rahim, and thank you for presenting. This was great.

00:59:21.000 --> 00:59:28.000
If there are no questions, I'll still just wait a moment, but I'm gonna throw something in the chat. There's 3 hyperlinks, just so everyone's aware I mentioned in the beginning.

00:59:28.000 --> 00:59:38.000
that if you are interested in, perhaps, private training, or even maybe a dedicated webinar, very much like the one that Fahim presented today for your organization or your team.

00:59:38.000 --> 00:59:49.000
You can click that first URL. The second URL is a webinar that we are doing January 21st, What's New in the AWS Data and Analytics stack for 2026.

00:59:49.000 --> 01:00:04.000
That will also be at 1PM Eastern. And then we have another webinar after that, February 5th, called VMware Workload Migration, Your Path Forward. That is also at 1PM Eastern, so you can register at those links below, but.

01:00:04.000 --> 01:00:11.000
If there are no other questions, I'll wait a moment, but again, just wanted to say thank you, Fahim.

01:00:11.000 --> 01:00:15.000
Thank you, Alex, and thank you, everyone, for joining the session.

01:00:15.000 --> 01:00:23.000
Perfect. Okay, if there are any questions, um, you can reach out afterwards, and then I will be sending a follow-up email with the recording as well for.

01:00:23.000 --> 01:00:25.000
your reference. So, uh, thank you everyone for joining, and I hope you have a good day, and thank you, Fahim.

01:00:25.000 --> 01:00:31.000
Bye

Beyond BI: A Data Scientist's Guide to Microsoft's Modern Data Stack [Webinar]

What You'll Experience

Audience

About the Presenter

Alert!

Modal Title

Error!

Default Title

Prompt

Confirm

Login