Deploying RAG Pipelines for Production at Scale

Topics Covered

In service of teaching and demonstrating how to deploy enterprise-scale LLM-based agentic and RAG applications this workshop will cover the following topics and technologies:

• The current landscape of enterprise generative AI applications
• NVIDIA NIM microservices
• Components and architecture of enterprise-grade RAG applications
• At-scale inference considerations and optimations
• Kubernetes, Helm, and the NVIDIA RAG operator to deploy, manage, and scale RAG application services
• Prometheus and Grafana for cluster-wide application behavior and performance visibility
• Techniques for deploying and scaling multimodal RAG applications at scale

Production Deployment of Generative AI Applications

• Survey the current landscape of generative AI applications across a wide variety of capabilities.
• Review the many constituent parts of enterprise grade generative AI applications including RAG pipelines.
• Understand the challenges that enterprises face when transitioning from naive to enterprise-grade generative AI applications.
• Learn about the capabilities of NVIDIA NIM microservices for deploying LLMs and other generative AI application components.
• Discuss techniques and technologies for improving the performance of LLM-based application inference at scale.

Core Concepts of Enterprise-scale Generative AI DevOps

• Survey the current landscape of available DevOps tools for enterprise-scale containerized application deployment.
• Learn about the value of the Kubernetes container orchestration platform.

Deploying Simple RAG Applications

• Learn how to access remote LLMs and RAG services over API calls.
• Review the core LangChain programming patterns used in engineering RAG applications.
• Build a simple RAG application using API-hosted services, and deploy it with Docker Compose.

Kubernetes Core Concepts

• Learn the core concepts and techniques required for working with Kubernetes clusters.
• Familiarize yourself with the interactive multi-node Kubernetes cluster programming environment provided to you in this workshop.
• Interactive utilize `kubectl` to deploy, manage, and monitor container-based applications across a cluster.

Deploying Self-hosted RAG Applications

• Deploy and coordinate a variety of containerized microservices in service of a cluster-based RAG application.
• Learn about and utilize the NIM operator to manage and scale a variety of RAG microservices.
• Configure storage on your cluster for various model caches.
• Deploy LLM, text embedding, and text retriever services onto your cluster.

Monitoring GPU Utilization

• Use NVIDIA Data Center GPU Manager (DCGM) to monitor and manage GPU resources across a Kubernetes cluster.
• Deploy Prometheus and Grafana onto the cluster to better monitor and visualize cluster resources.
• Use DCGM Exporter to export GPU utilization metrics for Prometheus which can be visualized with Grafana

Autoscaling NIM Microservices

• Use Prometheus service monitors to extract custom metrics from the NIM microservices running on the cluster.
• Create HorizontalPodAutoscalers to automatically scale the cluster's services based on custom metrics.
• Test and observe automatic horizontal autoscaling by performing load tests using Locust.

Building Multimodal RAG Pipelines

• Learn how to isolate different modalities from multimodal PDF documents like text, figures, and tables.
• Practice various strategies for chunking extracted text.
• Perform table extraction from PDF documents.
• Perform image/table extraction from PDF documents.
• Use the NV-YOLOX model to identify PDF page elements.
• Perform end-to-end multimodal data extraction on the ChipNemo technical paper.

Using Generative AI to Represent Extracted Modalities

• Create detailed, contextual descriptions for extracted elements such as text, figures, charts, and tables.
• Perform image transform using VLMs.
• Use a state-of-the-art context-aware chart element detection (CHART) model to detect classes for chart basic elements, including plot elements.
• Combine the use of LLMs and VLMs in an end-to-end example on the ChipNemo technical paper.

Multimodal Embedding, Storing, and Retrieving

• Convert all extracted modalities into a common format for use in a universal RAG pipeline.
• Construct an end-to-end multimodal RAG pipeline.

Workshop Assessment

• Use everything you've learned in the workshop to deploy and scale your own enterprise-grade RAG pipeline and earn a certificate of competency for the workshop.