Adding New Knowledge to LLMs

Topics Covered

In service of teaching and demonstrating how to add knowledge to and customize LLMs for enterprise use, this workshop will cover the following topics and technologies:

• Data Curation and Synthetic Data Generation
• Advanced LLM Evaluation Techniques (including LLM-as-a-Judge and ELO)
• Continued Pretraining (CPT) for Knowledge Injection
• Supervised Fine-Tuning (SFT) for Skill Acquisition
• Direct Preference Optimization (DPO) for Behavioral Alignment
• Model Optimization: Quantization, Pruning, and Knowledge Distillation
• NVIDIA NeMo Framework, NeMo Curator, NeMo Evaluator, and NeMo-RL
• TensorRT-LLM for High-Performance Inference

Data Curation and Synthetic Data Generation

• Learn to prepare large-scale, high-quality datasets using NVIDIA NeMo Curator.
• Perform essential data curation tasks: text cleaning, filtering, and PII removal.
• Generate high-quality synthetic Question-Answer pairs to create robust datasets for Supervised Fine-Tuning (SFT).
• Understand the importance of data quality in the LLM development lifecycle.

Evaluating Large Language Models

• Explore multiple LLM evaluation techniques, from simple "eyeballing" to systematic, quantitative methods.
• Evaluate models against industry-standard benchmarks like MMLU.
• Implement LLM-as-a-judge for nuanced, automated evaluation.
• Use the NeMo Evaluator microservice to compare zero-shot vs. few-shot (in-context learning) performance.
• Track and visualize evaluation experiments using MLflow.

Customizing LLMs

• Dive into three key customization techniques: CPT, SFT, and DPO.
• Use Continued Pretraining (CPT) to teach a model new knowledge about a specific domain.
• Apply Supervised Fine-Tuning (SFT) to teach a model new skills, such as solving math problems in a different language.
• Utilize Direct Preference Optimization (DPO) to align a model's conversational style to human preferences (e.g., formal vs. informal, specific dialects).
• Gain hands-on experience with the NeMo framework for all customization tasks.

Optimizing LLMs for Deployment

• Learn to compress and accelerate LLMs for efficient inference.
• Apply Post-Training Quantization (PTQ) to reduce model size and memory usage using TensorRT-LLM, focusing on the FP8 format.
• Use Depth Pruning to reduce model size by removing entire layers.
• Employ Knowledge Distillation to recover performance lost during pruning by training a smaller "student" model to mimic a larger "teacher" model.
• Evaluate the performance vs. accuracy trade-offs of each optimization technique.

Interactive Assessment

• Apply your knowledge in a hands-on coding assessment.
• Use Direct Preference Optimization (DPO) to align a Llama 3.1 8B model to a unique conversational style (Shakespearean English).
• Demonstrate your ability to prepare a preference dataset, run an alignment job with NeMo-RL, and evaluate the final model.
• Earn a certificate of competency by successfully completing the assessment.