DeepLearning.AI
Intermediate

Post-training of LLMs

Instructor: Banghua Zhu

Nexusflow logoUniversity of Washington logo

Earn an accomplishment with PRO

  • Intermediate
  • Earn an accomplishment with PRO
  • Instructor: Banghua Zhu
  • NexusflowNexusflowUniversity of WashingtonUniversity of Washington
  • Learn more aboutMembership PRO Plan

What you'll learn

  • Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning.

  • Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method.

  • Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities.

Course recap

PRO

This course, taught by Banghua Zhu (University of Washington / NexusFlow), covers three fundamental post-training methods for customizing pre-trained language models: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (PPO/GRPO). Students download pre-trained models and post-train them hands-on.

Concept map

Concepts in Post-training of LLMs and the courses that connect to themPost-training of LLMsFine tuningDirect Preference Optimization (DPO)Direct Preference Optimiz…Reinforcement learning from human feedbackReinforcement learning fr…GRPO (Group Relative Policy Optimization)GRPO (Group Relative Poli…Reward functionsParameter-Efficient Fine-Tuning (LoRA)Parameter-Efficient Fine-…QuantizationReinforcement Learning from Human FeedbackReinforcement Learning fr…Reinforcement Fine-Tuning LLMs with GRPOReinforcement Fine-Tuning…Pretraining LLMsQuantization FundamentalsDSPy: Build and Optimize Agentic AppsDSPy: Build and Optimize…Evaluating and Debugging Generative AIEvaluating and Debugging…AI Coding Workflows From Cloud to LocalAI Coding Workflows From…Building Adaptive AI AgentsBuilding Adaptive AI Agen…Concepts in Post-training of LLMs and the courses that connect to themPost-training of LLMsPost-training o…Fine tuningDirect Preference Optimization (DPO)Direct Preferen…Reinforcement learning from human feedbackReinforcement l…GRPO (Group Relative Policy Optimization)GRPO (Group Rel…Reward functionsParameter-Efficient Fine-Tuning (LoRA)Parameter-Effic…Quantization

Key concepts

  • Fine tuningSFT imitates labeled prompt-response pairs by maximizing response probability; best for jumpstarting new behaviors
  • Direct Preference Optimization (DPO)Contrastive learning from preferred/rejected response pairs; pushes model toward good and away from bad outputs
  • Reinforcement learning from human feedbackPPO uses a trained reward model and value function for token-level credit assignment
  • GRPO (Group Relative Policy Optimization)DeepSeek's algorithm that generates multiple responses per prompt and uses group-relative rewards, eliminating the need for a value model
  • Reward functionsTrained reward models for open-ended generation vs. verifiable rewards (math checkers, unit tests) for correctness-based tasks
  • Parameter-Efficient Fine-Tuning (LoRA)Adding low-rank matrices (B*A) instead of full delta-W, saving memory while learning less but forgetting less
  • QuantizationReferenced as complementary to post-training for making models accessible

Lesson highlights

  1. 1.Introduction — Overview of pre-training vs. post-training, three methods, course roadmap
  2. 2.Introduction to Post — training - When to post-train vs. prompt/RAG, evaluation suites (Chatbot Arena, LiveCodeBench, AIME, IFEval)
  3. 3.Basics of SFT — Loss function (negative log-likelihood on responses only), best use cases, data curation (distillation, best-of-k, filtering)
  4. 4.SFT in Practice — Training Qwen3-0.6B base model to follow instructions using HuggingFace TRL SFTTrainer
  5. 5.Basics of DPO — Contrastive loss derivation, reparameterized reward model, beta hyperparameter, correction-based data curation
  6. 6.DPO in Practice — Changing Qwen 2.5-0.5B identity from "Qwen" to "Deep Qwen" using DPO
  7. 7.Basics of Online RL — PPO vs. GRPO comparison, advantage estimation, verifiable vs. trained rewards
  8. 8.Online RL in Practice — GRPO training on GSM8K math problems with verifiable reward functions
  9. 9.Conclusion — Comparison of all three methods, why online RL degrades performance less than SFT

About this course

Learn to post-train and customize an LLM in this short course, “Post-training of LLMs,” taught by Banghua Zhu, Assistant Professor at the University of Washington, and co-founder of NexusFlow.

Before a large language model can follow instructions or answer questions, it undergoes two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning.

Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks.

In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on input-output pairs with ideal output responses. With DPO, you provide both a preferred (‘chosen’) and a less preferred (‘rejected’) response, and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance.

You’ll learn the basic concepts, common use-cases, and principles for curating high-quality data for effective training in each of these methods. Through hands-on labs, you’ll download a pre-trained model from HuggingFace and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior.

In detail, you’ll:

  • Understand what post-training is, when to use it, and how it differs from pre-training.
  • Build an SFT pipeline to turn a base model into an instruct model.
  • Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones.
  • Implement a DPO pipeline to change the identity of a chat assistant.
  • Learn online RL methods like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions.
  • Train a model with GRPO to improve its math capabilities using a verifiable reward.

Post-training is one of the most rapidly developing areas of LLM training.

Whether you’re looking to create a safer assistant, fine-tune a model’s tone, or improve task-specific accuracy, this course gives you hands-on experience with the most important techniques shaping how LLMs are post-trained today.

Who should join?

This course is for AI builders who want to adapt language models for specific tasks or behaviors. If you’re familiar with LLM basics and ready to go beyond pre-training, this course will help you understand and apply the key techniques that make LLMs truly useful.

Course Outline

0 Lessons・0 Code Examples
Unlock certificates

Elevate your learning experience with Pro

Upgrade to Pro and gain unlimited accomplishments on your resume

Learn More

Instructor

Banghua Zhu

Banghua Zhu

Course access is free for a limited time during the DeepLearning.AI learning platform beta!

Want to learn more about Generative AI?

Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!

Enroll for Free

Choose Your Plan

Planning for more users?
Learn More