Intermediate1h22m

Reinforcement Learning From Human Feedback

Instructor: Nikita Namjoshi

Earn an accomplishment with PRO

Enroll for Free

Intermediate
1h22m
6 Video Lessons
4 Code Examples
Earn an accomplishment with PRO
Instructor: Nikita Namjoshi
Google Cloud
Learn more aboutMembership PRO Plan

Enroll for Free

What you'll learn

Get a conceptual understanding of Reinforcement Learning from Human Feedback (RLHF), as well as the datasets needed for this technique
Fine-tune the Llama 2 model using RLHF with the open source Google Cloud Pipeline Components Library
Evaluate tuned model performance against the base model with evaluation methods

Course recap

PRO

This course provides a practical introduction to RLHF, the technique behind ChatGPT's alignment. Students work with the open-source Llama 2 model on a summarization task, learning to prepare preference and prompt datasets, train a reward model, and fine-tune the LLM using reinforcement learning.

Concept map

Key concepts

Reinforcement learning from human feedbackThree-step process: collect preferences, train reward model, fine-tune LLM with RL (PPO)
Preference datasetTriplets of (prompt, chosen_response, rejected_response) annotated by human labelers
Reward modelTrained on preference data to predict human preference scores for prompt-response pairs
Prompt datasetCollection of prompts from the same distribution as preference data, used in the RL loop
PPO (Proximal Policy Optimization)RL algorithm used to fine-tune the LLM based on reward model scores

Lesson highlights

1.Introduction — Conceptual overview of RLHF pipeline
2.Datasets for RL Training — Preference dataset structure (input_text, candidate_0, candidate_1, choice); prompt dataset
3.Reward Model Training — Training a reward model from preference pairs
4.Fine — tuning with RL - Using PPO to optimize the LLM against the reward model
5.Evaluation — Comparing pre- and post-RLHF model outputs
6.Conclusion

About this course

Large language models (LLMs) are trained on human-generated text, but additional methods are needed to align an LLM with human values and preferences.

Reinforcement Learning from Human Feedback (RLHF) is currently the main method for aligning LLMs with human values and preferences. RLHF is also used for further tuning a base LLM to align with values and preferences that are specific to your use case.

In this course, you will gain a conceptual understanding of the RLHF training process, and then practice applying RLHF to tune an LLM. You will:

Explore the two datasets that are used in RLHF training: the “preference” and “prompt” datasets.
Use the open source Google Cloud Pipeline Components Library, to fine-tune the Llama 2 model with RLHF.
Assess the tuned LLM against the original base model by comparing loss curves and using the “Side-by-Side (SxS)” method.