John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

What Hacker News said

Recommends mike_hearn · 2023-04 · on “It's not only you, there's an explosion of ChatGPT on HN”

Sure, here you go: [this video] Summarizing: 1. Obviously you can't completely "solve" truthfulness because people disagree on what is and is not true. But you can go a long way. 2. The models do know what they don't know. Their level of uncertainty is not only expressed in the final token logprobs but also seems to be reified somehow, such that they can express their own level of certainty in words. 3. Many of the problems are introduced by subtle issues introduced during training which effectively teach the AI to guess. One example is by training on QA data sets where the answer is never "I don't know". RLHF introduces its own problems because the human trainers don't know what the model knows,

Recommends KKKKkkkk1 · 2023-05 · on “Don Knuth Plays with ChatGPT”

There is this idea that the goal of RLHF is to make ChatGPT woke or as you put it to lobotomize it. I suspect that this is a conspiracy theory. There's a very good talk by John Schulman, chief architect of ChatGPT [0], where he explains that if you don't include a RL component in your training, you're essentially doing imitation learning. It's well known that imitation learning fails miserably when presented with conditions that are not in your training set, i.e., answering questions that don't exist on the Internet already. So the goal of RLHF is actually to reduce hallucination. [0] [this video]

Cites lysozyme · 2023-04 · on “Training open-source LLMs on ChatGPT output is a really bad idea.”

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer. The author

Cites airgapstopgap · 2023-05 · on “The False Promise of Imitating Proprietary LLMs”

This is exactly the reason OpenAI isn't afraid of the open-source community, like many kneejerk opponents of regulatory capture assume (they are probably still afraid of Google). Also why they still do the expensive and cumbersome RLHF training, instead of those deceptively cheap and fast finetunes. They understand their own tech and why there isn't free lunch. Recently, John Schulman explained the issue with behavior cloning and it's a very typical ML problem.[1] Basically: what are we training the model to do? The model updates after finetuning in a holistic manner, based on the sum total of its content and capability. Suppose GPT-4 can correctly answer to many requests because it knows co

Cites chongli · 2023-07 · on “AI Safety and the Age of Dislightenment”

How is that "as good as an intern?" Interns, for the most part, do not hallucinate and confidently lie on topics they know nothing about. Good interns tend to ask a lot of questions and quickly learn the cultural cues around the office, allowing them to fit in and become productive. LLMs are incapable of any of this. John Schulman from OpenAI recently gave a talk about the hallucination and uncertainty issues [1]. He claimed that models already have information about their uncertainty but that it remains an open problem as to how to express that uncertainty in natural language. One of the big issues with trying to prevent the model from hallucinating is that you can err too much on the side