Training AI Without Writing A Reward Function, with Reward Modelling

What Hacker News said

Strong rec sillysaurusx · 2019-12 · on “Autonomous DeLorean drives sideways to move forward”

I would highly recommend [this video] as a way of framing the problem. It seems like our options for ML are currently: self-play, dataset fitting, or human feedback. It's possible to solve any problem that can fit into one of those molds. (Unfortunately the opposite is true: if a problem doesn't fit into those categories, it seems very hard to solve with any form of ML.) But, now that I think about it a bit more, the sentence "use a trained net as a tool for building a controller with known continuity properties" seems neither obvious nor straightforward to do. I'm decently well-versed in AI techniques at this point, and I can't really think how I'd sit down and write a program that would generat

Strong rec cubefox · 2023-07 · on “What We Know About LLMs (A Primer)”

Specifically about RLHF, I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper: [this video] RLHF is actually older than GPT-1, which came out in 2018. It didn't get applied to language models until 2022 with InstructGPT, an approach which combined supervised instruction fine-tuning with RLHF.

Recommends afarrell · 2020-07 · on “Ask HN: How do I get out of the procrastination abyss?”

ADHDer here. Please steal some of the motivation-techniques I've learnt over the years: 1. Find a friend who is a software engineer or data analyst and cares about you as a person. Talk about this with them. Humans evolved being able to talk things over with their aunt or best friend. When you are perplexed by something, a rubber duck probably won't cut it -- you should talk with a human. 2. Make sure you're taking care of your body and brain: exercise, good food, good sleep (this is easier if you're getting exercise), meditation, etc. 3. Understand what your "reward model" is. Under what conditions do you recognize "Aha! I've made progress!"? To really understand what I'm getting at here, w

Cites afarrell · 2020-08 · on “Persisting as a solo founder”

as an ADHDer, one thing I've found key is figuring out what things in my environment and mind are the things I want to signal "you're doing the right thing, keep going." Essentially, what is your 'reward model'? -- [this video] Test-Driven-Development is one example of this.

Cites afarrell · 2021-01 · on “Setting goals for 2021 – A brief guide about personal goal setting”

I think your URL is a fantastic choice and hints at what a lot of "accountability coaching" really is: Camaraderie An easily-imaginable person who acts as a reward model[1] you can imagine saying "yea! Make progress on the thing! You're on the right path[2]! Keep going." When you anticipate [social] rewards for an action, that action is motivating. If you watch Dr. Andrew Huberman's talks about the roles of Dopamine in the brain, he'll say it is not a signal of reward, but of anticipated reward. It does spike when you recognise a surprisingly strong reward, (like when you hit a jackpot or an unexpectedly interesting HN link) to help you learn to repeat the path to that reward[2], but mostly

Cites afarrell · 2021-01 · on “Ask HN: How To Overcome Information Addiction?”

A. Meet your basic biological needs. Your brain is a part of your body. A person with an iron deficiency can feel compelled to chew ice[1]. Ice does not cures anemia, brains are weird parts of the body and do weird things when the body is not getting what it needs. So, get better sleep, food, and exercise. Prof Andrew Huberman has some good tips[2]. B. Decide what few things you really value and allocate your time to those on purpose. If you find yourself repeatedly drawn to the news, maybe you are genuinely give a fuck about civic engagement. Your personality has a need for it. Trying to meet that need by scrolling twitter is like eating 3 bags of tortilla chips for dinner -- quick-access,