InstructGPT

InstructGPT is a family of large language models introduced by OpenAI in early 2022. The models were described in the paper "Training language models to follow instructions with human feedback," which detailed the use of reinforcement learning from human feedback (RLHF) to align outputs with user intent.

While GPT-3 was trained purely on next-token prediction over internet text, InstructGPT adds a fine-tuning stage using RLHF so the model follows instructions more reliably. This results in outputs that are more helpful, honest, and aligned with what users actually want.

Yes, ChatGPT, released by OpenAI in November 2022, was built on a model from the GPT-3.5 family fine-tuned with the same RLHF methodology developed for InstructGPT. The InstructGPT research laid the direct technical groundwork for ChatGPT.

OpenAI trained InstructGPT models at multiple parameter sizes, including 1.3B, 6B, and 175B parameters, comparing them against the baseline GPT-3. The paper showed that human raters consistently preferred the smallest 1.3B InstructGPT model's outputs over the 175B GPT-3 model's outputs.