First described in May 2020, Generative Pre-trained<span class="mw-reflink-text">[lower-alpha 1]</span> Transformer 3 (GPT-3) is an unsupervised transformer language model and the successor to GPT-2.<span class="mw-reflink-text">[109]</span><span class="mw-reflink-text">[110]</span><span class="mw-reflink-text">[111]</span> OpenAI stated that full version of GPT-3 contains 175 billion parameters,<span class="mw-reflink-text">[111]</span> two orders of magnitude larger than the 1.5 billion parameters<span class="mw-reflink-text">[112]</span> in the full version of GPT-2 (although GPT-3 models with as few as 125 million parameters were also trained).<span class="mw-reflink-text">[113]</span>

OpenAI stated that GPT-3 succeeds at certain "meta-learning" tasks. It can generalize the purpose of a single input-output pair. The paper gives an example of translation and cross-linguistic transfer learning between English and Romanian, and between English and German.<span class="mw-reflink-text">[111]</span>

GPT-3 dramatically improved benchmark results over GPT-2. OpenAI cautioned that such scaling up of language models could be approaching or encountering the fundamental capability limitations of predictive language models.<span class="mw-reflink-text">[114]</span> Pre-training GPT-3…