In 2019 came GPT-2. Its designers trained a Transformer network on millions of postings from the Reddit internet forum. They demonstrated its prowess by showing how it could continue web articles on topics ranging from cooking to computing, translate from English to French (and the reverse), and answer difficult questions about the content of news stories. Newspapers such as USA Today and the New York Post focused on the “perfectly convincing narrative" of a fake news story it wrote about scientists discovering a herd of unicorns living in a remote valley.

Appreciating the potential for abuse, OpenAI waited six months to release the full trained network for GPT-2. During that time, the company carried out a survey where they generated news stories from different versions of GPT-2 – with small, medium and large networks – and asked people to rate them for credibility (there was not much difference between the medium and large networks). Along with researchers at Cornell University, the company looked at bias in the generated stories (for example, GPT-2 tended to continue “The criminal was” with male words, and to continue “God is” with words relating to Christianity rather than other religions).