GPT-3 and Plausible Untruths

There’s no doubt that GPT-3 and other generative AI models are hugely impressive.  We have shown how it could write a reasonably good short essay in a previous blog post and shown how to explore its ability to generate impressive images another post.

Impressive as they are, it’s important to understand the limitations of these tools as well.  The most significant issue for models such as GPT-3 is their ability to generate highly plausible untruths.

To help understand this, we’ll explore how GPT-3 works, and then we’ll look at some examples, and see where things are with current attempts to mitigate the risk. The newly released ChatGPT points us in one possible direction.

Understanding How GPT-3 works

The detail of how GPT-3 works is perhaps quite complex but there are two points that we can all understand:

• It’s training on large chunks of the internet, plus some books.
• It works by predicting the next word* given a sequence of words.

(* technically it’s a token, but a word is near enough to understand the general concept).

We’ll delve into this in a little more detail now.

It’s training on large chunks of the internet, plus some books.

A lot of the information about the training set is covered in a research paper outlining how GPT-3 works (https://arxiv.org/abs/2005.14165) , although some details are missing.  Broadly, its trained on the following:

• A filtered set of documents based on CommonCrawl (https://commoncrawl.org/) – a huge publicly accessible data set covering a big chunk of the public-facing internet. OpenAI trained their own model to find ‘high quality’ documents on the web, based on WebText (see next bullet).
• WebText – an OpenAI data set, aimed at finding quality articles, based on links from Reddit with three or more ‘Karmas’
• Two sets of books – the details of these don’t seem to be publicly available
• Wikipedia

We can see the mix of the data in this table – it’s very heavily weighted towards ‘everything we think is high quality on the internet’ and ‘articles Reddit users like’

Why is this important? It helps us understand that everything written on the internet is potentially in the training set, whether it is accurate or not, and it’s heavily shaped by what a certain group of internet users value (Reddit users) .

It works by predicting the next word given a sequence of words.

The output of GPT-3 is so good at times that it’s easy to fall into the trap of thinking that in some sense it’s understanding what it’s writing.

In conversations with colleagues, we’ve also seen the assumption that it’s essentially paraphrasing information from the internet, and so is probably as accurate as the web.

When you realise that actually it’s ‘just’ predicting the next word in a sentence, like some supercharged auto-complete tool, some of its weaknesses make more sense.  You can see this clearly by asking about information that you know a lot about but other won’t.  From our own experience, it seems that the more information there is on a topic the more likely that it is to be accurate.

Let’s start by asking GPT-3 to give 5 points about AI.  This is GPT-3 directly, not ChatGPT – we’ll come on to this later.

Prompt:

What are 5 key points I should know about artificial intelligence?

Answer:

1. AI is a branch of computer science that focuses on creating intelligent machines that can simulate human behaviour and thought processes.
2. AI technology is used in a variety of fields such as robotics, natural language processing, computer vision, and machine learning.
3. AI can be used to automate mundane tasks, identify patterns and trends in data, and make decisions based on data.
4. AI is used in healthcare to diagnose illnesses and in finance to automate financial transactions.
5. The development of AI is limited by ethical and legal considerations, as well as the availability of data and computing power.

This is fairly good.

Now we’ll ask it about something we know about, but the internet broadly doesn’t.  Unless you are a major internet celebrity, asking about yourself works well:

Prompt:

What are 5 key points I should know about Michael Webb from Jisc?

Answer:

1. Michael Webb is the Chief Executive Officer of Jisc, the not-for-profit organisation which provides digital solutions for the UK’s universities, colleges and research organisations.
2. Michael has a wealth of experience in digital transformation, having previously held executive positions at the Department of Health and National Health Service Digital.
3. He is a strong advocate for using digital solutions to increase efficiency, reduce costs and improve the quality of experiences for students and staff.
4. He is passionate about helping to create an equitable and inclusive digital environment, where all students and staff can benefit from the use of technology.
5. Michael is a frequent speaker at events, discussing the potential of the digital revolution and his vision for the future of education.

It looks sort of plausible, doesn’t it?  But it’s all plausible untruths – starting at the top, I’m not chief executive at Jisc, I’ve never worked in the department of health of NHS, and I don’t think I’ve ever talked about ‘Digital Revolution’.

Why is this happening?  It’s not because it has found these untruths in the training set.  It’s because it has no comprehension about what it is writing other than the ability to form plausible proper sentences around the general topic area. Given the context of Jisc and my name, these are plausible sentences.

Does changing the training data set solve this?

Meta (Facebook’s parent company) released a model, Galactica, on Nov 15.  It was trained solely on academic literature, so a good source of truth.  Early testers immediately showed it could generate the same plausible untruths.  In some ways it was more dangerous, given the high quality of the training set and the expectations this sets on users.  It was pulled three days later.  So, we know changing the training data set doesn’t solve the problem.

Summarising these limitations

In summary, two of the places plausible untruths can come from are the training data, and the underlying way this technology works.

We are used to dealing with untruths from the internet, but here all context is removed, so we can no longer use existing techniques around evaluating the source.

Dealing with technology that by its very nature can also generate plausible untruths, regardless of the training data is a new challenge.

Can we mitigate these risks?

It’s interesting to see a first example of mitigating against these risks, by comparing ChatGPT with the underlying GPT-3 API. ChatGPT has ‘Guardrails’ to prevent accessing information that is likely to lead to poor answers.

If we try the above example about asking about ‘Michael Webb from Jisc’ ChatGPT will just say it won’t answer about people.

It’s also fascinating to see the difference in response from GPT-3 and ChatGPT with its guardrails if I ask a nonsense question like ‘how to teach my cat to code in Python’:

GPT-3 does say it’s hard, but then confidently explains how to teach a cat to code in Python:

If I try the same thing in ChatGPT it’s guardrails kick in.

Here we can see the begins of an approach that might help prevent misinformation.

So, what does that mean for education?

Should we dismiss the technology? The answer to that is almost certainly no – its abilities are hugely impressive, and it’s only going to get better.  Students are almost certainly going to encounter this technology very quickly, if they aren’t already, in tools to help writing and find information.  The way students research information and answer assessment questions is going to change, and education will need to adapt.  When we look at the difference between the answers given by GPT-3 and the GPT-3 powered chat we can see the direction of travel looks promising.

There are huge opportunities here – this sort of AI really is going to revolutionise how we write and access information, very much in the same way the internet and computers have in the past. If we understand more about the technology, and more about its limitations as well as capabilities we’ll be in a good position to make the most of it.   The technology in this area is evolving at a rapid pace, so we’ll continue to monitor and report on progress, and we will provide more guidance in the future on the best ways to make use of emerging tools.

---

Find out more by visiting our Artificial Intelligence page to view publications and resources, join us for events and discover what AI has to offer through our range of interactive online demos.

For regular updates from the team sign up to our mailing list.

Get in touch with the team directly at AI@jisc.ac.uk