The Misinformation Susceptibility Test

University of Cambridge psychologists have developed the first validated “misinformation susceptibility test”.

The quick two-minute quiz gives a solid indication of how vulnerable a person is to being duped by the kind of fabricated news that is flooding online spaces. 

The test, proven to work through a series of experiments involving over 8,000 participants taking place over two years, has been deployed by polling organisation YouGov to determine how susceptible Americans are to fake headlines. 

The first survey to use the new 20-point test, called ‘MIST’ by researchers and developed using an early version of ChatGPT, has found that – on average – adult US citizens correctly classified two-thirds (65%) of headlines they were shown as either real or fake. 

However, the polling found that younger adults are worse than older adults at identifying false headlines, and that the more time someone spent online recreationally, the less likely they were to be able to tell real news from misinformation. 

This runs counter to prevailing public attitudes regarding online misinformation spread, say researchers – that older, less digitally-savvy “boomers” are more likely to be taken in by fake news. 

The study presenting the validated MIST is published in the journal Behavior Research Methods, and the polling is released today on the YouGov US website. 

Researchers encourage the public to test themselves: https://yourmist.streamlit.app.

Selecting true or false against 20 headlines gives the user a set of scores and a “resilience” ranking that compares them to the wider US population. It takes under two minutes to complete.  

“Misinformation is one of the biggest challenges facing democracies in the digital age,” said Prof Sander van der Linden, senior author of the MIST study, and head of the Cambridge Social Decision-Making Lab. 

“We are seeing how online falsehoods create polarised belief systems in major nations”

 _{Prof Sander van der Linden}

“To understand where and how best to fight misinformation, we need a unified way of measuring susceptibility to fake news. That is what our test provides,” said van der Linden, author of the new book Foolproof.

The Cambridge team developed assessment tools that enabled them to work out the right level and mix of fake and genuine headlines to produce the most reliable results. 

Examples of real news came from outlets such as the Pew Research Center and Reuters.

To create false but confusingly credible headlines – similar to misinformation encountered “in the wild” – in an unbiased way, researchers used artificial intelligence: ChatGPT version 2. 

“When we needed a set of convincing but false headlines, we turned to GPT technology. The AI generated thousands of fake headlines in a matter of seconds. As researchers dedicated to fighting misinformation, it was eye-opening and alarming,” said Dr Rakoen Maertens, MIST lead author.

However, another recent study by the same team used GPT to produce useful questions for a variety of psychological surveys. “We encourage our fellow psychologists to embrace AI and help steer the technology in beneficial directions,” said MIST co-author Dr Friedrich Götz.    

For the MIST, an international committee of misinformation experts whittled down the true and false headline selections. Variations of the survey were then tested extensively in experiments involving thousands of UK and US participants.

The latest YouGov survey saw 1,516 adult US citizens take the MIST in April 2023, and also respond to questions covering demographics, politics and online behaviour.  

When it came to age, only 11% of 18-29 year olds got a high score (over 16 headlines correct), while 36% got a low score (10 headlines or under correct). By contrast, 36% of those 65 or older got a high score, while just 9% of older adults got a low score. 

Additionally, the longer someone spent online for fun each day, the greater their susceptibility to misinformation, according to the MIST. Some 30% of those spending 0-2 recreation hours online each day got a high score, compared to just 15% of those spending 9 or more hours online. 

The survey also analysed channels through which respondents receive their news. The “legacy media” came out top. For example, over 50% of those who got their news from the Associated Press, or NPR, or newer outlets such as Axios, achieved high scores.  

Social media had the news audiences most susceptible to misinformation. Some 53% of those who got news from Snapchat received low scores, with just 4% getting high scores. Truth Social was a close second, followed by WhatsApp, TikTok and Instagram. 

Democrats performed better than Republicans on the MIST, with 33% of Democrats achieving high scores, compared to just 14% of Republicans. However, almost a quarter of both parties’ followers were in the low-scoring bracket. 

Perhaps alarmingly, half of all Americans now say they see what they believe to be misinformation online every day, according to the YouGov poll. 

Dr Maertens added: “Younger people increasingly turn to social media to find out about the world, but these channels are awash with misinformation.

"Approaches to media literacy, as well as algorithms and platform design, require an urgent rethink.”

“The MIST will allow us to verify the effectiveness of interventions to tackle fake news. We want to explore why some people are more resilient to misinformation, and what we can learn from them.”