This book is an intervention -
From predicting criminality to sexual orientation, fake and deeply flawed Artificial Intelligence (AI) is rampant. Amidst this feverishly hyped atmosphere, this book interrogates the rise and fall of AI hype, pseudoscience and snake oil. Bringing together different perspectives and voices from across disciplines and countries, it draws connections between injustices inflicted by inappropriate AI. Each chapter unpacks lazy and harmful assumptions made by developers when designing AI tools and systems, and examines the existential underpinnings of the technology itself to ask: why are there so many pointless, and even dangerously flawed, AI systems?
The book is free to download, or can be ordered in print from meatspacepress.com. NEW: You can also read all chapters below.
Content
Introduction
Frederike Kaltheuner
Chapter 1
An interview with Arvind Narayanan
Chapter 2
Abeba Birhane
Chapter 3
Deborah Raji
Chapter 4
Frederike Kaltheuner
Chapter 5
Razvan Amironesei, Emily Denton, Alex Hanna, Hilary Nicole, Andrew Smart
Chapter 6
Serena Dokuaa Oduro
Chapter 7
James Vincent
Chapter 8
Alexander Reben
Chapter 9
Gemma Milne
Chapter 10
Crofton Black
Chapter 11
Adam Harvey
Chapter 12
Andrew Strait
Chapter 13
Tulsi Parida, Aparna Ashok
Chapter 14
Favour Borokini, Ridwan Oloyede
Chapter 15
Fieke Jansen, Corinne Cath
Chapter 16
This project was made possible by the generous support of the Mozilla Foundation through its Tech Policy Fellowship programme.License: Creative Commons BY-NC-SA
“Much of what is sold commercially today as ‘AI’ is what I call ‘snake oil’. We have no evidence that it works, and based on our scientific understanding of the relevant domains, we have strong reasons to believe that it couldn’t possibly work.”
Arvind Narayanan
Introduction
Frederike Kaltheuner
Not a week passes by without some research paper, feature article or product marketing making exaggerated or even entirely unlikely claims about the capabilities of Artificial Intelligence (AI). From academic papers that claim AI can predict criminality, personality or sexual orientation, to the companies that sell these supposed capabilities to law enforcement, border control or human resources departments around the world, fake and deeply flawed AI is rampant.
The current amount of public interest in AI was spurred by the genuinely remarkable progress that has been made with some AI techniques in the past decade. For narrowly defined tasks, such as recognising objects, AI is now able to perform at the same level or even better than humans. However, that progress, as Arvind Narayanan has argued, does not automatically translate into solving other tasks. In fact, when it comes to predicting any social outcome, using AI is fundamentally dubious. [1]
The ease and frequency with which AI’s real and imagined gains are conflated results in real, tangible harms.
For those subject to automated systems, it can mean the difference between getting a job and not getting a job, between being allowed to cross a border and being denied access. Worse, the ways in which these systems are so often built in practice means that the burden of proof often falls on those affected to prove that they are in fact who they say they are. On a societal level, widespread belief in fake AI means that we risk redirecting resources to the wrong places. As Aidan Peppin argues in this book, it could also mean that public resistance to the technology will end up stifling progress in areas where genuine progress is being made.
What makes the phenomenon of fake AI especially curious is the fact that, in many ways, 2020-21 has been a time of great AI disillusionment. The Economist dedicated its entire summer Technology Quarterly to the issue, concluding that “An understanding of AI’s limitations is starting to sink in.” [2] For a technology that has been touted as the solution to virtually every challenge imaginable—from curing cancer, to fighting poverty, predicting criminality, reversing climate change and even ending death—AI has played a remarkably minor role [3] in the global response to a very real challenge the world is facing today, the Covid-19 pandemic. [4] As we find ourselves on the downward slope of the AI hype cycle, this is a unique moment to take stock, to look back and to examine the underlying causes, dynamics, and logics behind the rise and fall of fake AI.
Bringing together different perspectives and voices from across disciplines and countries, this book interrogates the rise and fall of AI hype, pseudoscience, and snake oil. It does this by drawing connections between specific injustices inflicted by inappropriate AI, unpacking lazy and harmful assumptions made by developers when designing AI tools and systems, and examining the existential underpinnings of the technology itself to ask: why are there so many useless, and even dangerously flawed, AI systems?
Any serious writing about AI will have to wrestle with the fact that AI itself has become an elusive term. As every computer scientist will be quick to point out, AI is an umbrella term that’s used for a set of related technologies. Yet while these same computer scientists are quick to offer a precise definition and remind us that much of what we call AI today is in fact machine learning, in the public imagination, the term AI has taken on a meaning of its own. Here, AI is a catch-all phrase used to describe a wide-ranging set of technologies, most of which apply statistical modelling to find patterns in large data sets and make predictions based on those patterns—as Fieke Jansen and Corinne Cath argue in their piece about the false hope that’s placed in AI registers.
Just as AI has become an imprecise word, hype, pseudoscience, and snake oil are frequently used interchangeably to call out AI research or AI tools that claim to do something they either cannot, or should not do. If we look more closely however, these terms are distinct. Each highlights a different aspect of the phenomenon that this book interrogates.
Dangerously, they’ve acquired a veneer of innovation, a sheen of progress, even. By contrast, in a wide-ranging interview that considers how much, and how little, has changed since his original talk three years ago, Arvind Narayanan hones in on “AI snake oil”, explaining how it is distinct from pseudoscience. Vendors of AI snake oil use deceptive marketing, fraud, and even scams to sell their products as solutions to problems for which AI techniques are either ill-equipped or completely useless.
The environment in which snake oil and pseudoscience thrives is characterised by genuine excitement, unchallenged hype, bombastic headlines, and billions of dollars of investment, all coupled with a naïve belief in the idea that technology will save us. Journalist James Vincent writes about his first encounter with a PR pitch for an AI toothbrush and reflects on the challenges of covering hyped technology without further feeding unrealistic expectations. As someone who used to work as a content moderator for Google in the mid 2010s, Andrew Strait makes a plea against placing too much hope on automation in content moderation.
Each piece in this book provides a different perspective and proposes different answers to problems which circle around the shared question of what is driving exaggerated, flawed or entirely unfounded hopes and expectations about AI. Against broad-brush claims, they call for precise thinking and scrupulous expression.
For Deborah Raji the lack of care with which engineers so often design algorithmic systems today belongs to a long history of engineering irresponsibility in constructing material artefacts like bridges and cars. Razvan Amironesei, Emily Denton, Alex Hanna, Andrew Smart and Hilary Nicole describe how benchmark datasets contribute to the belief that algorithmic systems are objective or scientific in nature. The artist Adam Harvey picks apart what exactly defines a “face” for AI.
A recurring theme throughout this book is that harms and risks are unevenly distributed.
Tulsi Parida and Aparna Ashok consider the effects of AI inappropriately applied through the Indian concept of jugaad. Favour Borokini and Ridwan Oloyede warn of the dangers that come with AI hype in Nigeria’s fintech sector.
Amidst this feverishly hyped atmosphere, this book makes the case for nuance. It invites readers to carefully separate the real progress that AI research has made in the past few years from fundamentally dubious or dangerously exaggerated claims about AI’s capabilities.
We are not heading towards Artificial General Intelligence (AGI). We are not locked in an AI race that can only be won by those countries with the least regulation and the most investment.
Instead, the real advances in AI pose both old and new challenges that can only be tamed if we see AI for what it is. Namely, a powerful technology that at present is produced by only a handful of companies with workforces that are not representative of those who are disproportionately affected by its risks and harms.
Notes
1. Narayanan, A. (2019) How to recognize AI snake oil. Princeton University, Department of Computer Science. https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
2. Cross, T. (2020, 13 June) An understanding of AI’s limitations is starting to sink in. The Economist. https://www.economist.com/technology-quarterly/2020/06/11/an-understanding-of-ais-limitations-is-starting-to-sink-in
3. Mateos-Garcia, J., Klinger, J., Stathoulopoulos, K. (2020) Artificial Intelligence and the Fight Against COVID-19. Nesta.
https://www.nesta.org.uk/report/artificial-intelligence-and-fight-against-covid-19/
4. Peach, K. (2020) How the pandemic has exposed AI’s limitations. Nesta.
https://www.nesta.org.uk/blog/how-the-pandemic-has-exposed-ais-limitations/
The current amount of public interest in AI was spurred by the genuinely remarkable progress that has been made with some AI techniques in the past decade. For narrowly defined tasks, such as recognising objects, AI is now able to perform at the same level or even better than humans. However, that progress, as Arvind Narayanan has argued, does not automatically translate into solving other tasks. In fact, when it comes to predicting any social outcome, using AI is fundamentally dubious. [1]
The ease and frequency with which AI’s real and imagined gains are conflated results in real, tangible harms.
For those subject to automated systems, it can mean the difference between getting a job and not getting a job, between being allowed to cross a border and being denied access. Worse, the ways in which these systems are so often built in practice means that the burden of proof often falls on those affected to prove that they are in fact who they say they are. On a societal level, widespread belief in fake AI means that we risk redirecting resources to the wrong places. As Aidan Peppin argues in this book, it could also mean that public resistance to the technology will end up stifling progress in areas where genuine progress is being made.
What makes the phenomenon of fake AI especially curious is the fact that, in many ways, 2020-21 has been a time of great AI disillusionment. The Economist dedicated its entire summer Technology Quarterly to the issue, concluding that “An understanding of AI’s limitations is starting to sink in.” [2] For a technology that has been touted as the solution to virtually every challenge imaginable—from curing cancer, to fighting poverty, predicting criminality, reversing climate change and even ending death—AI has played a remarkably minor role [3] in the global response to a very real challenge the world is facing today, the Covid-19 pandemic. [4] As we find ourselves on the downward slope of the AI hype cycle, this is a unique moment to take stock, to look back and to examine the underlying causes, dynamics, and logics behind the rise and fall of fake AI.
Bringing together different perspectives and voices from across disciplines and countries, this book interrogates the rise and fall of AI hype, pseudoscience, and snake oil. It does this by drawing connections between specific injustices inflicted by inappropriate AI, unpacking lazy and harmful assumptions made by developers when designing AI tools and systems, and examining the existential underpinnings of the technology itself to ask: why are there so many useless, and even dangerously flawed, AI systems?
Any serious writing about AI will have to wrestle with the fact that AI itself has become an elusive term. As every computer scientist will be quick to point out, AI is an umbrella term that’s used for a set of related technologies. Yet while these same computer scientists are quick to offer a precise definition and remind us that much of what we call AI today is in fact machine learning, in the public imagination, the term AI has taken on a meaning of its own. Here, AI is a catch-all phrase used to describe a wide-ranging set of technologies, most of which apply statistical modelling to find patterns in large data sets and make predictions based on those patterns—as Fieke Jansen and Corinne Cath argue in their piece about the false hope that’s placed in AI registers.
Just as AI has become an imprecise word, hype, pseudoscience, and snake oil are frequently used interchangeably to call out AI research or AI tools that claim to do something they either cannot, or should not do. If we look more closely however, these terms are distinct. Each highlights a different aspect of the phenomenon that this book interrogates.
As Abeba Birhane powerfully argues in her essay, Cheap AI, the return of pseudoscience, such as race science, is neither unique nor distinct to AI research. What is unique is that dusty and long-discredited ideas have found new legitimacy through AI.
Dangerously, they’ve acquired a veneer of innovation, a sheen of progress, even. By contrast, in a wide-ranging interview that considers how much, and how little, has changed since his original talk three years ago, Arvind Narayanan hones in on “AI snake oil”, explaining how it is distinct from pseudoscience. Vendors of AI snake oil use deceptive marketing, fraud, and even scams to sell their products as solutions to problems for which AI techniques are either ill-equipped or completely useless.
The environment in which snake oil and pseudoscience thrives is characterised by genuine excitement, unchallenged hype, bombastic headlines, and billions of dollars of investment, all coupled with a naïve belief in the idea that technology will save us. Journalist James Vincent writes about his first encounter with a PR pitch for an AI toothbrush and reflects on the challenges of covering hyped technology without further feeding unrealistic expectations. As someone who used to work as a content moderator for Google in the mid 2010s, Andrew Strait makes a plea against placing too much hope on automation in content moderation.
Each piece in this book provides a different perspective and proposes different answers to problems which circle around the shared question of what is driving exaggerated, flawed or entirely unfounded hopes and expectations about AI. Against broad-brush claims, they call for precise thinking and scrupulous expression.
For Deborah Raji the lack of care with which engineers so often design algorithmic systems today belongs to a long history of engineering irresponsibility in constructing material artefacts like bridges and cars. Razvan Amironesei, Emily Denton, Alex Hanna, Andrew Smart and Hilary Nicole describe how benchmark datasets contribute to the belief that algorithmic systems are objective or scientific in nature. The artist Adam Harvey picks apart what exactly defines a “face” for AI.
A recurring theme throughout this book is that harms and risks are unevenly distributed.
Tulsi Parida and Aparna Ashok consider the effects of AI inappropriately applied through the Indian concept of jugaad. Favour Borokini and Ridwan Oloyede warn of the dangers that come with AI hype in Nigeria’s fintech sector.
Amidst this feverishly hyped atmosphere, this book makes the case for nuance. It invites readers to carefully separate the real progress that AI research has made in the past few years from fundamentally dubious or dangerously exaggerated claims about AI’s capabilities.
We are not heading towards Artificial General Intelligence (AGI). We are not locked in an AI race that can only be won by those countries with the least regulation and the most investment.
Instead, the real advances in AI pose both old and new challenges that can only be tamed if we see AI for what it is. Namely, a powerful technology that at present is produced by only a handful of companies with workforces that are not representative of those who are disproportionately affected by its risks and harms.
Notes
1. Narayanan, A. (2019) How to recognize AI snake oil. Princeton University, Department of Computer Science. https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
2. Cross, T. (2020, 13 June) An understanding of AI’s limitations is starting to sink in. The Economist. https://www.economist.com/technology-quarterly/2020/06/11/an-understanding-of-ais-limitations-is-starting-to-sink-in
3. Mateos-Garcia, J., Klinger, J., Stathoulopoulos, K. (2020) Artificial Intelligence and the Fight Against COVID-19. Nesta.
https://www.nesta.org.uk/report/artificial-intelligence-and-fight-against-covid-19/
4. Peach, K. (2020) How the pandemic has exposed AI’s limitations. Nesta.
https://www.nesta.org.uk/blog/how-the-pandemic-has-exposed-ais-limitations/
“Just as the car manufacturer called out by Nader shifted blame onto car dealerships for failing to recommend tyre pressures to “correct” the Corvair’s faulty steering, algorithm developers also seek scapegoats for their own embarrassing failures.”
Deborah Raji
Want to read more? All Meatspace Press publications are free to download, or can be ordered in print from meatspacepress.com.
Design notes
AI-constrained design
By Carlos Romo-Melgar, John Philip Sage and Roxy Zeiher
The design of this book explores the use of artificial intelligence to conceptualise its main visual elements. From an optimistic standpoint of AI abundance, the book design is a semi-fiction, a staged and micromanaged use of a GAN (Generative Adversarial Network), an unsupervised machine learning framework, where two neural networks contest with each other in order to generate visual output. Stretching the narrative, this book could be framed as a/the (first) book designed by an AI. In this scenario, the collaborating AI (more like the AI-as-head-of-design-that-doesn’t-know- how-to-design), has informed, but also constrained the possibilities to work visually with the pages.
The design strategy adopts the Wizard of Oz Technique, a method originated from interaction design where what is seemingly auton- omous, is in reality disguising the work of humans ‘as a proxy for the system behind the scenes’. [1] The use of the GAN, which a reader could expect as a simplification, a symbol of technologicalergonomics, has instead complicated the process. As a result, the contents contort around the spaces that the AI imagination left them to exist, revealing an apparently spontaneous visual language.
The book features results from two separate datasets, addressing the overall layout composition, and a (overly sensitive) recognition algorithm which targets all instances of ‘AI, ai, Ai’, regardless of their position or meaning.
MetaGAN v.3 Layouts
The dataset used to produce the compositions above is a collec- tion of book scans. The purpose of an image GAN is to create new instances by detecting, deconstructing and subsequently reconstructing existing patterns to create speculations about con- tinuations. Reusing existing layout materials, conceived by human creativity, opens up the discussion of AI creativity. The outcomes, which could be perceived as surprising, original and novel, are however subject to human selection and valuation. In training the MetaGAN, the dissimilarity of the data points, in combination with the small size of the dataset (200 images), led to the idiosyncrasy of overfitting. An overfitted model generates outcomes ‘that correspond too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably’. [2]
AI type results
These AI letterings are results of a GAN using a dataset containing logos from various AI related brands (or belonging to Anguilla, whose country code top-level domain is ‘.ai’). The use of these characters is indeed automated in the design of the book, but it is done using GREP styles.
Notes
1. Bella, M. & Hanington, B., 2012. Universal Methods of Design, Beverly, MA: Rockport Publishers. p.204
2. https://www.lexico.com/definition/overfitting
Contributors
[in order of appearance]Abeba Birhane is a cognitive science PhD candidate at the Complex Software Lab, University College Dublin, Ireland, and Lero, the Science Foundation Ireland Research Centre for Software.
Deborah Raji is a Mozilla fellow, interested in algorithmic auditing. She also works closely with the Algorithmic Justice League initiative to highlight bias in deployed products.
Frederike Kaltheuner is a tech policy analyst and researcher. She is also the Director of the European AI Fund, a philanthropic initiative to strengthen civil society in Europe.
Dr Razvan Amironesei is a research fellow in data ethics at the University of San Francisco and a visiting researcher at Google, currently working on key topics in algorithmic fairness.
Dr Emily Denton is a Senior Research Scientist on Google’s Ethical AI team, studying the norms, values, and work practices that structure the development and use of machine learning datasets.
Dr Alex Hanna is a sociologist and researcher at Google.
Hilary Nicole is a researcher at Google.
Andrew Smart is a researcher at Google working on AI governance, sociotechnical systems, and basic research on conceptual foundations of AI.
Serena Dokuaa Oduro is a writer and policy researcher aiming for algorithms to uplift Black communities. She is the Policy Research Analyst at Data & Society.
James Vincent is a senior reporter for The Verge who covers artificial intelligence and other things. He lives in London and loves to blog.
Alexander Reben is an MIT-trained artist and technologist who explores the inherently human nature of the artificial
Gemma Milne is a Scottish science and technology writer and PhD researcher in Science & Technology Studies at University College London. Her debut book is Smoke & Mirrors: How Hype Obscures the Future and How to See Past It (2020).
Dr. Crofton Black is a writer and investigator. He leads the Decision Machines project at The Bureau of Investigative Journalism. He has a PhD in the history of philosophy from the Warburg Institute, London.
Adam Harvey is a researcher and artist based in Berlin. His most recent project, Exposing.ai, analyses the information supply chains of face recognition training datasets.
Andrew Strait is a former Legal Policy Specialist at Google and works on technology policy issues. He holds an MSc in Social Science of the Internet from the Oxford Internet Institute.
Tulsi Parida is a socio-technologist currently working on AI and data policy in fintech. Her previous work has been in edtech, with a focus on responsible and inclusive learning solutions.
Aparna Ashok is an anthropologist, service designer, and AI ethics researcher. She specialises in ethical design of automated decision-making systems.
Fieke Jansen is a doctoral candidate at Cardiff University. Her research is part of the Data Justice project funded by ERC Starting Grant (no.759903).
Dr Corinne Cath is a recent graduate of the Oxford Internet Institute's doctoral programme.
Aidan Peppin is a Senior Researcher at the Ada Lovelace Institute. He researches the relationship between society and technology, and brings public voices to ethical issues of data and AI.
Fake AI
Edited by: Frederike Kaltheuner
Publisher: Meatspace Press (2021)
Weblink: meatspacepress.com
Design: Carlos Romo-Melgar, John Philip Sage and Roxy Zeiher
Copy editors: David Sutcliffe and Katherine Waters
Format: Paperback and pdf
Printed by: Petit. Lublin, Poland.
Paper: Munken Print White 20 - 90 gsm
Set in: Roobert and Times New Roman
Length: 206 pages
Language: English
Product code: MSP112101
ISBN (paperback): 978-1-913824-02-0
ISBN (pdf, e-book): 978-1-913824-03-7
License: Creative Commons BY-NC-SA
Deborah Raji is a Mozilla fellow, interested in algorithmic auditing. She also works closely with the Algorithmic Justice League initiative to highlight bias in deployed products.
Frederike Kaltheuner is a tech policy analyst and researcher. She is also the Director of the European AI Fund, a philanthropic initiative to strengthen civil society in Europe.
Dr Razvan Amironesei is a research fellow in data ethics at the University of San Francisco and a visiting researcher at Google, currently working on key topics in algorithmic fairness.
Dr Emily Denton is a Senior Research Scientist on Google’s Ethical AI team, studying the norms, values, and work practices that structure the development and use of machine learning datasets.
Dr Alex Hanna is a sociologist and researcher at Google.
Hilary Nicole is a researcher at Google.
Andrew Smart is a researcher at Google working on AI governance, sociotechnical systems, and basic research on conceptual foundations of AI.
Serena Dokuaa Oduro is a writer and policy researcher aiming for algorithms to uplift Black communities. She is the Policy Research Analyst at Data & Society.
James Vincent is a senior reporter for The Verge who covers artificial intelligence and other things. He lives in London and loves to blog.
Alexander Reben is an MIT-trained artist and technologist who explores the inherently human nature of the artificial
Gemma Milne is a Scottish science and technology writer and PhD researcher in Science & Technology Studies at University College London. Her debut book is Smoke & Mirrors: How Hype Obscures the Future and How to See Past It (2020).
Dr. Crofton Black is a writer and investigator. He leads the Decision Machines project at The Bureau of Investigative Journalism. He has a PhD in the history of philosophy from the Warburg Institute, London.
Adam Harvey is a researcher and artist based in Berlin. His most recent project, Exposing.ai, analyses the information supply chains of face recognition training datasets.
Andrew Strait is a former Legal Policy Specialist at Google and works on technology policy issues. He holds an MSc in Social Science of the Internet from the Oxford Internet Institute.
Tulsi Parida is a socio-technologist currently working on AI and data policy in fintech. Her previous work has been in edtech, with a focus on responsible and inclusive learning solutions.
Aparna Ashok is an anthropologist, service designer, and AI ethics researcher. She specialises in ethical design of automated decision-making systems.
Fieke Jansen is a doctoral candidate at Cardiff University. Her research is part of the Data Justice project funded by ERC Starting Grant (no.759903).
Dr Corinne Cath is a recent graduate of the Oxford Internet Institute's doctoral programme.
Aidan Peppin is a Senior Researcher at the Ada Lovelace Institute. He researches the relationship between society and technology, and brings public voices to ethical issues of data and AI.
Fake AI
Edited by: Frederike Kaltheuner
Publisher: Meatspace Press (2021)
Weblink: meatspacepress.com
Design: Carlos Romo-Melgar, John Philip Sage and Roxy Zeiher
Copy editors: David Sutcliffe and Katherine Waters
Format: Paperback and pdf
Printed by: Petit. Lublin, Poland.
Paper: Munken Print White 20 - 90 gsm
Set in: Roobert and Times New Roman
Length: 206 pages
Language: English
Product code: MSP112101
ISBN (paperback): 978-1-913824-02-0
ISBN (pdf, e-book): 978-1-913824-03-7
License: Creative Commons BY-NC-SA
For press requests, please email: mail [a] frederike-kaltheuner.com
