What an error by Literary Digest in 1936 teaches us about Amazon and AI bias todayWhat an error by Literary Digest in 1936 teaches us about Amazon and AI bias today
What an error by Literary Digest in 1936 teaches us about Amazon and AI bias today
January 24, 2020
by Dr Clement Chastagnol, Sidetrade
24 January 2020
On 3 November 1936, Franklin D. Roosevelt was triumphantly elected
President of the United States, with over 60% of the votes, much to the
embarrassment of Literary Digest, a leading magazine, which had predicted
a landslide victory for his Republican opponent, Governor Alfred Landon of
Kansas. The poll used by the magazine was skewed, to say the least.
By today’s standards, in fact, the bias of the astronomical 2.4 million-person sample would even raise a few smiles.
The respondents were all readers of the magazine, registered vehicle owners, and telephone users. In other words, considering that this was the Great Depression, Literary Digest’s respondents were a representative sample of the wealthiest population of the country!
On the other hand, using a comparatively tiny sample of just 5,000
respondents, the statistician George H. Gallup predicted a Roosevelt victory
with 56% of the vote.
This event saw the birth of the science of
opinion polling. The 1936 American election is known by every statistics
Skip forward eight decades to Amazon’s facial recognition system, which proved to be inept at recognizing dark-skinned subjects. What might have been considered as a rather ordinary mishap in a lab, caused an uproar in the scientific community, once it was understood that the application was to be used in vivo by law enforcement agencies.
Amazon made the same blunder as Literary Digest:
using non-representative data. Since the machine learning was based mostly on
fair-skinned individuals, it logically proved to be faulty recognizing subjects
outside of its field of reference.
When the problem is posed in these terms, the
trick to solving it seems simple: just provide the algorithm with a
sufficiently diverse corpus of data to improve its performance. True. But what
do you do if the data is rare, expensive to come by, or worse, when bias is
consubstantial with the data?
Where does bias in artificial intelligence systems come from? Is it
intentional? Is there a way to avoid it or sustainably eliminate it? Are the
solutions only technical?
The issue is considered serious enough for governments to start addressing
it. In fact, the European Commission has recently presented a set of Ethics Guidelines for
Trustworthy AI, aiming to establish an appropriate
legal and ethical framework for artificial intelligence. It is true that there
have been shockwaves from scandals confirming these worries.
And the ethical concerns are not confined to AI applied to the public at
large. The business world also needs to sit up and take note.
AI ethics should be a business consideration, too
AI is becoming an indispensable part of digital
transformation in business. Strategic choices, winning new markets, customer
relations, financial management – AI holds a promise of enhanced efficiency in
every profession and every sector.
According to IDC, a leading provider of market intelligence, worldwide investment in artificial
intelligence (AI) systems is expected to double by 2022.?
Let’s take the case of an algorithm that we want
to train to determine credit risk, something businesses will be all too
Using machine learning, the system will analyze all the loans handled by the bank over the last ten years: which loan applications were accepted, at what rate, according to various criteria (e.g. revenue, family status place of residence).
But by doing so, the system is sure to reproduce the biases of the human decisions of that period. For example, if applicants from a specific geographic area tended to be rejected, or if they were required to pay higher interest rates, the prejudice would be perpetuated by the machine.
What had been simply a statistical bias observed
in a limited dataset would become a systematic rule for the algorithm. What’s worse
is that this rule would be implicit, for in most cases, the algorithm is a
“black box”, which is not intended to explain its choices.
In fact, it seemed to be at play in recent months, with prominent users of Apple Card, the new credit card service by Apple, such as David Hansson (co-founder of Basecamp) or Steve Wozniak (co-founder of Apple), reporting that wide differences in credit limit with their respective wives, even as they share the same account.
So far the communication by Apple seems to have
been insufficient, in part because it seems hard to explain the decisions made
by the algorithm behind the scenes.
In her book Weapons of math destruction, data
scientist and AI fairness expert Cathy O’Neil lays out the danger of algorithms that are
solely intended to create short-term value: they tend to exacerbate
However, algorithms are far easier to audit than
people, and they should be, particularly when they impact the experience of
users, and above all, the lives of private citizen.
How to fix the problem
What, then, can be done? Over and beyond technical solutions, the real
answer is a strategic approach, and that, we must say, is good news indeed!
An AI application is a computer program, and as
such, it pursues an objective determined by its designer. In the case of our
example, AI is intended to reproduce the banker’s decision-making process as
closely as possible.
This is making the implicit assumption that the
human decisions encoded in the data are perfect and that deviating from those
(or, in technical terms, generating a higher error-rate) is to be avoided.
To proceed differently, it would thus be
necessary, during the learning phase, to consider more than just the error
rate. It is perfectly possible to jointly optimize the error rate along with a
metric of gender equality for instance. It just has to be specified by the
designers of the algorithm. It is true that it will come at some cost to the
error rate, but overall the decisions made by the machine will match more
closely what is expected from users.
This all demands a technical approach which is
not only more complex (and costly), but also highly structuring in terms of
strategy. In our example, is the bank ready to take a little extra credit risk
for the sake of fairness? In the Artificial intelligence era, one can but dream
(and not just of electronic sheep)!
Here we clearly see that adopting AI actually
forces the company to formalize strategic choices. This involves both economic
and ethical considerations: What is our business plan? Is performance our only
horizon? What is our stance on corporate social responsibility? In an
unintended way, artificial intelligence can play a revealing role for the
company. It is therefore especially important for data scientists to explain
the issues upstream of the AI project.
The other safeguard essentially depends on the
user’s position. In an ultra-complex economic environment, where decision-makers
are bombarded with millions of contradictory data points, the data-driven business model is more attractive
than ever, and it is tempting to give the helm to an infallible machine which
will surely make the right decisions in our stead. This is a longstanding myth.
These proposals, and others,
can be seen in IBM’s Precision Regulation for Artificial Intelligence, published on 21st January 2020. It
sets our five proposals for eliminating bias outcomes from AI decision making,
such as the creation of a lead AI ethics official,
accountable for internal guidance and compliance mechanisms, such as an AI
Ethics Board. Also proposed are risk assessments based on application, end user
and level of automation, and great transparency around the purpose of AI
systems and audit trails surrounding their input and training data. It’s fifth
proposal is for testing throughout the entire life-cycle of the AI system:
pre-sale, deployment, and after it is operationalized.
Humans must be the safeguard for AI decision making
In fact, it has never been more crucial to
develop a reasonable approach centered on human instinct based on experience.
The purpose of AI is to provide optimal recommendations according to
pre-established criteria. But when the environment suddenly changes, the
system’s deductions become aberrant or discriminatory, especially when they are
based on biased data to begin with. Human analysis, and a sense of ethics and
professionalism are the only effective safeguards.
Just as with society at large, our businesses are at a crossroads with
regard to artificial intelligence. They hold all the keys to ensure that AI
keeps its great promise: augment human intelligence without encouraging its idiosyncrasies.
Former chess champion Garry Kasparov made an
interesting remark on this subject: “We can’t trivially design
machines that are more ethical than we are the way a programmer can create a
chess program that is far better at chess than they are. One key is to use them
to reveal our human biases so we can improve ourselves and our society in a
A naive vision? It’s up to us to decide.
About the Author(s)
You May Also Like