Artificial intelligence
We tend to think of machines, especially smart machines, as somehow cold, calculating, and impartial. We believe that self-driving cars have no preference in life-or-death decisions between a driver and a random pedestrian. We trust intelligent systems that perform credit assessments to ignore everything except the truly influential metrics like income and FICO scores. And we know that learning systems always converge toward the ground truth because unbiased algorithms guide them. For some of us, this is a drawback: machines should not be empathetic outside of their rigid self-perception. For others, it is a feature: they should be free from human bias. But somewhere in between, there is the view that they will be objective. Of course, nothing could be further from the truth. The reality is that not only are very few intelligent systems truly impartial, but there are also multiple sources of bias. These sources include the data we use to train systems, our interactions with them in “nature,” emergence bias, similarity bias, and conflicting goals bias. Most of these sources go unnoticed. But as we build and deploy intelligent systems, understanding them is critical so we can design with awareness and hopefully avoid potential problems.
Data-driven bias:
For any system that learns, the output is determined by the data it receives. This is not a new insight, it is just something that is often forgotten when looking at systems that are trained with millions of real examples. The idea has been that the sheer volume of examples would overwhelm any human bias. But if the training set itself is skewed, so will the output. More recently, this type of bias has surfaced in image recognition systems using deep learning. Nikon’s confusion over Asian faces and HP’s skin tone issues in their facial recognition software appear to be the product of learning from skewed example sets. While both are fixable and completely unintended, they illustrate the problems that arise from not paying attention to the bias in our data.
In addition to facial recognition, there are other troubling examples with real-world implications. Learning systems used to build sets of rules applied to predict recidivism rates for probationers, crime patterns, or potential employees are areas of potential negative consequences. When they are trained using skewed data, or even data that is balanced but the systems are biased in their decision-making, they also perpetuate bias.
Bias through interaction:
While some systems learn largely by looking at a set of examples, other types of systems learn through interaction. Bias is built on the biases of users that drive the interaction. A clear example of this bias was Microsoft’s Tay, a Twitter-based chatbot designed to learn from its interactions with users. Unfortunately, Tay was influenced by a community of users who taught Tay to be racist and misogynistic. In fact, the community repeatedly tweeted offensive remarks at Tay, and the system used these remarks as raw material for subsequent responses. Tay only lasted 24 hours and was shut down by Microsoft after it became an aggressive racist.
While Tay’s racist rants were limited to Twitter, they do illustrate the potential real-world consequences. As we build intelligent systems that make decisions with and learn from human partners, the same kind of problem of bad training can arise in more problematic situations. What if we instead partnered intelligent systems with people who would guide them over time? Consider our distrust of machines to decide who gets a loan or even who gets foreclosed. What Tay taught us is that such systems learn the biases of their environment and people, for better or worse, and reflect the opinions of the people who train them.
Emergent Bias:
Sometimes, decisions made by systems that are designed to be personalized end up creating “bubbles” of bias around us. We can look at the current state of Facebook to see this bias in action. At the top layer, Facebook users see their friends’ posts and can share information with them. Unfortunately, any algorithm that uses feed analysis to serve up other content will serve up content that fits the set of ideas the user has already seen. This effect is amplified as users open, like, and share the content. The result is a stream of information that is skewed toward the user’s existing set of beliefs.
While this is certainly personalized and often reassuring, it is no longer what we tend to think of as news. This bubble of information is the algorithmic version of “confirmation bias.” Users don’t have to protect themselves from information that conflicts with their beliefs because the system does it for them automatically.
The impact of these information biases on the world of news is worrying. But as we look to social media models as a way to support decision-making in organizations, systems that support the emergence of information bubbles have the potential to skew our thinking. The knowledge worker who receives information only from people who think like him will never see opposing views and will tend to ignore and deny alternatives.
Similarity Bias:
Sometimes bias is simply a product of systems doing their job. For example, Google News is designed to present stories that match a user’s searches and are a collection of related stories. That’s exactly what it’s designed to do, and it does it very well. Of course, the result is a collection of similar stories that tend to confirm and corroborate each other. That is, they define a bubble of information that resembles the personalization bubble associated with Facebook.
There are certainly problems with the role of news and its dissemination that are highlighted by this model – the most obvious of which is the balanced approach to information. There is a lack of “editorial control” in a wide range of situations.
