Weapons of Math Destruction

Chapter 1. Bomb Parts: What Is a Model?

Racism, at the individual level, can be seen as a predictive model whirring away in billions of human minds around the world. It is built from faulty, incomplete, or generalized data. Whether it comes from experience or hearsay, the data indicates that certain types of people have behaved badly. That generates a binary prediction that all people of that race will behave in that same way.
Needless to say, racists don't spend a lot of time hunting down reliable data to train their twisted models.

So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage.

Chapter 5. Civilian Casualties: Justice in the Age of Big Data

In the 2000s, the kings of finance threw themselves a lavish party. They lied, they bet billions against their own customers, they committed fraud and paid off rating agencies. Enormous crimes were committed there, and the result devastated the global economy for the best part of five years. Millions of people lost their homes, jobs, and health care.

My point is that police make choices about where they direct their attention.
[...]
The result is that we criminalize poverty, believing all the while that our tools are not only scientific but fair.

Chapter 10. The Targeted Citizen: Civic Life

As this happens, it will become harder to access the political messages our neighbors are seeing—and as a result, to understand why they believe what they do, often passionately. [...]
The result of these subterranean campaigns is a dangerous imbalance. The political marketers maintain deep dossiers on us, feed us a trickle of information, and measure how we respond to it. But we're kept in the dark about what our neighbors are being fed. This resembles a common tactic used by business negotiators. They deal with different parties separately so that none of them knows what the other is hearing. This asymmetry of information prevents the various parties from joining forces—which is precisely the point of a democratic government.

Conclusion

Big data processes codify the past. They don't invent the future.

[Facebook]'s rigorous policy to tie users to their real names severly limits the the research outsiders can carry out there. The real-name policy is admirable in many ways, not least because it pushes users to be accountable for the messages they post. But Facebook must also be accountable to all of us—which means opening its platform to more data auditors.