A Chinese-Style Digital Dystopia Isn’t As Far Away As We Think

Chinese journalist Liu Hu always knew he’d have trouble with the authorities; he had been exposing corruption and wrongdoing for years. He was used to being hassled with regular fines and forced apologies imposed by his authoritarian government. He nevertheless persisted in truth-telling.

One day in 2017, Hu logged onto a travel site, but couldn’t book a flight because the site said he was “not qualified.” Soon he discovered he was blocked from buying property, using the high-speed train network, or getting a loan. And there was nothing he could do about it. His rights to essential goods and services were now circumscribed through an algorithm designed to discriminate against the 7.5 million people on China's “Dishonest Persons Subject to Enforcement” list.

Welcome to the Chinese “social credit score” system, whose goal is to rank China’s 1.4 billion people. Conceptually, it is not that different from a financial credit score in the US. But the social credit score includes things like political outspokenness, shopping habits, friends, travel habits, and anything the authorities want to encourage or discourage. This score then fine-tunes your access to essential social goods based on a discriminatory algorithm.

Such a nightmarish system could never, of course, happen in the United States. Or could it? Three recent decisions in Washington suggest it is not as far-fetched as we might imagine, with both our courts and our government effectively endorsing the way a handful of giant companies are centralizing control over our society.

First, the Republican-controlled FCC abandoned rules that prevented internet providers from discriminating between different forms of data flowing through their networks. The result is that interfering with the flow of information, rather than facilitating it, will become a lucrative new business model for telecom giants like AT&T and Comcast.

Days later, a federal judge rejected the government’s attempt to block AT&T’s takeover of Time Warner, a giant media conglomerate that owns TV channels like CNN and HBO. The government argued that the combined company would be too powerful, and able to discriminate against rivals. The judge disagreed. He essentially said Google, Facebook, and Amazon have now become so powerful that the only way AT&T can compete is to bulk up its own power in a marketplace of goliaths. Within days of that decision, a bidding war between Comcast and Disney broke out over 21st Century Fox.

The reason these companies want to merge the pipelines of the internet with the content that flows through them is simple: They want to build detailed individual profiles of internet users — how they browse the web, what they watch. If there’s any doubt that’s the reason, consider the judge’s own words in allowing the AT&T deal. The merger allows AT&T to imitate “highly successful, data-driven entities” like Google, he said. To underscore the point, this week AT&T bought yet another company, AppNexus, which was one of the largest remaining digital advertising marketplaces not already controlled by a Silicon Valley giant.

The hits keep on coming: On Monday, the Supreme Court, in a case involving credit cards, issued a decision that will effectively immunize tech platforms and other networked systems against antitrust scrutiny.

All these decisions are dangerous on their own. Together they show how centralized control of the internet could enable the possibility of automated discrimination on the level of the individual citizen. Which is exactly the power we are seeing deployed in China, even if the politics are different.

To understand how frightening this power is, it’s important to first understand the concept of “first degree” price discrimination. First-degree price discrimination is when every individual gets a separate price based on their unique characteristics. We accept price discrimination all the time; going to the movies and getting a senior discount is price discrimination. But in that case, the decision of how to discriminate is done by class; it is publicly posted; and everyone accepts that, in this case, seniors get a discount. It is a public decision to discriminate.

Discriminating on an individual level is different and allows for powerful exploitation and manipulation of the citizen. In areas with first-degree price discrimination, like car insurance or credit cards, there are often gender- or race-based pricing choices. With increasing datafication of society, we can see this increasingly organized to the level of the individual.

An airline could, for instance, analyze your email for the words “death in the family” and “travel,” look at your credit limit, and then offer you a price based on this information. Or imagine a group of companies putting together a common list of troublemakers, perhaps negative online reviewers or commenters or consumers who frequently return items. All of a sudden, for no obvious reason, someone who returns an item to one store might find that prices on a host of socially essentially goods have done up.

Corporations generally deny they do anything like this or even that they can. But Uber recently let the cat out of the bag when it admitted that it does attempt at first degree price discrimination. Rather than charging based on the route, or demand for drivers, Uber is using “machine learning” to charge what it thinks individual customers are willing to pay. If there’s any doubt that this is the plan for AT&T, let’s turn to Judge Richard J. Leon’s decision, which he said that this new entity can use “customer data to inform their strategy and improve the customer’s experience in a number of ways.” This is code for discrimination. And the Supreme Court just made it clear it will be very hard to bring suits if new platform giants use this power to dominate markets.

All of this is being driven by the web giants — Amazon, Google, and Facebook — who use their power over shopping, search, and social networking to discriminate in favor of their own products and content. Amazon has built a massive manufacturing business by privileging its own private-label products, like Amazon batteries, over rival brands sold on its site. Google does the same thing, using its heft over search and maps to privilege its own products and services.

What these decisions collectively mean is becoming clearer — soon, all information production and distribution will increasingly be created for the profit of the web giants, or it will not be created and distributed at all. Media executives are now saying that within a few years it is likely that every major content company will be owned by a tech or telecom platform.

We are now in a totally unregulated world of lawless web giants who operate as the core infrastructure for our society. They can use their data and power to discriminate and exploit, and the strategy now for companies like AT&T is to emulate them, or die. And the deep links that intelligence agencies have with these giants suggest this power can, with a flip of a few switches, be easily weaponized by the state.

By endorsing the business model of total surveillance and discrimination, the Trump FCC, the Supreme Court, and Leon have made a deeply reckless decision about what kind of society we will have. A few goliath corporations in America may soon govern what we can see, say, do, and hear. It is not hard to see a social credit score–like system emerging in the United States. After all, Facebook, Google, Amazon, and now AT&T are in a race to effectively build one.