You are now a number in China
The Chinese government plans to rank its people according to their trustworthiness as determined by their activities on social media
It sounds like the script to an episode of cult Netflix satire Black Mirror: a world where each citizen is identified by an algorithm linked to their social media profile. Armed with innermost knowledge, the algorithm ends up controlling what schools, housing, jobs and travel visas an individual can secure.
This may lack the dark humour of the first episode, where a princess is kidnapped and a hapless prime minister is ordered to have televised sex with a pig in order to secure her release. Yet in the real world, the Facebook data of 50 million people has allegedly been used by Cambridge Analytica to help Donald Trump become US President, Russian hackers are said to be in the midst of a widespread attack on crucial components of American infrastructure, including its electric grid, nuclear power system, water, aviation and manufacturing.
A person’s score will also be affected by what their online friends say and do, beyond their own contact with them. If someone they are connected to online posts a negative comment, their score will also be dragged down
And China is rolling out a programme to rate its 1.3 billion people, ranking their behaviour with ‘citizen scores’ of their trustworthiness. Outlined in a dry 2014 Chinese government policy document entitled Planning Outline for the Construction of a Social Credit System, the programme takes the technological possibilities of social media into a Big Brother environment.
According to Rachel Botsman, author of Who Can We Trust: How Technology Brought Us Together and Why It Could Drive Us Apart, it’s a ‘world where many of your daily activities are constantly monitored and evaluated: what you buy at the shops and online, where you are at any given time, who your friends are, how you interact with them and what bills and taxes you do or don’t pay.’
‘The Chinese government is pitching the system as a desirable way to measure and enhance ‘trust’ nationwide and to build a culture of ‘sincerity’,’ she states. ‘A person’s score will also be affected by what their online friends say and do, beyond their own contact with them. If someone they are connected to online posts a negative comment, their score will also be dragged down.’
The system is being rolled out in China under a ‘watch and learn’ approach, with eight companies granted licences to develop systems and algorithms for social credit scores. It is set to become mandatory in China in 2020 and the government has announced that a ‘warning and punishment mechanism’ will apply to people guilty of ‘trust-breaking’.
Citizens with low scores can expect to suffer slower Internet speeds, restricted access to restaurants and clubs and the loss of rights to travel freely abroad, while a 2016 document also warns of difficulties in being hired by certain employers and enrolling children into elite private schools. Star performers can attain a range of benefits.
Sesame Credit, one of the eight programmes under the scheme, offers ‘special’ privileges including loans, deposit-free car rental, the use of VIP lounges at airports and fasttracking for travel visas. Operated by an affiliate company of Chinese Internet giant Alibaba, it has already signed up millions of affluent and aspiring Chinese citizens. According to Roger Creemers, a Chinese governance specialist at Leiden University in The Netherlands, the Sesame programme is ‘a sort of bastard love child of a loyalty scheme’.
Think about your Uber rating, views on YouTube, Likes on Facebook or followers on Twitter. These are all ranking metrics so the concept of being ranked by some number is already alive and kicking here in the West
Botsman sees it becoming even more popular and expects the emergence of ‘reputation black markets’ that sell ‘under-the-counter’ ways of boosting trustworthiness’ and allow citizens to pay to manipulate their score in the same way that Facebook Likes and Twitter followers can be bought.
‘The new system reflects a cunning paradigm shift,’ she says. ‘Instead of trying to enforce stability or conformity with a big stick and a good dose of top-down fear, the government is attempting to make obedience feel like gaming. It’s a method of social control dressed up in some points-reward system. It’s a gamified obedience.’
Henry Timms, co-founder of global philanthropic movement Giving Tuesday, agrees, warning that the Social Credit System is creating a world where the government could know far more about its citizens than even George Orwell imagined in 1984. Together with the controversy emerging from Cambridge Analytica’s use of Facebook data he fears that the supposed transparent new world ushered in by the Internet 20 years ago is becoming corrupted and controlled.
‘This was supposed to be an era of democratisation and openness,’ he opines. ‘But in some ways the two strongest forces are turning out to be platforms and strongman governments,’ says Timms. ‘You have the hidden hand of the platform and the heavy hand of the strongman and you see those dynamics playing out around the world, from the tensions around Facebook and what it can do to elections to a resurgence of strongman governments that seem to want to take us back to the 20th Century.
‘We’re already seeing that networked models like Airbnb are proving very capable of building trust. So the question isn’t whether networked models are able to create trust; the question is: trust in what?’
Could China’s model be replicated in the West? It already is, according to Rupert Younger, director of the Oxford University Centre for Corporate Reputation, who says the main differences between it and Western corporate systems are China’s state ownership and what it might be prepared to do with social media data
‘Think about your Uber rating, views on YouTube, Likes on Facebook or followers on Twitter,’ he says. ‘These are all ranking metrics so the concept of being ranked by some number is already alive and kicking here in the West. The difference is that China is creating a government-centralised version of it and that version fits with its idea of central command and control. ‘In the West, the market does exactly the same things, though with market mechanisms, rather than state ones. China has just gone a stage further by creating an algorithmic single social credit score.’
Some social media experts believe the impact of the China experiment can be exaggerated. It needs to be detached from the current Facebook controversy, argues Drew Benvie, founder of social media agency Battenhall, since Facebook, along with Twitter and Pinterest, has long been banned in China. Neither, he says, should it be a surprise to see China limiting individual rights of expression and exerting strong state control.
‘Social media in China has never been the same as everywhere else in the world,’ he states. ‘The biggest social networks everywhere else in the world are banned there and China has its own social media ecosystem, where services such as Sina Weibo and WeChat have become enormous.
‘Social media in China is very sophisticated and very different from that in the West. It’s normal, for example, to find a barcode for WeChat printed on products in shops so customers can interact with the product on social media. So it’s not surprising that the Chinese see an opportunity to monitor social media activity and to do something with that information.’
Benvie believes the key issue is the scale of information-sharing by social media sites. ‘The question is how much social media sites know about their users and who they share this information with,’ he says. ‘Consumer understanding of this has been poor and what we’re going to see in China is a bit of a global test bed for what people are sharing with companies, governance bodies and others and what they are doing with that information.
‘People have assumed that what they put on Facebook or share in text messages or phone calls is between them and the people they’re communicating with. But in reality, this information can be used by bad actors.’
Katie Buckett, co-founder of social media consultancy One Fifty Consultancy, agrees, arguing that China’s experiment, though extreme, actually highlights possibilities for how social media activity can be used to indicate levels of trustworthiness, positions in a community and responsibility. ‘People spend a lot of time on social networks and they are a good indication of peoples’ behaviours and what they might to likely to do as a result,’ she says.
Are we heading for a future where we will all be branded online and data-mined?
‘Certain things can be inferred from that information as to how trustworthy a person may or may not be. I’m not sure we’re going to see the level that this is reaching in China anywhere else in the world but you might see more specialised local applications. Insurers already use telematics systems in policyholders’ cars to determine behaviour and risk, So is there something around looking at people’s social standing and trustworthiness and what their behaviours are likely to be in order to give them access to or discounts on certain things?
‘If you’re well-networked, it’s a good indicator that you’re socially responsible and considered to be more trustworthy than perhaps some others because people are Liking you and interacting with you. However, there then has to be a different conversation around what you do with that information and how you use it.’
Botsman is more pessimistic. ‘Are we heading for a future where we will all be branded online and data-mined?’ she asks. ‘It’s certainly trending that way. ‘Barring some kind of mass citizen revolt to wrench back privacy, we’re entering an age where an individual’s actions will be judged by standards they can’t control and where that judgment can’t be erased. The consequences are not only troubling; they’re permanent… What will happen when these systems, charting the social, moral and financial history of an entire population, come into full force? How much further will privacy and freedom of speech be eroded?’
That remains the key question and Timms also feels that the exploitation of social networks for commercial gain and government control is approaching a tipping point. ‘There’s a trade-off here between liberty and user experience and what liberties people are prepared to give up for a frictionless experience,’ he says. ‘We’re getting to a stage where he who mobilises best will win. If you can make it easy, attractive and rewarding for people to do things, they will increasingly do them and the people who are making it easy right now aren’t necessarily on the side of the angels.
‘China’s Social Credit System is part of a very troubling silent but steady erosion of a bunch of liberties. If we give them up slowly in favour of a more frictionless life then I think that’s very problematic.’
As for Black Mirror, the first episode ends with the PM giving in to the blackmail and having sex with the pig, not realising the princess has already been released. The kidnapper commits suicide, while a year later the PM’s public approval ratings have fully recovered, yet his marriage is in tatters. The recently-commissioned fifth series is going to have a veritable feast of new international plotlines.
But this new China syndrome is fact, not fiction. How the world responds will shape the future not only of the Internet but of governance, societies and individual and corporate freedoms. Tread very carefully
