Dangerous content and the pursuit of profit: Google and Meta under fire from abuse victims

From South Korea’s secret sex chats, to conflict in Ethiopia; online abuse survivors want more action

A group of South Korean journalists work overtime to expose a secret group targeting women and girls online.

They find eight group chats on the Telegram messaging platform.

Inside, there are thousands of videos of women and girls showcasing explicit non-consensual sexual content.

The videos are allegedly sold using cryptocurrency to avoid detection.

South Korean police would soon find over 60,000 people took part in these crimes by entering these so-called ‘rooms’, which has become known as the ‘Nth Room’ case.

In October 2021, one of the operators behind the Nth Rooms was sentenced to 42 years behind bars.

It is a small victory for law enforcement agencies who are in a constant war against these criminals, and the social media platforms they occur on.

But digital sex crimes continue around the world. In Australia, one in 10 people have reported someone posting nude, or semi-nude images online without permission.

Recent criminal cases also show perpetrators habitually threaten survivors with existing video content to force them into producing more sexually abusive content.

Jihyun Yoon is the director of Amnesty International Korea, who said technology companies are partly to blame.

“As a wave of digital sex crimes in South Korea causes severe harm to the women and girls who have been targeted, Google’s inadequate system for reporting non-consensual explicit content is making matters even worse.

“Google must do more to prevent the spread of online gender-based violence—not just in Korea, but everywhere,” she said.

In response to the Nth Room case, Amnesty International Korea carried out a survey of 25 survivors and activists.

Eleven said it was difficult to confirm whether their requests had been properly processed by Google.

“This was mainly due to a lack of communication from Google during the reporting process,” Jihyun Yoon said.

“Survivors around the world are forced to use this same flawed reporting system when they try to get harmful content removed, so it is highly likely this issue extends way beyond Korea.”

Jihyun Yoon, amnesty international

When users report sexually explicit content, they must tick a box saying they understand there are punishments if the submission is not true.

Google also refuses to process incomplete complaints or concerns.

One survivor, who has asked to remain anonymous, waited just over a year between receiving a confirmation receipt from Google and being informed of the outcome.

“I submitted it with difficulty, but rather than being convinced that it would be deleted, I became more anxious because I thought that if it didn’t work, it would be my responsibility,” they said.

What responsibility do social media companies have?

In Kenya, Facebook’s parent company, Meta was recently sued for its algorithms, which allegedly promote hatred online.

One Amnesty International staff member said they were targeted because of posts on the social media platform.

“I saw first-hand how the dynamics on Facebook harmed my own human rights work and hope this case will redress the imbalance,” said Fisseha Tekle, who is a legal advisor at Amnesty International.

Meta will answer to Kenya’s High Court over a landmark legal case. Amnesty International believes Facebook’s algorithms fuels ethnic conflict.

The legal action claims Meta promoted speech, which ultimately led to a string of ethnic violence and killings in Ethiopia.

Like many parts of the world, in Ethiopia, people often rely on social media for news and information.

But Amnesty International believes the platform’s algorithm prioritises and recommends hateful and violent content.

“Because of the hate and disinformation on Facebook, human rights defenders have also become targets of threats and vitriol,” Mr Tekle said.

Petitioners want to end Facebook’s algorithms from recommending such content.

In addition, they are seeking a create a US$1.6 billion victims’ fund.

Amnesty International’s deputy regional director of East Africa, Flavia Mwangovya, said dangerous content lies at the heart of Meta’s profit-making regime.

“From Ethiopia to Myanmar, Meta knew or should have known that its algorithmic systems were fuelling the spread of harmful content leading to serious real-world harms.”

“Meta has shown itself incapable to act to stem this tsunami of hate.”

Flavia Mwangovya, amnesty international

“Governments need to step up and enforce effective legislation to rein in the surveillance-based business models of tech companies,” she said.

What are governments doing?

In Australia, the e-Safety Commissioner issued legal notices to some of the biggest technology companies in the world last year.

It required them to report on measures to tackle the spread of child sexual exploitation material on their platforms and services.

“Some of the most harmful material online today involves the sexual exploitation of children and, frighteningly, this activity is no longer confined to hidden corners of the dark web but is prevalent on the mainstream platforms we and our children use every day,” said eSafety Commissioner Julie Inman Grant.

In Europe, the Netherlands once hosted 41 per cent of the world’s online child sexual abuse material. By March 2022, the figure had dropped to 13 per cent.

The Dutch Government made the removal of such content a priority. In 2020, it named and shamed internet hosting providers who failed to remove the material within 24 hours.

In South Korea, Google did not offer an official response to Amnesty International’s concerns.

But in a private meeting, the search engine technology reportedly said it wants to improve the way in which these concerns are managed.

However, Amnesty believes Google is failing to respect human rights.

“It must adopt a survivor-centered reporting system that prevents re-traumatization and is easy to access, navigate and check on,” Jihyun Yoon said.