The Rise of Synthetic Media & Digital Creators

This is Miko. She’s a virtual streamer. And with nearly 700K followers, she’s one of the most popular streamers on Twitch.

Miko is the creation of “The Technician”, a real-life developer who controls Miko with a motion capture body suit. The suit alone costs upwards of $30,000.

Miko’s creator—who only ever refers to herself as The Technician—painstakingly develops Miko using her mocap suit and the Unreal Engine. In a typical day, she wakes up at 2:00am to dev, devs until 12:00pm, and then streams from 12:00pm to 6:00pm.

The Technician’s career embodies the recent surge of the creator economy. At the beginning of the pandemic, The Technician was laid off from the animation studio she worked at, just weeks after moving to Los Angeles. Stuck with a $2,000-a-month lease, she remembers: “I thought, you know what would be the good thing to do right now isn’t to try to look for work. Let me put down 20 grand and try to make it on Twitch.”

The early days were slow-going. The Technician made $300 a month and was tens of thousands of dollars in debt from purchasing expensive equipment. But The Technician got creative: she allowed viewers to use bits, Twitch’s native currency, to interact directly with Miko. Viewers could spend money to make Miko dance or even (temporarily) die. They could use bits to tweak Miko’s appearance or change her surroundings. The dollars started to pour in. Since December, Miko has grown from 2,000 followers to 700,000 followers, and The Technician now employs a team of seven.

Miko embodies the rise of “VTubers” (Virtual YouTubers), a catchall term for online creators who digitally obscure their faces and bodies. But more broadly, Miko embodies a nascent, but fascinating space: synthetic media.

Synthetic Media & Deepfakes

Synthetic media is media content that’s generated or altered by artificial intelligence. Today, most people are familiar with synthetic media through deepfakes, a form of synthetic media in which a person in a video or image is replaced with someone else.

The term deepfake—a portmanteau of “deep learning” and “fake”—comes from the handle of a Reddit user who digitally swapped the faces of Scarlett Johansson and Gal Gadot onto the bodies of X-rated actresses. The early uses of deepfakes, unsurprisingly, were in pornography.

(Side note: when a new phenomenon comes to porn, that often means it’ll soon go mainstream. Porn has a long history of being the first stop for innovation. Much of today’s legislation around internet governance was a reaction to porn, with one Senator remarking in 1995, “If nothing is done now, the pornographers may become the primary beneficiary of the information revolution.” One recent survey ranks PornHub as the 3rd-most-influential company of the 21st Century, ahead of companies like Apple, Amazon, Microsoft, and Netflix.)

Predictably, deepfakes moved beyond porn and today ~80% of deepfakes are non-pornographic.

Source: Sentinel Deepfake Report

From 2019 to 2020, deepfake videos grew 6,820x (!), from 14,678 videos to 100 million videos. They began to invade mainstream internet platforms. Tom Cruise, for example, isn’t on TikTok. But you’d be forgiven for thinking he is: the account @deeptomcruise posts (very convincing) deepfakes of Tom Cruise. You can watch examples here and here.

Tom Cruise deepfakes from the TikTok account @deeptomcruise

Today, deepfakes—and, by extension, synthetic media—have a negative public perception: 9 out of 10 Americans believe deepfakes could cause more harm than good. Their concerns aren’t unfounded. Look no further than last fall’s deepfake of Joe Biden announcing that he has cancer, or the 2019 deepfake of Nancy Pelosi appearing to slur her words due to dementia. The Pelosi video racked up millions of views on Facebook after the platform refused to remove it. (Ironically, Facebook was also forced to keep up a deepfake video of Mark Zuckerberg saying “whoever controls the data, controls the future” in a dictator-like speech.) If deepfakes had been prevalent in 2016, it’s easy to imagine Donald Trump claiming that the Access Hollywood tape is simply a deepfake. Deepfakes could exacerbate “fake news” and permanently distort the line between fiction and reality.

Like all technologies, synthetic media isn’t moral or immoral; it’s amoral, and the right safeguards and regulations need to be put in place to ensure that the technology is used responsibly. But synthetic media’s seedy underbelly shouldn’t take away from its potential upside.

The Third Evolutionary Stage of Media

In its Synthetic Media Landscape report, Samsung Next argues that synthetic media is the third evolutionary stage of media:

The Past: Old Media. Enabled mass distribution for a select few through TV, radio, and print. Enabling Technology: Broadcasting

The Present: New Media. Enabled democratized distribution for everyone through social media. Enabling Technology: The Internet

The Future: Synthetic Media. Will democratize media creation and creativity for everyone. Enabling Technology: AI / Deep Learning

While more sinister use cases leap to mind, there are a variety of fascinating ways that synthetic media can unlock creativity and improve lives:

High-Quality Content Production at Low Costs

Technology has consistently brought down the costs to produce quality content. What used to be a million-dollar helicopter aerial shot can now be done with a GoPro and drone for under $1,000. Apple’s editing software costs $299 and has been used to edit Oscar-winning films. Creation is becoming more and more accessible.

But that’s largely solo creation—a YouTuber or TikTok creator posting videos. Synthetic media could let anyone produce content with an entire cast of actors. Imagine selecting virtual actors, customizing them, and integrating them into your work.

Interactive Media

The arc of content over the last 25 years has been from passive viewership to active participation. Films and TV gave way to YouTube and TikTok.

But imagine you and your friends literally participating in films and TV. With synthetic media, you could upload your friends’ faces and then all enjoy a thrilling adventure movie starring your friend group. Instead of watching The Avengers, your friend group could be The Avengers.

People will also adopt digital personas. Apps like Reface and Avatarify (below) let you swap faces in videos, GIFs, and memes.

Avatarify also released a free Windows application that lets anyone synthetically generate a 3rd-party guest for their Zoom meeting—for example, you can surprise your colleagues by joining the meeting as Elon Musk.

There are clearly ethical safeguards to put in place, but you can imagine a creator earning income every time their likeness is used in a deepfake.

Synthetic Video and Voice Creation

Synthetic media lets creators make changes to video and audio content. In the past, you’d need to go back and reshoot a scene or re-record a podcast. New tools let you synthetically edit and create media.

Descript, for instance, is an audio creation tool that lets you make changes to your voice track simply by editing a text script. It’s like magic: you type words into a Word-like document and hear your own voice say them.

Synthesia lets companies create videos in multiple languages for purposes like corporate training or CEO announcements. An English-speaking CEO can use AI to easily broadcast to different offices in Portuguese, Mandarin, Spanish, and so on.

Digital Characters

In 2019, KFC unveiled a digital Colonel Sanders. For some reason, they also made him a total thirst trap. As a Mashable headline put it, “KFC's ‘Virtual Influencer Colonel’ is pretty damn hot”.

Synthetic media allows brands to have lifelike digital mascots. Instead of an animated Ronald McDonald, customers could interact with a virtual recreation.

Digital characters will also populate stores: imagine a virtual person taking your order at a fast food drive-thru or answering your questions at the Apple Store. Artificial intelligence will automate rote tasks and jobs, often taking the form of a digital human. This may seem ominous and threatening. As I wrote in The Age of Automation, AI will ultimately be immensely positive, but will need to be paired with workforce development and lifelong learning.

Synthetic Advertising and Shopping

People who watch movies, TV, and video games in the future will see individualized, synthetic product placements tailored to their tastes and needs. Samsung NEXT gives an example:

Emily plays a game and sees a game character wearing a Nike sneaker. Her friend David plays the same game and sees the same game character wearing an Adidas sneaker, as David is more of an Adidas guy.

When shopping online, clothing brands will offer consumers the ability to see themselves and their bodies in the clothing.

Anonymous Personas

In the HBO documentary Welcome to Chechnya, the filmmakers used deepfakes to let survivors of LGBTQ+ persecution remain anonymous while recounting their stories. Volunteers supplied their faces, which overlaid the survivors’ faces as they spoke.

A deepfake overlaid a volunteer’s face onto an anonymous survivor of LGBTQ+ persecution in Chechnya for the HBO documentary Welcome to Chechnya

Being a creator today is by nature public, meaning that people who are less publicly-inclined don’t have access to the same avenues for creativity and self-expression. The VTuber trend is one answer to this: most VTubers remain anonymous, and Miko is an exception in sharing her real face.

Earlier this year, I wrote about the backlash to video: pandemic-fueled Zoom fatigue is leading people to look for new ways to interact online. One effect is the rise of avatar-based social networks. ItsMe, for example, lets you meet new people through an avatar that maps to your face.

ItsMe is an avatar-based social network in which your avatar makes to your face; Source

Synthetic media will let people use AI-driven avatars and deepfakes to decouple their private identities from their online expressions.

Digitally Rendering the Dead

This use case sounds dystopian, but bear with me. For 2016’s Rogue One, the filmmakers wanted to include the character of Grand Moff Tarkin from the original 1977 Star Wars film, but the actor died in 1994. They solved the problem by digitally rendering the character.

The real Peter Cushing in 1977’s A New Hope (left) vs. the digital rendering in 2016’s Rogue One (right); Source

Synthetic media makes it possible to recreate people’s faces and voices. It’s an eerie thought—people “coming back” from the dead—but it can have powerful use cases. The Illinois Holocaust Museum allows visitors to talk and interact with Holocaust survivors using holographic technology. One edtech company is using deepfakes to better teach students; in the CEO’s words:

“Our application of deepfake technology means pupils can have Stephen Hawking teaching physics or Shakespeare explaining his writing—this will be as revolutionary for education as the first blackboard was.”

The app MyHeritage brings photos to life, letting people animate old photos of deceased loved ones and ancestors. This has spurred a TikTok trend, including an emotional video of an elderly man seeing his deceased wife “come back to life” using synthetic media:

Rex Woodbury @rex_woodbury

This video is so moving and shows the power of synthetic media

7:48 PM ∙ Apr 27, 202127Likes4Retweets

Scalable Digital Creators

The supermodel Linda Evangelista once said that she doesn’t get out of bed for less than $10,000. But what if she could digitally license her image and make money while she sleeps?

What’s fascinating about synthetic media is that it can dramatically scale a creator’s likeness and, consequently, scale her earning power and creative output. The internet made creators more scalable by unlocking instant, global distribution. But this has had the added effect of pressuring creators to always be producing content. Ninja, the most popular streamer on Twitch, captures the problem:

Ninja @Ninja

Wanna know the struggles of streaming over other jobs? I left for less than 48 hours and lost 40,000 subscribers on twitch. I’ll be back today (Wednesday) grinding again.

12:09 PM ∙ Jun 13, 201865,240Likes3,686Retweets

The architecture of the modern creator economy sets creators up for exhaustion; it’s a never-ending hamster wheel. With synthetic media, creators can scale their output in a more sustainable and economically-rewarding way. It’s easy to imagine creators allowing their likeness to be used in videos, podcasts, and livestreams—and getting paid for each use.

While synthetic media can amplify real people’s creativity at greater scale, it can also generate digitally-native creators. This was Trevor McFedries’ vision for Brud. McFedries was moved by how media can be a vehicle for social change: he points to research showing Will & Grace was largely responsible for public support of same-sex marriage. McFedries wanted to capture that power at scale, building digital characters who can be in multiple places at once and speak multiple languages.

Brud’s most famous product is Miquela, a digital creator with 3 million followers on both Instagram and TikTok. Miquela can produce a theoretically infinite amount of content, unburdened by the real-world constraints of non-digital creators.

Brud’s digital creator Miquela

Following in Miquela’s virtual footsteps, Shudu is another digital creator and is known as the first digital supermodel. Shudu’s image went viral and was reposted by Fenty Beauty and Tyra Banks, who thought she was a real model.

Shudu’s artist, Cameron-James Wilson, is pushing even more boundaries for his next virtual creation: he’s collaborating with his Instagram followers, who can influence how the model will look.

Final Thoughts

Synthetic media will have a dark side. All new technologies do. An entire separate piece could be written on the ethical standards that need to be designed and implemented as safeguards for synthetic media.

But synthetic media will also usher in a new wave of creativity and art. There’s a robust ecosystem already forming:

Artificial intelligence will lead to greater experimentation with media. Today’s creators will produce at a greater scale—in terms of both quantity of content and quality of content—and a new class of synthetic-native creators like Miko will emerge.

Last week, I wrote about how new technologies enable new forms of art and creativity. Synthetic media is one of the most interesting and consequential new technologies, refined and improved and expanded every day. Like past technologies, synthetic media will determine new forms of creating, and those new forms of creating will determine the next generation of culture.

Sources & Additional Reading

Synthetic Media, Virtual HumansFrancesca Tabor