The First Selfie
In 1839, an amateur photographer named Robert Cornelius set up a primitive camera in the back of his family’s silversmith shop. He sat in front of the lens, slightly right of center, his hair just a tiny bit mussed, and snapped his photo. After printing his portrait, he inscribed on the back: “The first light picture ever taken.”
Today, self-portraits taken with a camera go by a different name–the selfie—and they’re arguably the most pervasive media in the digital realm. More than an estimated one million selfies are captured and shared every day. According to a recent study, 30% of the photos taken by people between the ages 18 and 24 are selfies. Yet for all their ubiquity, each selfie is as unique as the person who takes it; and even for that specific person, no two selfies are exactly alike. Inspired by this global expression of individuality and a deep appreciation for yet another medium of expression – music – The Lincoln Motor Company wanted to make the selfie do something it had never done before: sing.
“Selfies are not one-to-one. They’re all different, depending on what kind of mood you’re in, the face you’re making, or where you are,” says Heather Phenix, a producer at Jam3, the digital design agency that helped Lincoln bring the Music Selfie Experiment to life. “We wanted to drive that idea towards a more musical interpretation.
The idea was simple: users would upload or take a photo of themselves on a site where it would be scanned with facial detection technology. Plot points would be distributed at key intersections on the face and, using an algorithm, a musical sound would be assigned to each point. The technology would combine each sound, and the end result would be a unique song that only that user’s face could produce.
“It’s such a simple idea,” says Michael Dobbel, a producer at Jam3. “It just needed some technology and great craft to make it sing.”
To read the subtle variations on users’ faces, Jam3 utilized hypersensitive technology that works in two basic steps. First, it determines the location of the face in each photo. Then, using microscopic measurements, Music Selfie determines precisely where each feature is, plotting specific ratios and distances.
“In order for us to create unique pieces for each user, we need to know the exact size of their eyes, their nose, the position of their mouth—as much detail as possible,” says Mikko Haapoja, a creative director at Jam3. “For this, you basically need a feature detection library versus a traditional facial recognition library.”
But, as anyone who’s taken a selfie knows, getting the perfect shot is not always easy. To control for factors like lighting, Jam3 made sure that the technology was dynamic, allowing users to actually see where the plot points were placed and edit them as needed.
Once the technology was in place, next came the music. A distinct sound was needed for each of the 50+ plot points on each user’s face, a feat in itself. But to create personalized, multi-layered songs that could come together uniquely to match the calculations of every face submitted, each sound needed to harmonize with the others– in any possible combination. Enter Plan8, a music house from Stockholm, that took on the colossal task of composing for the project.
The team started small, composing a set of what they called “themes,” measures of music with similar tempos, key and harmonic structures. Once they were satisfied with the simplest arrangements, the fun began.
“For each of those themes, we basically started having fun in the studio, improvising over the back track,” says Calle Stenqvist, Plan8’s founder and creative director. “And we did that enough until we had variations that we could feed back into the algorithm.”
To ensure the highly technical experience produced music that retained a warm, human sound, the Plan8 team used vintage instruments and amps to create the harmonies, coming up with enough combinations that, once they were all fed into the algorithm, had the potential to create more than four million unique songs! Calle and his team spent hours listening to their combinations, but have yet to even scratch the surface of all the possibilities.
“This project represents the perfect synergy between traditional craftsmanship and computer technology,” says Calle. “There’s no way a musician or a composer can compose and record 4 million tracks, and there’s no way a computer could record something that needs the real touch of the human hand.”