Before we talked to Siri, we all wanted to meet her. From our living room couches, we watched her make gazpacho with Samuel L. Jackson, dance to “Shake, Rattle, and Roll” with Zooey Deschanel, and help John Malkovich through an existential crisis. Steve Jobs died just one day after Apple’s personal digital assistant was unveiled to the world, but he’d long before taught us how to hyperbolize her arrival. CNN called her “the stuff of science fiction.” Apple marketing head Phil Schiller said she’d be “amazing right out of the box.”
We should have known better. Siri, the marquee feature of Apple’s iPhone 4s, wasn’t quite as futuristic as she was made out to be. She was a bit hard of hearing. Even when she listened to our questions, she could provide an accurate answer only about two-thirds of the time. A surprising number of queries led to Google search results rather than a snappy, vocal response. If she were your actual, in-the-flesh assistant, you’d probably fire her. Some frustrated Apple customers even sued the company because they said Siri didn’t live up to the hype.
That was then, though, and Siri’s performance has improved considerably since her 2011 launch. She now has an industry-leading 5 percent word error rate, Apple claims. She handles 2 billion requests per week, double the figure from a year ago. And she has new features on the way, like Mac functionality and better integration with third-party apps.
But first impressions last, and Siri didn’t make a good one. Five years on, we’re left with the nagging feeling that Siri still hasn’t reached the potential we were promised. Apple has a knack for making its products feel essential just through its marketing. When the company declared “there’s an app for that,” it was a revelation that the iPhone carried infinite possibilities, and it turned out to be true. Siri’s ads, on the other hand, featured some telling fine print: “sequences shortened.” The Siri in those original commercials wasn’t real.
Apple did what it does best with the launch of Siri, inspiring wonder at the many possibilities of a new frontier of technology. But the digital assistant has gained some more able competition. Google has leveraged its insatiable hunger for data to offer an assistant, Google Now, that can predict what you want before you ask for it. Microsoft has developed Cortana into an assistant for the office by making it compatible with desktop computers. And Amazon, in an Apple-like move, gave us something we didn’t even know we wanted: Alexa, a smart-assistant for the home that doesn’t have a screen. Siri picked up some of these features, but they often arrived late.
The assistant’s slow evolution illustrates the challenges in bringing truly useful, natural human-computer interaction to the masses, especially at a place like Apple. The kind of company that created something as brilliant as the iPhone isn’t necessarily as well suited to win the artificial intelligence race; that requires vast amounts of user data and the share-and-share-alike ethos of academia. (The latter of which Apple isn’t particularly known for.)
“I think … Apple always wanted to be a consumer-electronics company,” says Pedro Domingos, a computer science professor at the University of Washington and author of The Master Algorithm. “But now I think the computing world is moving beyond that.”
Domingos knows Siri well. He worked on the wildly ambitious military project that birthed her.
In 2003, the Defense Advanced Research Projects Agency — or DARPA — launched a program to build an AI digital assistant that could be used as a military tool. DARPA wanted technology that would be able to help officers coordinate military tasks. First, though, it needed to help office workers battling email overload.
The project was dubbed “Cognitive Agent That Learns and Organizes,” or CALO, and it brought together top AI researchers from more than a dozen universities. Utilizing natural language processing and machine-learning techniques, CALO was envisioned as a highly personalized assistant that would learn the routines of its owner and help him complete daily tasks. After you showed CALO how to reserve a hotel room or order a book on Amazon, it would be able to follow that procedure itself via a typed-out command. The assistant was able to scan emails to organize meeting times and listen in on conference calls to highlight relevant action items to follow up on. CALO was a productivity tool first and foremost. It wasn’t assigned a gender, and it didn’t talk. “The idea was that CALO would be a personal assistant trained by you to do things the way you do them,” says Thomas Dietterich, the director of intelligent systems at Oregon State University who oversaw machine learning efforts on CALO.
The initiative was spearheaded by SRI International, a nonprofit research institute based in Menlo Park, California. Dag Kittlaus, SRI’s entrepreneur-in-residence at the time, immediately saw the commercial opportunities for the technology. As the project wound down in 2007, he partnered with CALO chief architect Adam Cheyer to license the military-backed tech and launch a startup. They called it Siri. CALO was a desktop assistant, but Kittlaus knew Siri was best suited for the newly released iPhone.
After years of fundraising and additional development, Siri debuted on the App Store in early 2010, with many features inspired by CALO. At a South by Southwest demo that year, the company showed how Siri, now controllable by voice, could book a table at a restaurant or summon a taxi by tapping into third-party APIs. It couldn’t yet speak, so it communicated through a chat interface. The range of queries it could address was purposefully limited to specific use cases, like weather, travel, and restaurants. It was meant to be an expert in executing specific tasks, not answer any random question that popped into your head.
Two months after the app launched, Apple acquired Siri for a reported $200 million. Steve Jobs reached out to the Siri founders personally to recruit them, inviting them to his home. “I believe that Steve was the biggest champion of the Siri acquisition at Apple,” Cheyer said in a 2015 interview. “He saw Siri as a transformative technology that could revolutionize and integrate every aspect of what Apple did.”
The app went dark for a year and a half, then reemerged as the flagship feature of the iPhone 4s in 2011. But Siri was different now. It — she — was gendered, with a robotic female voice. She didn’t play as nicely with friends — while the first version of Siri had been compatible with about 45 third-party services, including OpenTable and StubHub, the iPhone 4s version worked with fewer than six, according to Wired. And yet this walled-off version of Siri proclaimed to be more ambitious. She was not a mere task manager, at least according to Apple’s hype. She was a digital friend who should be able to field any type of request. “Your wish is its command,” the company’s marketing proclaimed.
Machine-learning researchers familiar with the technology knew that couldn’t be true. Many were excited to see an AI assistant getting full-throated support from a company as powerful as Apple, but they knew a truly smart digital assistant was still a long ways off. “The savvier machine learning and AI people, they always knew Siri wasn’t going to be that amazing thing that it was being sold as,” Domingos says. “We have 15 years of experience in this area now of people thinking that a problem is easy and making these grand claims and then it turns out to be harder. It was a positive thing, obviously, but the fact that it wasn’t everything it was cracked up to be, that wasn’t a huge surprise.”
Within a year of the iPhone 4s launch, two of Siri’s three founders left Apple (they’re now working on their own digital assistant). And very quickly, Apple found the field it had popularized swarmed with competitors. Google retooled its voice search feature into a direct Siri competitor in 2012. Then Microsoft and Amazon launched their own takes on the concept in 2014. Last year, Facebook began testing an assistant called M within Messenger app. And others, like SoundHound, are also crowding the space. For Apple, keeping Siri at pace with these younger, nimbler companies — ones that have arguably benefited from waiting for AI technology to mature instead of being first out the gate — may be tougher than the tech giant imagines.
It’s a well-known fact that design rules the day at Apple. The company’s industrial design and human interface design teams are its heartbeat, carefully crafting how Apple’s devices will feel in our hands and under our thumbs.
But there’s not much that chief design officer Jony Ive can do to “bring a calm and simplicity” when Siri botches a question. Increasingly, it’s the experts in machine learning and artificial intelligence who are playing a critical role in mapping out the future of the world’s biggest tech companies. This isn’t just about digital assistants, either. AI is already being used to organize our photos and answer our search engine queries; soon it will be used to whisk us away in driverless cars, as well.
This reordering of the occupational hierarchy creates challenges for Apple. The Cupertino company may be a dream job for designers, but things are more complicated for computer scientists, academics say.
“To do well in these areas like machine learning and computer vision and speech, these days the biggest obstacle is recruiting people,” Domingos says. “The talent is extremely scarce. If you’re a machine-learning person, would you rather go work for Google, where machine learning is the big deal, or go work for Apple, where in some sense you’re going to be subordinate to the designers?”
Apple and other tech giants recruit heavily from research universities to nab top computer science talent. But these PhDs hail from a world that encourages publishing research and learning from others in the field. Apple’s painstakingly secret R&D process can be a jarring transition. When Google and Microsoft are opening machine-learning research centers, working in the shadows of Cupertino might sound less appealing.
“I think for your average student, it’s very appealing to go to a company where you know that they’re doing this really exciting, cutting-edge work,” says Matt Gormley, a machine-learning assistant teaching professor at Carnegie Mellon University who interned at Google and Microsoft. “That’s definitely a big draw and I would say a big advantage that companies like Google and Microsoft Research have.”
Privacy presents another challenge for Apple. The company proudly claims to be dedicated to protecting user data and keeping sensitive info locked on an iPhone instead of floating in the cloud. But that approach makes it harder for Siri to compete with a service like Google Now, which absorbs as much info about users as possible to offer creepy-but-helpful suggestions on what to watch on TV or when to leave for work.
Still, Apple is getting more aggressive in its dedication to AI. In 2014, the company reportedly ditched speech-recognition technology from the popular vendor Nuance in favor of a homegrown solution that made use of deep learning, an en vogue type of machine learning that utilizes neural networks to mimic the functionality of the human brain. Last year, the company went on a hiring spree for AI experts — if you happen to be one, Apple still has nearly 100 machine-learning job openings today. They’re also embracing the academic community more. Apple served as a sponsor for the Neural Information Processing Systems Conference — one of the biggest gatherings of AI experts on the planet — for the first time in 2015.
But if Apple has a super-smart computer lounging in Tim Cook’s office that could trounce the world’s best Go player, we’ll likely never know about it. The company won’t divulge how many people it has dedicated to research on artificial intelligence, and it’s never published an academic paper in the area, according to Bloomberg.
It’s difficult to definitively proclaim a “best” digital assistant, Gormley says, because they all approach the concept differently and boast varying strengths. Siri has the most engaging personality, which Domingos says is an important factor in driving user adoption. Google’s assistant is the best at language processing and surfacing relevant info from its massive knowledge database. Amazon’s Echo has the sharpest “ears,” able to hear voice commands from across a room.
Despite Siri’s’ relative strengths, it appears she’s struggling to match her competitors. In the past year, The New York Times, TechCrunch, and CNN have all conducted head-to-heads of the major assistants, and Siri never came out on top.
This fall, though, Apple wants to change all that.
At the Worldwide Developers Conference, Apple made Siri a central focus of its upcoming operating systems. Her digital smarts are finally coming to the Mac, where she’ll be able to help users multitask or find long-lost files. (“It’s pretty epic,” Apple dad-in-residence Craig Federighi promises.) On the iPhone, she will be embedded in the keyboard through a new feature called QuickType — which will, among other things, scan text messages to help automatically schedule appointments (sounds like CALO, right?). And while not specifically Siri-driven, a new feature called Memories that can automatically recognize and sort photos using object recognition is being powered by Apple’s AI advances. The company also says it’s using a new statistical method called differential privacy to collect more user data for its services without compromising individual privacy.
Perhaps most critically, Apple is opening up Siri to more third-party developers. Soon you’ll be able to book a ride through Lyft or pay a friend with Square using voice commands. This will inch Siri closer to the original vision the app’s creators had, but it’s limited to specific categories and excludes popular apps such as Spotify.
To be sure, Apple is playing catch-up with these features. Ultimately, though, here’s Siri’s saving grace: All of the digital assistants on the market are still pretty dumb. They can now hear what we’re saying (most of the time), but they still have a difficult time acting on the information they receive. Three of the major issues: Assistants struggle to answer complex questions that require juggling different data sets (“What teams won the Super Bowl when George Bush was president?”), they’re terrible at remembering what you said even 60 seconds ago, and they still lack a broad enough knowledge base to understand all the minutiae of our daily lives.
“Siri is kind of like having a person you can ask one question and get one answer from. And if the [assistant] doesn’t answer your question, then they just turn it into a web search,” says Oregon State’s Dietterich. “There’s a lot of room to improve those systems.”
We know Google is tackling these issues, with research projects like Knowledge Vault, a deep-learning system that can autonomously scan the internet for information and add relevant facts to its “brain.” Apple probably is too, but it’s unlikely the company will ever let us see the way the sausage gets made the way Google does. We’ll have to wait until the next WWDC to see how the company’s research has been translated into user-friendly features.
No matter the future, Apple will always be the company that made digital assistants cool. In some ways, it’s a reversal of the company’s typical strategy of letting others pioneer a technology before offering a definitive product with improved design and slick marketing. That they were able to create a (sort of) usable, consumer-friendly product and introduce it to millions before anyone else is an impressive feat. “I think it was really important because somebody actually finally sat down and put in the proper amount of engineering to turn those kind of ideas into a working product,” says Alex Rudnicky, a research professor at Carnegie Mellon’s School of Computer Science. “I think that was kind of the real success of Siri.”
But staying the king (or queen, in this case) is tough. As long as our thumbs are still our primary method for interacting with the digital world, Apple’s empire will remain intact. But the moment we open our mouths, all bets are off.
“The way people are going to interact is going to be different,” Domingos says. “Being obsessed with the user experience, which is what they are, it’s as big an asset in this new world as it was in the old one. If they do this right, they could actually leapfrog the Googles and Microsofts and Amazons. But they have to combine that with this type of machine learning and AI.”
Is Apple ready for this new challenge? “I don’t think it’s going to be easy,” says Domingos, “largely because the culture of a company is hard to change.”
