Exclusive | ‘Camera Control is akin to building a ship in a miniature bottle’

Apple executives Richard Dinh, Johnnie Manzari and Piyush Pratik, detail the vision for Camera Control physical interaction introduced with the iPhone 16 series

In September, when Apple trained the spotlight on the iPhone 16 series and introduced another set of physical controls on the phone, it caught many by surprise. Camera Control however isn’t just limited to photography or video recording on an iPhone, though that obviously constitutes a major part to its existence. Camera Control, as the iOS 18.2 update released this Thursday illustrates, it is an integral part of the Visual Intelligence element of Apple’s artificial intelligence suite too.

Camera Control is an integral part of the Visual Intelligence element of Apple’s artificial intelligence suite too. (Apple)

“The new camera control in the iPhone 16 lineup packs so much innovation into this really small space. For us, it was like building a ship inside a miniature bottle,” says Piyush Pratik, who manages iPhone Product Marketing at Apple, in an exclusive conversation with HT. This isn’t hyperbole. Richard Dinh, who is VP of iPhone Product Design explains the layers of hardware that make up the Camera Control physical interface on an iPhone and says it is “unlike anything we’ve previously done”.

The layering begins with a high precision, tight tolerance tactile switch that enables the click experience to quickly launch a camera app and takes photos or videos. This will work, even if the user has wet hands or is wearing gloves. Secondly, a high precision force sensor that detects input from differing touch force from a user’s finger, using micron scale resolution—this will enable the light brush gesture, that is crucial to navigating Camera Control’s quick settings. The next layer includes a multi pixel capacitive sensor that’s embedded directly behind the sapphire crystal that detects the touch interactions as a user will slide their finger on the sensor, to make adjustments or switch modes.

I remember meeting with Pratik immediately after the iPhone keynote in September, and he had shared a trick to bypass the potential Camera Control learning curve. Never lift your finger between the light taps, harder taps and swipes, for seamless navigating through the camera options, he had told me. That advice, something I’ve shared with many an iPhone 16 user since, helps get attuned quicker.

As Dinh points out, the multi-touch co-processor in the A18 and A18 Pro (these chips power the iPhone 16 line-up), dedicated silicon for the force and capacitive sensor and their interaction with the Taptic Engine come together for precise touch and haptic feedback. “Camera control has algorithms that also utilises what we call sensor fusion, which is looking at sensors across other systems to identify and resolve when a click was unintentional. So, when an iPhone 16 is in your pocket, or lying face down, or when you have game mode activated, it will just know to ignore those accidental clicks,” he explains.

The choreography of innovation

Johnnie Manzari, who is the Human Interface Designer at Apple, tells us that “these features don’t begin as these fully resolved ideas. They begin as questions, and there’s a lot of innovation and discovery that happens along the way.” It was a similar journey for Camera Control, as different teams within Apple including the design studio and the camera architecture team, built prototypes. “One of the fundamental experience questions that we’re trying to address was really around speed and simplicity, and we wanted to make it as fast and easy as possible to launch or capture photos and videos,” Manzari explains.

The key was mapped to a click to launch the camera, a subsequent click to capture a photo and a click that’s held for video recording. “We didn’t want that speed to come with a sacrifice of quality,” he adds. The question then was, how to move along the spectrum of a 4:3 aspect ratio photo to a 6:9 aspect ratio video also bringing in Dolby Vision HDR, and the interface challenges that come with it.

“How do you do that choreography in a way that feels smooth, seamless and considered? No one team can just sort of go off and do it all on their own. And we’re also able to apply our most advanced engineering in the service of creating something that’s really simple and intuitive. As we were prototyping, we realised that we have this notion of intent, and what I mean by that is, if someone’s resting the finger on the button and exerting a little bit of pressure, we can use that as a signal,” Manzari adds.

We asked Pratik about the feedback from iPhone 16 users, based on their experience with Camera Control thus far. He points to the new iOS 18.2 update adding the two-step shutter functionality, as a derivative of what they’ve heard from users. “The inputs we get, we are constantly pouring them into our process to make sure it only gets better over time,” he says.

Camera Control, which follows Apple’s introduction of the Action Button a generation earlier, may feel at odds with broader perception of minimalism. That is, letting go of physical buttons. “Is the goal to have no physical buttons and to just be everything through multi touch? Unsurprisingly, just based on what we have done, Apple doesn’t think that’s true and we don’t think the goal is to eliminate all buttons, says Manzari. Often, he says, the best solution to a problem comes when experts work on the process of discovery.

iOS 18.2, and building for the future

Part of the iOS 18.2 update is the strengthening of the Apple Intelligence suite, with the addition of Visual Intelligence that builds on the iPhone’s camera to generate environmental context for users. OpenAI’s ChatGPT integration arrives in this update too, as part of Siri and Visual Intelligence. One part of that suite is a user invoking the functionality by pressing and holding the Camera Control key to invoke the AI mode. The camera then, pointed at a shop, restaurant or establishment for instance, would provide contextual information overlays such as business hours, menus and so on.

Does Apple believe this is a learning curve for users as they retain their habits towards a different way of searching for information? Pratik believes that may be reasonably so.

“I think with any new technology or new experience, it’s reasonable to believe that people might take some time, but that’s been the whole thought and intent behind designing Camera Control and designing the visual intelligence experience so. It has to be intuitive and easy to use. We must make sure that we are able to give users that intelligence in a way that’s faster and more seamless. It is like that true North Star,” he says.

“This is first time that we’ve integrated them so deeply into the OS. As a user, you just want answers, understand the world around you. You don’t necessarily have to think out how you use visual intelligence, simply use camera control,” he adds. This leads the conversation to the question of building the data set on a global level, to cover most if not all of the important physical locations. Thats what the users would expect. “It is a huge problem to solve, technologically and from a user experience perspective. We’re taking this head on,” says Pratik, but the company doesn’t give any specific timelines for global coverage.

For now, Visual Intelligence is leveraging a combination of Apple Intelligence and OpenAI’s GPT models to provide data and inference for searches.

We simply had to ask the Apple executives a question many have mentioned—can the Camera Control, at some stage, double as a Touch ID sensor to allow unlocking the phone via fingerprint scanning? It may not be possible, at least with this generation. “The button has a multi-sensor but it isn’t the resolution you would need for touch,” clarifies Dinh.