A dormant chip in the Pixel 2 will soon let developers write better camera and AI apps

Here’s a surprise: Google’s Pixel 2 phones include a custom system-on-a-chip (SoC) that’s optimized for image processing — but it currently just sits there, doing nothing.

Google says it’ll turn this chip on in the coming weeks as a developer option in the preview of Android Oreo 8.1. This will enable developers to include the same HDR+ image processing that allows Google’s camera app to produce great pictures with hardware that is, at least on the spec sheet, not up to par with that of its competitors.

This custom system-on-a-chip (SoC) marks Google’s first foray into custom chips in a consumer product. This chip, the “Pixel Visual Core,” features eight custom-designed image processing unit (IPU) cores and 512 arithmetic logic units. This allows the Pixel’s camera to shoot images that use the company’s HDR+ algorithm for a wider color spectrum (by quickly taking and combining multiple images at different exposure levels) with none of the delay you’d typically expect for HDR images. Google says that using the Pixel Visual Core speeds up HDR+ processing by 5x, all while using only a tenth of the energy of running that same algorithm on a regular CPU.

Google notes that the IPU uses two domain-specific languages: Halide for image processing and — no surprise there — TensorFlow for machine learning applications.

It’s pretty curious, though, that right now, Google says this chip just quietly sits on the phone, doing nothing. It’s not even being used by Google’s own camera app. As a Google spokesperson told me, the company has “managed to achieve the HDR+ through tight coupling of hardware, software and ML.” There must be some speed and power advantages to using it even in Google’s own applications, though, so my best guess is that the chip was a very late addition to the phone’s hardware — so late that Google wasn’t even able to work with third-party developers to write public demo apps and tout it in its launch keynote.

For now, the focus here is squarely on allowing developers to use HDR+ in their applications, but given that this is a programmable chip, it’s no surprise that Google plans to open it up to other use cases over time, too. “HDR+ is the first application to run on Pixel Visual Core,” Google notes in today’s announcement. “As noted above, Pixel Visual Core is programmable and we are already preparing the next set of applications. The great thing is that as we follow up with more, new applications on Pixel Visual Core, Pixel 2 will continue to improve.”