December 4, 2018 weblog
Pixel 3: A turn to machine learning for depth estimations
OK, we get it. In the ever-jostling vendor struggle to overpower the smartphone market, Pixel 3 is successfully marking its own turf as the phone with very good camera capabilities. And, now, a recent blog on the Google AI blog will only further delight Pixel 3 fans, as it details how "Portrait Mode" was achieved on Pixel 3.
Think of a team turning to machine learning, TensorFlow and depth-perception goals to the nines.
Scott Adam Gordon in Android Authority shared with readers what was rather clever about Google's approach and techniques. "With the Google Pixel 3's camera, Google included more depth cues to inform this blur effect for greater accuracy. As well as parallax, Google used sharpness as a depth indicator—more distant objects are less sharp than closer objects—and real-world object identification. For example, the camera could recognize a person's face in a scene, and work out how near or far it was based on its number of pixels relative to objects around it."
Rahul Garg, research scientist and Neal Wadhwa, software engineer, posted the blog. "This year, on the Pixel 3, we turn to machine learning to improve depth estimation to produce even better Portrait Mode results."
(Google thus far kept the magic to themselves, said Isaiah Mayersen in TechSpot. "As Google continues to pave the way in smartphone photography, we'll have to wait and see what competitors bring to the table.")
Wait, what is Portrait Mode? Webopedia, thank you:
Not only the single subject but more: "Not only will the person in the portrait be in focus, items near that same plane of focus will also be sharper, with a realistic increasing blur as items are further in front of and behind that plane," said Ryne Hager in Android Police.
Got that? Background intentionally out of focus; the subject carries the maximum impact in focus. This technique if often used in ads and promotional messages to draw maximum attention to a face by blurring the background.
Google needed some photos to train its AI. Enter as 5-phone clump—ok, more politely stated, an assemblage of five phones. They spoke about depth maps, training data and this assemblage, the "Frankenphone," a rig with 5 phones, all Pixel 3, and Wi-Fi based solution.
CNET described Frankenphone. "A quintet of phones sandwiched together generated the data to train the Pixel 3 to judge depth." Stephen Shankland described a "hacked-together clump of five phones to improve how the feature worked in this year's Pixel 3."
They used TensorFlow Lite, a cross-platform solution for running machine learning models on mobile and embedded devices, and the Pixel 3's GPU to compute depth quickly. Then, they combined resulting depth estimates with masks from their "person segmentation neural network to produce beautiful Portrait Mode results."
Why this matters does not require much analysis: better looking portrait mode shots.
In the bigger picture of picture-taking smartphones, vendors accept that their sales pitch had best include tempting camera features. Google's computational photography methods can give Pixel a real edge. "Smartphones have small image sensors that can't compete with traditional cameras for image quality, but Google is ahead of the pack with computational photography methods that can do things like blur backgrounds, increase resolution, tweak exposure, improve shadow details and shoot photos in the dark," said Stephen Shankland, CNET.
© 2018 Science X Network