Following the keynote speech from Apple's Apple Worldwide Developers Conference (WWDC) 2020, the company spelled out in detail this afternoon ARKit 4, the latest version of the Augmented Reality (AR) App Development Kit for iOS devices. It is available in beta and introduces a depth API that offers a new way to access depth information on the iPad Pro. Location anchoring – another new feature – uses data from Apple Maps to place AR experiences at geographical points in iPhone and iPad apps. Face tracking of photos and videos is now supported on every device with the Apple Neural Engine and a front-facing camera.
According to Apple, the depth API uses the functions integrated in the lidar scanner of the iPad Pro 2020 to understand the scene in order to collect pixel-by-pixel information about an environment. Combined with 3D mesh data, the API makes the occlusion of virtual objects more realistic by enabling immediate placement of digital objects and seamlessly integrating them into their physical environment.
The anchorage of ARKit 4 supports the placement of AR experiences in cities, next to famous sights and elsewhere. More specifically, developers can anchor AR creations at specific latitude, longitude, and height coordinates so that users can move around in virtual objects and view them from different perspectives.
With advanced face tracking, it works on any iOS smartphone or tablet with A12 Bionic chip and higher, including iPhone X, iPhone XS, iPhone XS Max, iPhone XR, iPad Pro and iPhone SE. According to Apple, up to three faces can be tracked at the same time.
In addition to these enhancements, ARKit 4 comes with motion detection, which allows iOS apps to understand body position and motion as a series of joints and bones, and use motion and poses as inputs for AR experiences. It is now possible to capture face and world tracking simultaneously with the front and rear cameras of the devices and work with multiple people to create an AR world map. Every object, surface and character can display video textures. Partly thanks to machine learning enhancements, apps created with ARKit 4 can recognize up to 100 images simultaneously and get an automatic estimate of the physical size of the object in an image, allowing better detection in complex environments.
In the previous version of ARKit – ARKit 3.5, which was released in March – a new Scene Geometry API was added, which uses the lidar scanner of the iPad Pro 2020 to create a 3D map of a room in which between floors, Walls, ceilings, windows, doors and and seats. The ability to quickly measure lengths, widths, and depths of objects from a distance of up to five meters has been introduced, so that users can create digital facsimiles that can be used for object exclusion (ie, digital objects appear mixed in scenes behind real objects ). .
ARKit 3.5 has also been improved in the Motion Detection and Locking Department, with a better depth estimate for people and a better height estimate for motion detection. On the iPad Pro 2020, the lidar scanner enables more precise three-axis measurements and offers advantages to existing apps without the need for code changes.