Meta Achieves Breakthrough: Full-Body Codec Avatars Now Renderable on Quest 3 with Near-PC Quality
MENLO PARK, CA – August 29, 2024 – Meta researchers have announced a critically important advancement in realistic avatar technology, successfully demonstrating the ability to render three full-body Codec Avatars at 72 frames per second on a meta Quest 3 headset – achieving performance comparable to PC-based rendering. This marks a crucial step toward bringing metas long-promised,high-fidelity avatars to consumers,particularly as competition heats up from Apple’s recently released Vision Pro with its “Personas” feature.
For over a decade, Meta (formerly Facebook) has been investing heavily in photorealistic avatar technology under the “Codec Avatars” project. The goal: to create digital representations of people that capture nuance and realism far beyond current cartoonish avatars common in virtual reality. While previous demonstrations required powerful PC setups, this new research, dubbed “SqueezeMe,” dramatically reduces the computational burden.
The breakthrough hinges on a distillation technique that allows for high-quality avatar rendering directly on the Quest 3’s mobile processor. According to a paper released alongside the announcement, SqueezeMe achieves this by optimizing the avatar data and rendering pipeline. The result is a visually compelling experience with minimal quality loss compared to PC-rendered versions.
Though, the current implementation comes with limitations.Unlike Meta’s latest PC-based Codec Avatars, these quest 3 avatars utilize flat lighting and lack dynamic relighting capabilities. Dynamic relighting is essential for avatars to realistically interact with changing light sources within VR and mixed reality environments, making them appear naturally integrated into the virtual space.Furthermore, the avatars are generated using a traditional, high-fidelity capture setup involving over 100 cameras and hundreds of lights. This contrasts with Meta’s more recent research focusing on smartphone-based scanning for avatar creation – a possibly more accessible method for widespread adoption. The capture system used for squeezeme is located at Meta’s Reality Labs Research facility in Redmond, Washington.
The Context: A Decade-Long Pursuit & Rising Competition
The development of Codec Avatars has been a central pillar of Meta’s vision for the metaverse – a future were immersive digital experiences blur the lines between the physical and virtual worlds. The company believes realistic avatars are key to fostering genuine social connection within these spaces.
The pressure on Meta to deliver on this promise has intensified this year with the launch of Apple’s Vision pro. Apple’s “Personas” feature, while not identical to Meta’s Codec Avatars, offers a compelling level of realism and personalization for users within visionOS. The Vision Pro utilizes a combination of sensors and machine learning to create avatars that reflect a user’s facial expressions and hand movements.
A significant hurdle for Meta is the lack of key hardware features in its current Quest lineup. Neither the Quest 3 nor the anticipated Quest 3S include eye tracking or face tracking – technologies crucial for capturing the subtle nuances of human expression and translating them into realistic avatar movements. The Quest Pro, which did feature these sensors, was discontinued in early 2024, leaving a gap in Meta’s hardware offerings.
What’s Next?
Meta may initially release a simplified version of Codec Avatars for use in 2D applications like WhatsApp and Messenger video calls. This could leverage AI-powered face tracking to simulate realistic expressions, even without dedicated hardware. This approach would allow Meta to showcase the technology and gather user feedback while addressing the hardware limitations of its current headsets.
Details are expected to be unveiled at Meta Connect 2025, scheduled for September 17-18. The company has teased that the event will reveal “tomorrow’s tech,” and further advancements in Codec Avatars are widely anticipated. Industry analysts, such as those at Moor Insights & Strategy, predict Meta will focus on software optimizations and AI-driven solutions to bridge the gap until more advanced hardware becomes available.
