What is foveated rendering? Explaining the VR technology key to lifelike realism

The first details of Sony's upcoming virtual reality headset for the PlayStation 5 game console leaked out this week. Among expected improvements like 4K resolution, USB-C and haptic feedback, we felt one feature needed further explanation.

That feature is called foveated rendering. Don't worry if you haven't heard of it before; this isn't a case of a pre-existing technology making the leap from smartphones, televisions or computers to VR, but a new technology directly related to virtual and augmented reality.

Read More:

• The new PlayStation 5 VR headset is claimed to have 4K, eye-tracking and more
• Ask these 11 questions before buying your first VR headset
• Sony keeps the faith with VR, announces new headset for PS5

What is foveated rendering?

At its simplest foveated rendering seeks to produce a more realistic viewing experience while also reducing the graphical demands of a VR headset. This can also help save battery life for wireless headsets, but the next-generation PlayStation VR is expected to be powered via a USB-C cable, so battery efficiency isn't a factor in that application.

Foveated rendering is able to both improve the visual experience and lower the processing requirements by adjusting the resolution of a VR environment depending on where you are looking at any one moment. This requires eye-tracking – another technology rumored to be featured on Sony's new headset – so the VR system can see precisely where the wearer is looking.

Once their gaze is known, the system can dynamically adjust the resolution of the VR experience in real-time. The resolution of the user's periphery vision is decreased, while the resolution of where they are looking is increased. This means graphical processing power can be managed more efficiently, with fewer pixels used to create parts of the VR environment that the user isn't looking at.

A more lifelike VR experience

As well as being more energy efficient, foveated rendering also promises to create a more lifelike VR experience, by blurring the edges of a user's peripheral vision instead of producing VR with a consistently sharp resolution across the entire environment. Just like how objects in your real-world peripheral vision aren't as clear as those you are looking at directly.

Foveated rendering was first seen in 2014, on a headset by a VR startup called Fove. This was also the first VR headset to feature eye-tracking. A couple of years later, Nvidia demonstrated its own method for creating foveated rendering, followed by Qualcomm in 2017.

HTC then revealed the Vive Pro Eye headset in January 2019, complete with eye-tracking and foveated rendering. This enterprise-grade VR headset works with Windows 10 computers and is currently priced at around $1,350 – considerably more than the expected circa $400 to $500 for Sony's upgraded PlayStation VR.

The feature also appears in the VR headsets of Varjo. Addressing its customers, the Finnish company explains the technology as such: "Foveated rendering uses the eye tracking functionality built into your headset to improve the performance of applications and produce a better virtual experience.

"With foveated rendering, your headset can render fewer pixels in peripheral areas while maintaining the highest resolution in the center of your vision. The area with the highest resolution will follow your gaze so that you can experience the best possible image quality at all times."

Foveated rendering without eye-tracking

This visual trick doesn't necessarily need eye-tracking technology. Facebook-owned Oculus added dynamic fixed foveated rendering to the software development kit (SDK) of the Oculus Quest in late-2019. Oculus explains to developers: "The effect lowers the fidelity of the scene in the viewer's peripheral vision and is nearly imperceptible. This reduces the GPU load due to the reduction in pixel shading requirements."

We don't know for sure yet if Sony's next VR headset will have eye-tracking and foveated rendering, but given the power of the PlayStation 5 we don't think this, and a 4K resolution, would be too much to ask. It should also lead to more realistic VR games and experiences, and hopefully pave the way for more growth in the consumer VR market.

HTC VIVE Pro Virtual Reality System