Skip to content

# Foveated rendering

The human eye “resolution” is not uniformly distributed in its field of view. Instead, its profile is such that the highest visual accuity occurs in the retinal area known as the fovea, and rapidly decreases towards the periphery. Game developers can take advantage of this by rendering at a higher image quality in the center of where the user is looking, while reducing the image quality in the periphery. This can increase framerates or reduce power consumption, at seemingly no cost to the user in terms of perceived image quality.

To enable foveated rendering, the Beam Eye Tracker SDK provides a stream of `FoveatedRenderingState` objects, each containing the real-time rendering profile to be applied, meaning, which area (position and radius) to render at the highest quality and how that visual fidelity should decrease towards the periphery, already mapped to screen space. This stream, in contrast to the UserState, has been optimized for this task by the Beam Eye Tracker to achieve the best user experience possible.

While foveated rendering is a generic concept and can be applied in multiple ways, one approach we here consider is Variable Rate Shading (VRS), which is explained in the next section.

## Variable rate shading (VRS)

Variable rate shading (VRS) allows to bundle pixels into “coarser” tiles to create a single shading call per tile, rather than a separate call for each pixel. This can drastically reduce the GPU load and thus increase the framerate. With the introduction of Tier 2 VRS, it is now possible to specify a different shading rate per tile on screen space, rather than a single global shading rate. This is done by creating a `Shading Rate Image` (SRI) that specifies the shading rate pattern across the screen.

Variable rate shading tiles.

This opened the door to many techniques, such as AMD’s FidelityFX Variable Shading, or Intel’s Velocity & Luminance Adaptive Rasterization (VALAR) algorithms which are able to create `SRIs` that maintains visual quality while reducing the GPU load. Given that an eye tracker is essentially indicating where a user is looking on screen, VRS is a natural fit for foveated rendering.

## Foveated VRS implementation

To implement foveated rendering with VRS, the process is as follows: at each rendering frame you can poll to retrieve the latest `TrackingStateSet` (see Polling data access). From this `TrackingStateSet` you can access the latest `FoveatedRenderingState` which holds the information necessary to generate the `SRI` for the current frame. Such generation typically is done by a custom shader. Once the `SRI` is generated, it can be assigned to the rendering pipeline. In DirectX12, you assign it using `ID3D12GraphicsCommandList5::RSSetShadingRateImage` and in Vulkan, you assign it by specifying it through `vkCmdSetFragmentShadingRateKHR`.

There is nuance when it comes to details such as shading rate image size, tile size, supported shading rates, etc. Thus, for a complete example, please refer to Foveated Rendering using FidelityFX™.

Foveated Shading Rate Image.

## The `FoveatedRenderingState` object

`FoveatedRenderingState` contains the information necessary to generate the `SRI` for the current frame. It holds the `foveation center` coordinates and the size of the shading rate regions, described by 4 radii held by a `FoveationRadii` type. However, it needs to be taken into account that some hardware supports only a maximum shading rate of `2x2`, instead of `4x4`. Therefore:

2x2 max shading rate

- `radius_level_1` describes the distance from the foveation center to the `1x1` shading rate boundary.

- `radius_level_2` describes the distance from the foveation center to the `2x1/1x2` shading rate boundary.

- everywhere outside `radius_level_2` the shading rate should be `2x2`.

4x4 max shading rate

- `radius_level_1` describes the distance from the foveation center to the `1x1` shading rate boundary.

- `radius_level_2` describes the distance from the foveation center to the `2x1/1x2` shading rate boundary.

- `radius_level_3` describes the distance from the foveation center to the `2x2` shading rate boundary.

- `radius_level_4` describes the distance from the foveation center to the `2x4/4x2` shading rate boundary.

- everywhere outside `radius_level_4` the shading rate should be `4x4`.

For directional tiles regions (`2x1`, `1x2`, `2x4`, `4x2`), we don’t have a strong recommendation. Thus you can either chose to use the same direction for the given region, or chose a different orientation based on the vector from the given tile to the foveation center.

Warning

The `FoveatedRenderingState` data is generated from the Beam Eye Tracker application rather than at the client library level, and it is optimized to consider variables such as screen size, multiple displays, eye tracking accuracy, user experience, etc. You can indeed use your own criteria to compute the sizes for each of the regions. However, we strongly recommend that you use at least the `foveation center` coordinates and the `radius_level_1` parameters as intended, and that the other regions sizes are computed based on such size reference.

## Foveated VRS compatibility with DLSS, XeSS, FSR…

Foveated VRS can be combined with super resolution techniques, such as FidelityFX Super Resolution or DLSS as it can be applied to the lower/base resolution prior to the upscaling step delivering compounded performance gains. The combination of VALAR with XeSS, as described in this article, provides a good real-world example of such combination.

## Combining with other VRS techniques

As described previously, other techniques have been proposed to create `SRIs` that maintain visual quality while reducing the GPU load. Foveated rendering is not replacement, but rather a complement to these techniques, as a combined shading rate image can be obtained by using the maximum shading rate for each tile between the foveated VRS `SRI` and the other `SRIs`. For performance reasons, this can be done in the same shader, as it is the case for the Foveated Rendering using FidelityFX™ integration.

## User experience considerations

To achieve a good user experience, it’s critical to consider the following:

- The `TrackingStateSet` should be polled as late as possible, just prior to `SRI` generation/dispatch, so that it uses the most recent eye tracking data.

- The `SRI` generation/dispatch should be delayed as much as possible in the rendering pipeline, just before the GPU starts shading pixels/fragments.

- Webcam frame rates and integration time can have an impact on latency, and thus user experience.

Dynamic HUDs
Plugins and samples