Skip to content

# Dynamic HUDs

With the Beam Eye Tracker `API` you can get, out-of-the-box, an estimate in real-time of whether the user is looking at static HUD elements in the screen-space, such as a minimap, speedometer, etc. This opens the door to enhance gaming experiences. Examples of use cases are:

- Immersive HUDs, which automatically fade out when the user is not looking at them, but immediately reappear when the user glances at them.

- Reactive HUDs, which are then zoomed-in, popped up, focused, make a sound, etc. based on whether the user is looking at them.

An example of an Immersive HUD. The minimap is only visible when the user is looking at it, letting the user focus on the game scene, boosting the overall immersive experience.

This page explains how to implement such functionalities leveraging convenient `API` functionality.

The integration process consists of these steps:

1. Keep the viewport geometry updated;

2. Estimate if the user is looking at a HUD element;

3. Make the HUD element(s) react;

4. Optionally, Add in-game settings

We will now describe each of these steps in detail.

## Keep the viewport geometry updated

Make sure that the viewport geometry is correctly set at `API` initialization and updated during gameplay. This is explained in the Viewport section. Once this is achieved, the gaze coordinates mapped to the viewport coordinate system will be readily available as part of the `UserState` data structure, as explained in the Real-time tracking section.

## Estimate if the user is looking at a HUD element

The Beam Eye Tracker `API` provides ready-to-use signals that indicate whether the user is looking at pre-defined HUD regions, but you can also implement this manually. Both approaches are here explained.

### Ready to use “looking at HUD” signals

The ready to use signals are based on pre-defined HUD regions as seen in the image below

To access the respective signals you have to first retrieve the `GameImmersiveHUDState` data structure from the stream of `TrackingStateSet` objects (see Data Access Methods). Then, the `GameImmersiveHUDState` contains a float value per point/region (for example `looking_at_viewport_top_left` ) that should be interpreted as the likelihood (value between 0 and 1) of the user looking at that region. Typically you can use a threshold of `0.5` to decide whether the user is looking at a region or not as a boolean value.

Note that more than one point/region can be looked by the user at the same time, based on the current implementation.

Note

The current mechanism that the Beam Eye Tracker uses to determine if the user is looking at a region is based on the normalized distance between the gaze coordinates and the point/region, whereas each point is at distance of `0.1` from the viewport border (i.e. `10%` of the width or height of the viewport). If the distance is below `0.3` then we say that the user is looking at that region. Note that this is rather large margin.

However, this algorithm is subject to change in future releases that could account for improvements in the eye tracking technology and is a further reason why we recommend to use the provided signals instead of a manual implementation.

A priori, our implementation philosophy is to prefer false positives over false negatives, as it is more annoying to have a HUD element not react when the user is looking directly at it than to have a HUD element react when the user is looking nearby but not directly at it, specially if the interaction is for the HUD element to disappear.

### Manual implementation of “looking at HUD” signals

The manual implementation is based on the `UserState` data structure, which contains the gaze coordinates in the viewport coordinate system `UserState::viewport_gaze`.

Then, in your game engine, you can define screen-space objects in the position of HUD elements that you want to detect and add a simple collision detection algorithm to return a boolean value indicating if the gaze coordinates are within the bounds of said object.

Important

It is critical to consider that webcam eye tracking can be innacurate for some setups, which gets even worse in complex scenarios such as gaming in multiple monitors. Thus, when creating the collision objects please either make them much larger than the HUD elements or use generous margins to account for such innacuracies.

We argue that having false positives is preferrable to having false negatives, in use cases such as immersive HUDs (which disappear when the user is not looking at them), as it could be frustrating to the user if the HUD element is not reappearing despite the user looking at it.

Alternatively, you can simply rely on the Ready to use “looking at HUD” signals whose behavior is intended to evolve and improve as the Beam Eye Tracker itself evolves.

## Make the HUD element(s) react

Now that you have a signal indicating if the user is looking at a HUD element or not, you can implement the logic to make the HUD element react. Clearly, this is specific to the use case you are implementing.

Assuming your current goal is to implement an Immersive HUD, in which HUD elements are only rendered when the user is looking at them, you can implement an algorithm in which:

- If the user is looking at the HUD element, you make it fully visible quickly by setting the opacity to 100% (e.g. over 50ms);

- If the user is not looking at the HUD element, you slowly fade it out by slowly decreasing the opacity to a minimum value such as 30% (e.g. over 1 second).

An example implementation is detailed in the Game Engine Integration sample.

## Add in-game settings

In is not necessary to expose any in-game settings to the user. However, if you want to, you can expose the following settings

### Device selection

Same Device selection as for the In-game camera control feature.

### ON/OFF Toggle

You can let the user activate or deactivate the feature.

### Sensitivity/Range Sliders

To account for complex setups, including multiple monitors or a variety of webcams, you can customize the margins used to determine if the user is looking at a HUD element or not if you used a Manual implementation of “looking at HUD” signals impleemntation.

## Non-static HUD elements

This page focused on eye-tracking interactions with static HUD elements in the screen-space, such as a minimap, speedometer, etc. The same principles apply to dynamic objects (which change position or size during gameplay) following the Manual implementation of “looking at HUD” signals mechanics, but you need to carefully take into account the recommendations explained in that section including those related to innacuracies of webcam eye tracking.

In-game camera control
Foveated rendering