WebXR DOM Overlays Module

Draft Community Group Report,

This version:
https://immersive-web.github.io/dom-overlays/
Issue Tracking:
GitHub
Editors:
(Google)
Participate:
File an issue (open issues)
Mailing list archive
W3C’s #immersive-web IRC
Unstable API

The API represented in this document is under development and may change at any time.

For additional context on the use of this API please reference the WebXR DOM Overlays Module Explainer.


Abstract

The WebXR Augmented Reality module expands the WebXR Device API with the functionality available on AR hardware.

Status of this document

1. Introduction

This module describes a mechanism for showing interactive 2D web content during an immersive WebXR session. When the feature is enabled, the user agent will display the content of a single DOM element as a transparent-background 2D rectangle.

1.1. Overview

While the DOM overlay is active, the UA enables user interactions with the DOM overlay’s content using platform-appropriate mechanisms. For example, when using XR controllers, the primary action dispatches DOM pointer events and click events at the location where the controller’s pointing ray intersects the DOM overlay.

A new beforexrselect event provides a way to suppress XR input events for specific regions of the DOM overlay and helps applications distinguish DOM UI interactions from XR world interactions.

2. HTML API Integration

This module adds a new event type to the definition of GlobalEventHandlers.

2.1. onbeforexrselect

An XRSessionEvent of type beforexrselect is dispatched on the DOM overlay element before generating a WebXR selectstart input event if the input source’s targetRaySpace intersects the DOM overlay element at the time the input device’s primary action is triggered.

partial interface mixin GlobalEventHandlers {
  attribute EventHandler onbeforexrselect;
};

This event is an XRSessionEvent with type beforexrselect that bubbles, is cancelable, and is composed. Its target element is the topmost event target being intersected by the targetRaySpace and is either a descendant of the DOM overlay element or the DOM overlay element itself.

Cancelling this event by calling preventDefault() suppresses default WebXR input events that would normally be generated by the input source for this primary action. The selectstart, selectend, and select events will not be fired for this interaction sequence.

Note: Future WebXR modules MAY define additional events or WebXR input dependent data that are affected by cancelling this event, for example suppressing results from a transient input source’s hit test subscription.

This event and the actions taken by the event handler have no effect on DOM event processing, and are not synchronized with DOM event dispatch. The user’s action will separately generate appropriate DOM events such as `"pointerdown"`, and those DOM events can happen before or after the corresponding beforexrselect event. This happens regardless of whether the beforexrselect event was cancelled or not, and is independent of any further actions taken in XR input event handlers.

Note: This event provides a way for applications to suppress duplicate XR input events while the user is interacting with a DOM UI. Since this is a bubbling event, the application can register handlers on appropriate container elements, effectively marking regions of the DOM overlay as blocking XR input. This is independent of the visual opacity of DOM elements. It is possible to show noninteractive opaque or translucent DOM content such as text explanations that don’t block XR input events.

The following code installs an event handler on an interactive part of the DOM overlay to selectively suppress XR events for that region, while continuing to generate XR events for other parts of the DOM overlay that are treated as transparent for interaction purposes.
document.getElementById('button-container').addEventListener(
  'beforexrselect', ev => ev.preventDefault());

2.2. CSS pseudo-class

The :xr-overlay pseudo-class MUST match the document’s root element for the duration of an immersive session using a DOM Overlay.

2.3. User-agent level style sheet defaults

The user-agent style sheet defaults for the DOM Overlay element are as follows:

:xr-overlay {
    /* force a transparent background */
    background: rgba(0,0,0,0) !important;

    /* the following styling is identical to :fullscreen */
    position: fixed !important;
    top: 0 !important;
    right: 0 !important;
    bottom: 0 !important;
    left: 0 !important;
    margin: 0 !important;
    box-sizing: border-box !important;
    min-width: 0 !important;
    max-width: none !important;
    min-height: 0 !important;
    max-height: none !important;
    width: 100% !important;
    height: 100% !important;
    transform: none !important;

    /* intentionally not !important */
    object-fit: contain;
}

/* If there’s a backdrop from fullscreen mode, set that transparent. */
:xr-overlay:fullscreen::backdrop {
  background: rgba(0,0,0,0) !important;
}

NOTE: This is based on Fullscreen API §5.4 User-agent level style sheet defaults, with additional styling to make the overlay element transparent. A user agent MAY use the Fullscreen API to implement the DOM Overlay, and in that case its backdrop must also be transparent. The styling for :xr-overlay does not explicitly depend on the Fullscreen API’s pseudoclass or styling so that user agents have the flexibility to implement it independently of the Fullscreen API.

NOTE: Applications are encouraged to use the :xr-overlay pseudo-class for conditionally styling UI elements during the session, including controlling visibility of interface elements.

2.4. Fullscreen API integration

The UA MAY implement DOM Overlay as a special case of the [FULLSCREEN] API. In this case, the UA MUST prevent changes to the active fullscreen element, rejecting requestFullscreen requests for the duration of the immersive session.

NOTE: The DOM Overlay API requires specifying the overlay element at session start, and does not provide a mechanism to change the active overlay element during the session. Applications would behave inconsistently across platforms if they could use the Fullscreen API to indirectly change the active overlay element.

3. WebXR Device API Integration

This module expands the definitions of XRSessionInit and XRSession, and modifies the behavior of XRInputSource events.

3.1. XRSessionInit

This module introduces the string dom-overlay as a new valid feature descriptor for use in the requiredFeatures or optionalFeatures sequences for immersive sessions.

A device is capable of supporting the DOM overlay feature if it provides a way for the user to see and interact with DOM content for the duration of the immersive session.

NOTE: Implementation choices include a fullscreen overlay on a handheld AR device, or a floating rectangle in space for a VR or AR headset.

The DOM content MUST be composited as if it were the topmost content layer. It MUST NOT be occluded by content from the XRWebGLLayer or by images from a passthrough camera for an AR device. Applications can use normal CSS rules to control transparency and 2D placement of content within the DOM overlay itself.

The DOM overlay MUST be automatically visible to the user from the start of the session, without requiring the user to press buttons or take other manual actions to make it visible.

NOTE: A device should not claim to support a DOM overlay if the content element is only indirectly visible, for example if the user would need to take off their headset or manually enable a passthrough camera to view content on a separate 2D monitor that’s not normally visible during the session. However, an immersive CAVE system where a user is carrying a physical touchscreen device showing the DOM overlay content would be a valid implementation.

The XRSessionInit dictionary is expanded by adding a new domOverlay member. This is an optional member of XRSessionInit, but it MUST be specified when using the DOM overlay feature since there is no default overlay element.

partial dictionary XRSessionInit {
  XRDOMOverlayInit? domOverlay;
};

If the DOM overlay feature is a required feature but the application did not supply a domOverlay member, the UA MUST treat this as an unresolved required feature and reject the requestSession() promise with a NotSupportedError. If it was requested as an optional feature, the UA MUST ignore the feature request and not enable a DOM overlay.

NOTE: The UA MAY emit local warnings such as developer console messages explaining why the DOM overlay was not enabled.

3.2. XRSession

This module extends the XRSession interface to add a new readonly attribute which reflects the current state of the active DOM overlay element.

partial interface XRSession {
  readonly attribute XRDOMOverlayState? domOverlayState;
};

The domOverlayState attribute MUST be null if the dom-overlay feature is not supported or not enabled.

If the feature is enabled, the attribute value MUST be present.

NOTE: Applications can check the presence domOverlayState to verify that the DOM overlay feature is enabled and working, for example if it was requested as an optional feature.

NOTE: The DOM overlay may be temporarily invisible to the user, for example if the user agent places it at a fixed orientation or location where it may end up outside the user’s field of view after user movement. The domOverlayState attribute still remains set while this is happening.

While the session is active with a visible DOM overlay, the UA MUST treat this as rendering opportunity and execute Window requestAnimationFrame() callbacks at a rate suitable for animating DOM content. These MAY run at different times and frequencies than requestAnimationFrame() callbacks as used for drawing XRWebGLLayer content.

3.3. XRInputSource

When an XRInputSource begins the platform-specific action corresponding as its primary action the UA MUST run the following steps before starting input processing to decide if this is treated as a primary action:

  1. If the input source’s targetRaySpace intersects the DOM overlay at the time the input device’s primary action is triggered:

    1. Queue a task to fire an XRSessionEvent named beforexrselect on the topmost event target within the DOM overlay root being intersected by the targetRaySpace, setting target to that element. This events bubbles, is cancelable, and is composed.

    2. Check how XR input should be handled as follows:

      If the event was cancelled
      1. If the input source is a transient input source, treat this as an auxiliary action. Otherwise, ignore this action for the purpose of generating XR input events.

      Otherwise
      Treat the action as a primary action as usual for the input source.

NOTE: Effectively, cancelling the beforexrselect event suppresses XR input select events, none of selectstart, selectend, or select are generated for this action. For transient input sources, inputsourceschange events are still generated, but cancelling the beforexrselect event causes the action to be treated as an auxiliary action, similar to a secondary finger input.

4. Initialization

The application MUST provide configuration for the DOM overlay through the domOverlay dictionary.

dictionary XRDOMOverlayInit {
  required Element root;
};

The root attribute specifies the DOM element that will be displayed to the user as the content of the DOM overlay. This is a required attribute, there is no default.

The following code requests DOM overlay as an optional feature.
let uiElement = document.getElementById('ui');
navigator.xr.requestSession('immersive-ar', {
    optionalFeatures: ['dom-overlay'],
    domOverlay: { root: uiElement } }).then((session) => {
    // session.domOverlayState.type is now set if supported,
    // or is null if the feature is not supported.
  }
}

While active, the DOM overlay element is automatically resized to fill the dimensions of the UA-provided DOM overlay rectangle. Its background color is automatically styled as transparent for the duration of the session.

NOTE: A UA MAY use the Fullscreen API §5.4 User-agent level style sheet defaults to style the DOM overlay element, with an additional rule containing background-color: rgba(0,0,0,0) !important; to set the background transparent.

Once the session is active, the domOverlayState attribute provides information about the DOM overlay.

enum XRDOMOverlayType {
  "screen",
  "floating",
  "head-locked"
};

dictionary XRDOMOverlayState {
  XRDOMOverlayType type; 
};

NOTE: From the user’s point of view, a "floating" overlay is perceived as stationary when rendered as if anchored to a real-world location, and this style is a common choice for interactive display surfaces in VR. A "head-locked" overlay moves along with head rotations and does not have a fixed real-world location.

NOTE: Future versions of this spec may add additional attributes to the overlay state, for example the current location in world space for a floating overlay.

5. Event handling for cross-origin content

The user agent MUST NOT provide poses or gamepad input state for user interactions with cross-origin content such as an HTMLIFrameElement nested within the DOM overlay element.

A user agent MAY meet this requirement by preventing user interactions with cross-origin content, for example by blocking DOM events that would normally be received by that content, or by not loading and displaying cross-origin content at all.

If the user agent supports interactions with cross-origin content in the DOM overlay, and if an input source’s targetRaySpace intersects cross-origin content as the topmost event target, the UA MUST enable WebXR Device API §13.4.3 Limiting data adjustment, and populate the pose for XRSpaces associated with that input source accordingly treating the limit boolean as true. In addition, the UA MUST NOT update gamepad data for this input source while poses are limited.

NOTE: The application does not receive pose updates for the controller or its targeting ray while poses are limited in this way. The UA is responsible for drawing a pointer ray or other appropriate visualization as needed to enable interactions.

NOTE: the restriction on updating gamepad data is intended to avoid information leakage from interactions with the cross-origin content to the application. For example, if an input source’s primary action uses an analog trigger where the primary action happens at a certain trigger threshold, the application could infer when the user started and ended a primary action from the trigger value even if the corresponding events are blocked. Another example would be a device that uses a trackpad or joystick for text input in DOM content, where reading the axis values would allow the application to infer what text was being entered.

If a primary action ends inside cross-origin content, the UA MUST treat the primary action as cancelled, and MUST NOT send a select event. The UA MUST send the selectend event using the last available pose before entering the cross-origin content due to treating poses as limited.

If the input is a transient input source, and if the transient action begins inside cross-origin content, the user agent MUST delay adding the input source until the input location moves out of the cross-origin content. If the transient action ends while still inside the cross-origin content, the transient input source does not get added at all.

NOTE: On a handheld AR device using screen-mode input, this means that touches that stay inside cross-origin content don’t create an input source or associated XR input events. If a drag movement starts inside cross-origin content, the input source is created at the location where the touch location leaves the cross-origin content, emitting a cancelable beforexrselect event as usual.

6. Security, Privacy, and Comfort Considerations

6.1. Protected functionality

The DOM overlay does not in itself introduce any new sensitive information. However, since it combines existing technologies, it’s important to ensure that this combination does not lead to any unexpected interactions.

A primary design goal for this module was that the DOM overlay should follow existing semantics for 2D content where possible. Specifically, the information flows related to cross-origin embedded content should be similar to using iframes on a 2D page. For example, a 2D page can embed cross-origin content in an iframe, and then cover this iframe with a transparent element. In that case, the page will continue to receive mouse movement information, but the cross-origin content does not receive any input events in the covered area. For a DOM overlay, XR input event data is treated as similar to mouse movement data. Poses remain available to the outer page if there is no cross-origin content, or if the cross-origin content is not receiving input, but are limited (blocked) when interacting with cross-origin content.

Cross-origin content is potentially vulnerable to clickjacking threats. The UA MUST continue to apply mitigations such as Content Security Policy §6.1.5 frame-src when iframes are used in a DOM overlay. The UA MAY implement additional restrictions specifically for cross-origin content in a DOM overlay if necessary to address specific threats.

7. Acknowledgements

The following individuals have contributed to the design of the WebXR DOM Overlay specification:

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[DOM]
Anne van Kesteren. DOM Standard. Living Standard. URL: https://dom.spec.whatwg.org/
[FULLSCREEN]
Philip Jägenstedt. Fullscreen API Standard. Living Standard. URL: https://fullscreen.spec.whatwg.org/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[SELECTORS-4]
Elika Etemad; Tab Atkins Jr.. Selectors Level 4. 21 November 2018. WD. URL: https://www.w3.org/TR/selectors-4/
[UIEVENTS]
Gary Kacmarcik; Travis Leithead; Doug Schepers. UI Events. 30 May 2019. WD. URL: https://www.w3.org/TR/uievents/
[WebIDL]
Boris Zbarsky. Web IDL. 15 December 2016. ED. URL: https://heycam.github.io/webidl/
[WEBXR]
Brandon Jones; Nell Waliczek. WebXR Device API. 10 October 2019. WD. URL: https://www.w3.org/TR/webxr/

IDL Index

partial interface mixin GlobalEventHandlers {
  attribute EventHandler onbeforexrselect;
};

partial dictionary XRSessionInit {
  XRDOMOverlayInit? domOverlay;
};

partial interface XRSession {
  readonly attribute XRDOMOverlayState? domOverlayState;
};

dictionary XRDOMOverlayInit {
  required Element root;
};

enum XRDOMOverlayType {
  "screen",
  "floating",
  "head-locked"
};

dictionary XRDOMOverlayState {
  XRDOMOverlayType type; 
};