WebXR Lighting Estimation API Level 1

Editor’s Draft,

More details about this document
This version:
https://immersive-web.github.io/lighting-estimation/
Latest published version:
https://www.w3.org/TR/webxr-lighting-estimation-1/
Feedback:
GitHub
Editor:
(Google)
Former Editor:
([Mozilla until 2020])
Participate:
File an issue (open issues)
Mailing list archive
W3C’s #immersive-web IRC

Abstract

This specification describes support for exposing estimates of environmental lighting conditions to WebXR sessions.

Status of this document

This section describes the status of this document at the time of its publication. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document was published by the Immersive Web Working Group as an Editors' Draft. This document is intended to become a W3C Recommendation. Feedback and comments on this specification are welcome. Please use Github issues. Discussions may also be found in the public-immersive-web-wg@w3.org archives.

Publication as an Editors' Draft does not imply endorsement by W3C and its Members. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 1 August 2017 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 2 November 2021 W3C Process Document.

1. Introduction

The WebXR Lighting Estimation module expands the WebXR Device API, the WebXR Augmented Reality Module, and the WebXR Layers module with the ability to expose estimates of the lighting conditions of the user’s environment.

2. Light Primitives

2.1. XRLightProbe

XRLightProbe

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

An XRLightProbe collects estimated lighting information at a given point in the user’s environment.

[SecureContext, Exposed=Window]
interface XRLightProbe : EventTarget {
  readonly attribute XRSpace probeSpace;
  attribute EventHandler onreflectionchange;
};

XRLightProbe/probeSpace

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

The probeSpace attribute is an XRSpace that has a native origin tracking the position and orientation that the XRLightProbe's lighting estimations are being generated relative to.

XRLightProbe/reflectionchange_event

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

The onreflectionchange attribute is an Event handler IDL attribute for the reflectionchange event type.

2.2. XRReflectionFormat

enum XRReflectionFormat {
  "srgba8",
  "rgba16f",
};

Reflection cube maps have an internal reflection format that indicates how the texture data is represented, and may change how applications choose to use the texture. Cube maps MAY be requested with the "srgba8" format or the preferredReflectionFormat of the light probe.

XRReflectionFormat WebGL Format WebGL Internal Format WebGPU Format HDR
"srgba8" RGBA SRGB8_ALPHA8 "rgba8unorm-srgb"
"rgba16f" RGBA RGBA16F "rgba16float"

2.3. XRLightEstimate

XRLightEstimate

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

An XRLightEstimate provides the estimated lighting values for an XRLightProbe at the time represented by an XRFrame. XRLightEstimates are queried by passing an XRLightProbe to the getLightEstimate() method of an XRFrame.

[SecureContext, Exposed=Window]
interface XRLightEstimate {
  readonly attribute Float32Array sphericalHarmonicsCoefficients;
  readonly attribute DOMPointReadOnly primaryLightDirection;
  readonly attribute DOMPointReadOnly primaryLightIntensity;
};

XRLightEstimate/sphericalHarmonicsCoefficients

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

The sphericalHarmonicsCoefficients attribute returns a Float32Array containing 9 spherical harmonics coefficients. The array MUST be 27 elements in length, with every 3 elements defining the red, green, and blue components respectively of a single coefficient. The first term of the sphericalHarmonicsCoefficients, meaning the first 3 elements of the array, MUST be representative of a valid lighting estimate. All other terms are optional, and MAY be 0 if a corresponding lighting estimate is not available due to either user privacy settings or the capabilities of the platform.

The order of coefficients in sphericalHarmonicsCoefficients, is [C00, C1-1, C10, C11, C2-2, C2-1, C20, C21, C22], where Clm is the coefficient of spherical harmonic Ylm.

XRLightEstimate/primaryLightDirection

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

The primaryLightDirection represents the direction to the primary light source from the native origin of the probeSpace of the XRLightProbe that produced the XRLightEstimate. The value MUST be a unit length 3D vector and the w value MUST be 0.0. If estimated values from the user’s environment are not available the primaryLightDirection MUST be { x: 0.0, y: 1.0, z: 0.0, w: 0.0 }, representing a light shining straight down from above.

XRLightEstimate/primaryLightIntensity

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

The primaryLightIntensity represents the color of the primary light source. The value MUST represent an RGB value mapped to the x, y, and z values respectively where each component is greater than or equal to 0.0 and the w value MUST be 1.0. If estimated values from the user’s environment are not available the primaryLightIntensity MUST be {x: 0.0, y: 0.0, z: 0.0, w: 1.0}, representing no illumination.

3. WebXR Device API Integration

Both the XRSession and XRFrame interfaces from the WebXR Device API are expanded by this module.

3.1. Session Initialization

The string "light-estimation" is introduced by this module as a new valid feature descriptor. Applications that wish to use light estimation features MUST be requested with an the "light-estimation" feature descriptor.

3.2. XRSession

The XRSession interface is extended with the ability to create new XRLightProbe instances. XRLightProbe instances have a session object, which is the XRSession that created this XRLightProbe. And an reflection format object, which is the XRReflectionFormat that the light probe may retrieve.

The XRSession interface is further extended with an attribute preferredReflectionFormat, indicating the XRReflectionFormat most closely supported by the underlying XR device

XRSession/preferredReflectionFormat

In only one current engine.

FirefoxNoneSafariNoneChrome90+
OperaNoneEdge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone
dictionary XRLightProbeInit {
  XRReflectionFormat reflectionFormat = "srgba8";
};

partial interface XRSession {
  Promise<XRLightProbe> requestLightProbe(optional XRLightProbeInit options = {});
  readonly attribute XRReflectionFormat preferredReflectionFormat;
};

XRSession/requestLightProbe

In only one current engine.

FirefoxNoneSafariNoneChrome90+
OperaNoneEdge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone
When the requestLightProbe(options) method is invoked on XRSession session, the user agent MUST run the following steps:
  1. Let promise be a new Promise.

  2. If the light-estimation feature descriptor is not contained in the session’s list of enabled features, reject promise with NotSupportedError and abort these steps.

  3. If session’s ended value is true, throw an InvalidStateError and abort these steps.

    If options’s reflectionFormat is "srgba8" or matches session’s preferredReflectionFormat:
    1. Let probe be a new XRLightProbe.

    2. Set probe’s session to session.

    3. Set probe’s reflection format to options’s reflectionFormat

    4. Resolve promise with probe.

    else
    1. Reject promise with a "NotSupportedError" DOMException

3.3. XRFrame

The XRFrame interface is extended with the ability to query the XRLightEstimate for a given XRLightProbe.

partial interface XRFrame {
  XRLightEstimate? getLightEstimate(XRLightProbe lightProbe);
};

XRFrame/getLightEstimate

In only one current engine.

FirefoxNoneSafariNoneChrome90+
OperaNoneEdge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone
When the getLightEstimate(lightProbe) method is invoked on XRFrame frame, the user agent MUST run the following steps:
  1. If frame’s active boolean is `false`, throw an InvalidStateError and abort these steps.

  2. Let session be frame’s session object.

  3. If lightProbe’s session does not equal session, throw an InvalidStateError and abort these steps.

  4. Let device be session’s XR device.

  5. If device cannot estimate the lighting for this frame, return null.

  6. Let estimate be a new XRLightEstimate.

  7. Populate estimate’s sphericalHarmonicsCoefficients, with the coefficients provided by device.

    If device has an estimated direction for the light source
    1. Set estimate’s primaryLightDirection to the estimated direction of the light source.

    else
    1. Set estimate’s primaryLightDirection to { x: 0.0, y: 1.0, z: 0.0, w: 0.0 }

    If device has an estimated intensity for the light source
    1. Set estimate’s primaryLightIntensity to the estimated intensity of the light source.

    else
    1. Set estimate’s primaryLightIntensity to {x: 0.0, y: 0.0, z: 0.0, w: 1.0}

  8. Return estimate.

4. WebXR Layers Integration

The XRWebGLBinding interface from the WebXR Layers module is expanded by this module.

4.1. XRWebGLBinding

The XRWebGLBinding interface is extended with the ability to query a reflection cube map for a given XRLightProbe.

partial interface XRWebGLBinding {
  WebGLTexture? getReflectionCubeMap(XRLightProbe lightProbe);
};

XRWebGLBinding/getReflectionCubeMap

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera Mobile64+
When the getReflectionCubeMap(lightProbe) method is invoked on XRWebGLBinding binding, the user agent MUST run the following steps:
  1. If binding’s context is lost, throw an InvalidStateError and abort these steps.

  2. Let session be binding’s session.

  3. If session is ended, throw an InvalidStateError and abort these steps.

  4. If session does not match lightProbe’s session, throw an InvalidStateError and abort these steps.

  5. Let device be session’s XR Device.

  6. If no reflection cube map is available from device, return null.

  7. Return a new WebGLTexture cubemap in the format specified by lightProbe’s reflection format and populated with the data from device.

5. Events

The task source for all tasks queued in this specification is the XR task source, unless otherwise specified.

5.1. Event Types

XRLightProbe/reflectionchange_event

In only one current engine.

FirefoxNoneSafariNoneChrome90+
Opera76+Edge90+
Edge (Legacy)NoneIENone
Firefox for AndroidNoneiOS SafariNoneChrome for Android90+Android WebViewNoneSamsung Internet15.0+Opera MobileNone

The user agent MUST fire an event named reflectionchange on an XRLightProbe object each time the contents of the cube map returned by calling getReflectionCubeMap() have changed.

6. Privacy & Security Considerations

The lighting estimation API shares many potential privacy and security risks with the Ambient Light Sensor API [AMBIENT-LIGHT], including:

In addition to these, there are a few vectors unique to lighting estimation to consider.

Lighting estimation must be declared when creating an XR Session as a feature descriptor, which will allow the user agent to notify the user of the potential privacy implications of allowing the lighting estimation API to be used by the website. The user agent is encouraged to NOT provide real-time updates to any portion of the lighting estimation API, especially the reflection cube map. By default, only low spaital frequency and low temporal frequency information should be returned by the WebXR API. Reflection cube maps should be kept low resolution, unless the user has also consented to camera permissions for a particular origin. As further mitigation, the Spherical Harmonics and primary light direction MAY be quantized.

Changes

Changes from the First Public Working Draft 9 September 2021

Conformance

Document conventions

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

Conformant Algorithms

Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc) used in introducing the algorithm.

Conformance requirements phrased as algorithms or specific steps can be implemented in any manner, so long as the end result is equivalent. In particular, the algorithms defined in this specification are intended to be easy to understand and are not intended to be performant. Implementers are encouraged to optimize.

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[AMBIENT-LIGHT]
Anssi Kostiainen; Rijubrata Bhaumik. Ambient Light Sensor. 14 December 2021. WD. URL: https://www.w3.org/TR/ambient-light/
[DOM]
Anne van Kesteren. DOM Standard. Living Standard. URL: https://dom.spec.whatwg.org/
[GEOMETRY-1]
Simon Pieters; Chris Harrelson. Geometry Interfaces Module Level 1. 4 December 2018. CR. URL: https://www.w3.org/TR/geometry-1/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://datatracker.ietf.org/doc/html/rfc2119
[WEBGL-2]
Dean Jackson; Jeff Gilbert. WebGL 2.0 Specification. 12 August 2017. URL: https://www.khronos.org/registry/webgl/specs/latest/2.0/
[WEBIDL]
Edgar Chen; Timothy Gu. Web IDL Standard. Living Standard. URL: https://webidl.spec.whatwg.org/
[WEBXR]
Brandon Jones; Manish Goregaokar; Rik Cabanier. WebXR Device API. 31 March 2022. CR. URL: https://www.w3.org/TR/webxr/

IDL Index

[SecureContext, Exposed=Window]
interface XRLightProbe : EventTarget {
  readonly attribute XRSpace probeSpace;
  attribute EventHandler onreflectionchange;
};

enum XRReflectionFormat {
  "srgba8",
  "rgba16f",
};

[SecureContext, Exposed=Window]
interface XRLightEstimate {
  readonly attribute Float32Array sphericalHarmonicsCoefficients;
  readonly attribute DOMPointReadOnly primaryLightDirection;
  readonly attribute DOMPointReadOnly primaryLightIntensity;
};

dictionary XRLightProbeInit {
  XRReflectionFormat reflectionFormat = "srgba8";
};

partial interface XRSession {
  Promise<XRLightProbe> requestLightProbe(optional XRLightProbeInit options = {});
  readonly attribute XRReflectionFormat preferredReflectionFormat;
};

partial interface XRFrame {
  XRLightEstimate? getLightEstimate(XRLightProbe lightProbe);
};

partial interface XRWebGLBinding {
  WebGLTexture? getReflectionCubeMap(XRLightProbe lightProbe);
};