WebXR Lighting Estimation API Level 1

Editor’s Draft,

Previous Versions:
Issue Tracking:
GitHub
Editors:
(Google)
(Mozilla)

Abstract

This specification describes support for exposing estimates of environmental lighting conditions to WebXR sessions.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document was published by the Immersive Web Working Group as an Editors' Draft. This document is intended to become a W3C Recommendation. Feedback and comments on this specification are welcome. Please use Github issues. Discussions may also be found in the public-immersive-web@w3.org archives.

Publication as an Editors' Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 15 September 2020 W3C Process Document.

1. Introduction

The WebXR Lighting Estimation module expands the WebXR Device API, the WebXR Augmented Reality Module, and the WebXR Layers module with the ability to expose estimates of the lighting conditions of the user’s environment.

2. Light Primitives

2.1. XRLightProbe

An XRLightProbe collects estimated lighting information at a given point in the user’s environment.

[SecureContext, Exposed=Window]
interface XRLightProbe : EventTarget {
  readonly attribute XRSpace probeSpace;
  attribute EventHandler onreflectionchange;
};

The probeSpace attribute is an XRSpace that has a native origin tracking the position and orientation that the XRLightProbe's lighting estimations are being generated relative to.

The onreflectionchange attribute is an Event handler IDL attribute for the reflectionchange event type.

2.2. XRReflectionFormat

enum XRReflectionFormat {
  "srgba8",
  "rgba16f",
};

Reflection cube maps have an internal reflection format that indicates how the texture data is represented, and may change how applications choose to use the texture. Cube maps MAY be requested with the "srgba8" format or the preferredReflectionFormat of the light probe.

XRReflectionFormat WebGL Format WebGL Internal Format WebGPU Format HDR
"srgba8" RGBA SRGB8_ALPHA8 "rgba8unorm-srgb"
"rgba16f" RGBA RGBA16F "rgba16float"

2.3. XRLightEstimate

An XRLightEstimate provides the estimated lighting values for an XRLightProbe at the time represented by an XRFrame. XRLightEstimates are queried by passing an XRLightProbe to the getLightEstimate() method of an XRFrame.

[SecureContext, Exposed=Window]
interface XRLightEstimate {
  readonly attribute Float32Array sphericalHarmonicsCoefficients;
  readonly attribute DOMPointReadOnly primaryLightDirection;
  readonly attribute DOMPointReadOnly primaryLightIntensity;
};

The sphericalHarmonicsCoefficients attribute returns a Float32Array containing 9 spherical harmonics coefficients. The array MUST be 27 elements in length, with every 3 elements defining the red, green, and blue components respectively of a single coefficient. The first term of the sphericalHarmonicsCoefficients, meaning the first 3 elements of the array, MUST be representative of a valid lighting estimate. All other terms are optional, and MAY be 0 if a corresponding lighting estimate is not available due to either user privacy settings or the capabilities of the platform.

The order of coefficients in sphericalHarmonicsCoefficients, is [C00, C1-1, C10, C11, C2-2, C2-1, C20, C21, C22], where Clm is the coefficient of spherical harmonic Ylm.

The primaryLightDirection represents the direction to the primary light source from the native origin of the probeSpace of the XRLightProbe that produced the XRLightEstimate. The value MUST be a unit length 3D vector and the w value MUST be 0.0. If estimated values from the users’s environment are not available the primaryLightDirection MUST be { x: 0.0, y: 1.0, z: 0.0, w: 0.0 }, representing a light shining straight down from above.

The primaryLightIntensity represents the direction to the primary light source from the origin of the probeSpace of the XRLightProbe that produced the XRLightEstimate. The value MUST represent an RGB value mapped to the x, y, and z values respectively where each component is greater than or equal to 0.0 and the w value MUST be 1.0. If estimated values from the users’s environment are not available the primaryLightIntensity MUST be {x: 0.0, y: 0.0, z: 0.0, w: 1.0}, representing no illumination.

3. WebXR Device API Integration

Both the XRSession and XRFrame interfaces from the WebXR Device API are expanded by this module.

3.1. Session Initialization

The string "light-estimation" is introduced by this module as a new valid feature descriptor. Applications that wish to use light estimation features MUST be requested with an the "light-estimation" feature descriptor.

3.2. XRSession

The XRSession interface is extended with the ability to create new XRLightProbe instances. XRLightProbe instances have a session object, which is the XRSession that created this XRLightProbe. And an reflection format object, which is the XRReflectionFormat that the light probe may retrieve.

The XRSession interface is further extended with an attribute preferredReflectionFormat, indicating the XRReflectionFormat most closely supported by the underlying XR device

dictionary XRLightProbeInit {
  XRReflectionFormat reflectionFormat = "srgba8";
};

partial interface XRSession {
  Promise<XRLightProbe> requestLightProbe(optional XRLightProbeInit options = {});
  readonly attribute XRReflectionFormat preferredReflectionFormat;
};
When the requestLightProbe(options) method is invoked on XRSession session, the user agent MUST run the following steps:
  1. Let promise be a new Promise.

  2. If the light-estimation feature descriptor is not contained in the session’s list of enabled features, reject promise with NotSupportedError and abort these steps.

  3. If session’s ended value is true, throw an InvalidStateError and abort these steps.

    If options’s reflectionFormat is "srgba8" or matches session’s preferredReflectionFormat:
    1. Let probe be a new XRLightProbe.

    2. Set probe’s session to session.

    3. Set probe’s reflection format to options’s reflectionFormat

    4. Resolve promise with probe.

    else
    1. Reject promise with a "NotSupportedError" DOMException

3.3. XRFrame

The XRFrame interface is extended with the ability to query the XRLightEstimate for a given XRLightProbe.

partial interface XRFrame {
  XRLightEstimate? getLightEstimate(XRLightProbe lightProbe);
};
When the getLightEstimate(lightProbe) method is invoked on XRFrame frame, the user agent MUST run the following steps:
  1. If frame’s active boolean is `false`, throw an InvalidStateError and abort these steps.

  2. Let session be frame’s session object.

  3. If lightProbe’s session does not equal session, throw an InvalidStateError and abort these steps.

  4. Let device be session’s XR device.

  5. If device cannot estimate the lighting for this frame, return null.

  6. Let estimate be a new XRLightEstimate.

  7. Populate estimate|'s sphericalHarmonicsCoefficients, with the coefficients provided by device.

    If device has an estimated direction for the light source
    1. Set estimate’s primaryLightDirection to the estimated direction of the light source.

    else
    1. Set estimate’s primaryLightDirection to { x: 0.0, y: 1.0, z: 0.0, w: 0.0 }

    If device has an estimated intensity for the light source
    1. Set estimate’s primaryLightIntensity to the estimated intensity of the light source.

    else
    1. Set estimate’s primaryLightIntensity to {x: 0.0, y: 0.0, z: 0.0, w: 1.0}

  8. Return estimate.

4. WebXR Layers Integration

The XRWebGLBinding interface from the WebXR Layers module is expanded by this module.

4.1. XRWebGLBinding

The XRWebGLBinding interface is extended with the ability to query a reflection cube map for a given XRLightProbe.

partial interface XRWebGLBinding {
  WebGLTexture? getReflectionCubeMap(XRLightProbe lightProbe);
};
When the getReflectionCubeMap(lightProbe) method is invoked on XRWebGLBinding binding, the user agent MUST run the following steps:
  1. If binding’s context is lost, throw an InvalidStateError and abort these steps.

  2. Let session be binding’s session.

  3. If session is ended, throw an InvalidStateError and abort these steps.

  4. If session does not match lightProbe’s session, throw an InvalidStateError and abort these steps.

  5. Let device be session’s XR Device.

  6. If no reflection cube map is available from device, return null.

  7. Return a new WebGLTexture cubemap in the format specified by lightProbe’s reflection format and populated with the data from device.

5. Events

The task source for all tasks queued in this specification is the XR task source, unless otherwise specified.

5.1. Event Types

The user agent MUST fire a reflectionchange event on an XRLightProbe object each time the contents of the cube map returned by calling getReflectionCubeMap() have changed. The event MUST be of type Event.

6. Privacy & Security Considerations

The lighting estimation API shares many potential privacy and security risks with the Ambient Light Sensor API [AMBIENT-LIGHT], including:

In addition to these, there are a few vectors unique to lighting estimation to consider.

Lighting estimation must be declared when creating an XR Session as a feature descriptor, which will allow the user agent to notify the user of the potential privacy implications of allowing the lighting estimation API to be used by the website. The user agent is encouraged to NOT provide real-time updates to any portion of the lighting estimation API, especially the reflection cube map. By default, only low spaital frequency and low temporal frequency information should be returned by the WebXR API. Reflection cube maps should be kept low resolution, unless the user has also consented to camera permissions for a particular origin. As further mitigation, the Spherical Harmonics and primary light direction MAY be quantized.

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[AMBIENT-LIGHT]
Anssi Kostiainen. Ambient Light Sensor. 7 March 2019. WD. URL: https://www.w3.org/TR/ambient-light/
[DOM]
Anne van Kesteren. DOM Standard. Living Standard. URL: https://dom.spec.whatwg.org/
[GEOMETRY-1]
Simon Pieters; Chris Harrelson. Geometry Interfaces Module Level 1. 4 December 2018. CR. URL: https://www.w3.org/TR/geometry-1/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[WEBGL-2]
Dean Jackson; Jeff Gilbert. WebGL 2.0 Specification. 12 August 2017. URL: https://www.khronos.org/registry/webgl/specs/latest/2.0/
[WebIDL]
Boris Zbarsky. Web IDL. 15 December 2016. ED. URL: https://heycam.github.io/webidl/
[WEBXR]
Brandon Jones; Manish Goregaokar; Nell Waliczek. WebXR Device API. 24 July 2020. WD. URL: https://www.w3.org/TR/webxr/

IDL Index

[SecureContext, Exposed=Window]
interface XRLightProbe : EventTarget {
  readonly attribute XRSpace probeSpace;
  attribute EventHandler onreflectionchange;
};

enum XRReflectionFormat {
  "srgba8",
  "rgba16f",
};

[SecureContext, Exposed=Window]
interface XRLightEstimate {
  readonly attribute Float32Array sphericalHarmonicsCoefficients;
  readonly attribute DOMPointReadOnly primaryLightDirection;
  readonly attribute DOMPointReadOnly primaryLightIntensity;
};

dictionary XRLightProbeInit {
  XRReflectionFormat reflectionFormat = "srgba8";
};

partial interface XRSession {
  Promise<XRLightProbe> requestLightProbe(optional XRLightProbeInit options = {});
  readonly attribute XRReflectionFormat preferredReflectionFormat;
};

partial interface XRFrame {
  XRLightEstimate? getLightEstimate(XRLightProbe lightProbe);
};

partial interface XRWebGLBinding {
  WebGLTexture? getReflectionCubeMap(XRLightProbe lightProbe);
};