WebXR Hand Input Module - Level 1

1. Introduction

2. Initialization

If an application wants to view articulated hand pose information during a session, the session MUST be requested with an appropriate feature descriptor. The string "hand-tracking" is introduced by this module as a new valid feature descriptor for articulated hand tracking.

The "hand-tracking" feature descriptor should only be granted for an XRSession when its XR device has physical hand input sources that support hand tracking.

The user agent MAY gate support for hand based XRInputSources based upon this feature descriptor.

NOTE: This means that if an XRSession does not request the "hand-tracking" feature descriptor, the user agent may choose to not support input controllers that are hand based.

3. Physical Hand Input Sources

An XRInputSource is a physical hand input source if it tracks a physical hand. A physical hand input source supports hand tracking if it supports reporting the poses of one or more skeleton joints defined in this specification.

Physical hand input sources MUST include the input profile name of "generic-hand-select" in their profiles.

For many physical hand input sources, there can be overlap between the gestures used for the primary action and the squeeze action. For example, a pinch gesture may indicate both a "select" and "squeeze" event, depending on whether you are interacting with nearby or far away objects. Since content may assume that these are independent events, user agents MAY, instead of surfacing the squeeze action as the primary squeeze action, surface it as an additional "grasp button", using an input profile derived from the "generic-hand-select-grasp" profile.

3.1. XRInputSource

partial interface XRInputSource {
   [SameObject] readonly attribute XRHand? hand;
};

The hand attribute on a physical hand input source that supports hand tracking will be an XRHand object giving access to the underlying hand-tracking capabilities. hand will have its input source set to this.

If the XRInputSource belongs to an XRSession that has not been requested with the "hand-tracking" feature descriptor, hand MUST be null.

3.2. Skeleton Joints

A physical hand input source is made up of many skeleton joints.

A skeleton joint for a given hand can be uniquely identified by a skeleton joint name, which is an enum of type XRHandJoint.

A skeleton joint may have an associated bone that it is named after and used to orient its -Z axis. The associated bone of a skeleton joint is the bone that comes after the joint when moving towards the fingertips. The tip and wrist joints have no associated bones.

A skeleton joint has a radius which is the radius of a sphere placed at its center so that it roughly touches the skin on both sides of the hand. The "tip" skeleton joints SHOULD have an appropriate nonzero radius so that collisions with the fingertip may work. Implementations MAY offset the origin of the tip joint so that it can have a spherical shape with nonzero radius.

This list of joints defines the following skeleton joints and their order:

3.3. XRHand

enum XRHandJoint {
  "wrist",

  "thumb-metacarpal",
  "thumb-phalanx-proximal",
  "thumb-phalanx-distal",
  "thumb-tip",

  "index-finger-metacarpal",
  "index-finger-phalanx-proximal",
  "index-finger-phalanx-intermediate",
  "index-finger-phalanx-distal",
  "index-finger-tip",

  "middle-finger-metacarpal",
  "middle-finger-phalanx-proximal",
  "middle-finger-phalanx-intermediate",
  "middle-finger-phalanx-distal",
  "middle-finger-tip",

  "ring-finger-metacarpal",
  "ring-finger-phalanx-proximal",
  "ring-finger-phalanx-intermediate",
  "ring-finger-phalanx-distal",
  "ring-finger-tip",

  "pinky-finger-metacarpal",
  "pinky-finger-phalanx-proximal",
  "pinky-finger-phalanx-intermediate",
  "pinky-finger-phalanx-distal",
  "pinky-finger-tip"
};

[Exposed=Window]
interface XRHand {
    iterable<XRHandJoint, XRJointSpace>;

    readonly attribute unsigned long size;
    XRJointSpace get(XRHandJoint key);
};

The XRHandJoint enum defines the various joints that each XRHand MUST contain.

Every XRHand has an associated input source, which is the physical hand input source that it tracks.

NOTE: The handedness property of XRInputSource describes which hand the XR input source is associated with, if any.

If an individual device does not support a joint defined in this specification, it MUST emulate it instead.

The size attribute MUST return the number 25.

3.4. XRJointSpace

[Exposed=Window]
interface XRJointSpace: XRSpace {
  readonly attribute XRHandJoint jointName;
};

The native origin of an XRJointSpace is the position and orientation of the underlying joint.

The native origin of the XRJointSpace may only be reported when native origins of all other XRJointSpaces on the same hand are being reported. When a hand is partially obscured the user agent MUST either emulate the obscured joints, or report null poses for all of the joints.

Note: This means that when fetching poses you will either get an entire hand or none of it.

This by default precludes faithfully exposing polydactyl/oligodactyl hands, however for fingerprinting concerns it will likely need to be a separate opt-in, anyway. See Issue 11 for more details.

The native origin has its -Y direction pointing perpendicular to the skin, outwards from the palm, and -Z direction pointing along their associated bone, away from the wrist.

For tip skeleton joints where there is no associated bone, the -Z direction is the same as that for the associated distal joint, i.e. the direction is along that of the previous bone. For wrist skeleton joints the -Z direction SHOULD point roughly towards the center of the palm.

Every XRJointSpace has an associated hand, which is the XRHand that created it.

jointName returns the joint name of the joint it tracks.

Every XRJointSpace has an associated joint, which is the skeleton joint corresponding to the jointName.

4. Frame Loop

4.1. XRFrame

partial interface XRFrame {
    XRJointPose? getJointPose(XRJointSpace joint, XRSpace baseSpace);
    boolean fillJointRadii(sequence<XRJointSpace> jointSpaces, Float32Array radii);

    boolean fillPoses(sequence<XRSpace> spaces, XRSpace baseSpace, Float32Array transforms);
};

NOTE: if the user agent can’t determine the pose of any of the spaces belonging to the same XRHand, all the spaces of that XRHand must also not have a pose.

NOTE: if any of the spaces belonging to the same XRHand return null when populating the pose, all the spaces of that XRHand must also return null when populating the pose

4.2. XRJointPose

An XRJointPose is an XRPose with additional information about the size of the skeleton joint it represents.

[Exposed=Window]
interface XRJointPose: XRPose {
    readonly attribute float radius;
};

The radius attribute returns the radius of the skeleton joint in meters.

The user-agent MUST set radius to an emulated value if the XR device does not have the capability of determining this value, either in general or in the current animation frame (e.g. when the skeleton joint is partially obscured).

5. Privacy & Security Considerations

Since this feature returns new sensor data, the User Agent MUST ask for explicit consent from the user at session creation time.

Data returned from this API, MUST NOT be so specific that one can detect individual users. If the underlying hardware returns data that is too precise, the User Agent MUST anonymize this data before revealing it through the WebXR Hand Input API.

This API MUST only be supported in XRSessions created with XRSessionMode of "immersive-vr" or "immersive-ar". "inline" sessions MUST not support this API.

Changes

Changes from the First Public Working Draft 22 October 2020

• Mention grasp profile (GitHub #68)

• Change from constants to enums + change XRHand into a map (GitHub #71)

• Added additional clarification in security section (GitHub #87)

• Marked hand as sameobject + added a clarifying note (GitHub #93)

• Nonzero radius for tip (GitHub #111)