Unreal Engine 5.8, OpenXR hand tracking. Updated November 2026. Written by the Algosyntax team.
Quick answer: your normal UMG widgets work with hands. Put them on a Widget Component in the world, and drive a Widget Interaction Component from the hand: a fingertip poke for close panels, a pinch ray for far ones. The Widget Interaction Component simulates mouse clicks, so buttons work unchanged. The fixes most people need:
- A slider the ray can’t hit needs Is Focusable ticked.
- Scroll boxes need Up and Down buttons, because a hand can’t drag a scroll bar or turn a mouse wheel.
- Aim the pinch ray from the knuckles, not the thumb tip.
- Turn off Cast Shadow on widget components.
This is the setup behind VR Musician‘s song list and transport panels, which you use with bare hands.
The setup
- Read the hand joints with Get Hand Tracking State, from Unreal’s OpenXR Hand Tracking plugin. You get 26 joints per hand. The Meta XR plugin isn’t needed.
- Give each hand a Widget Interaction Component.
- Poke: place it at the index fingertip, pointing forward. Press when the tip touches the panel, release when it leaves.
- Pinch ray: point it from the hand towards the panel. Press when thumb and index meet, release when they separate.
Poke for near panels plus point-and-pinch for far panels is also the pattern Meta recommends.
Fix 1: the slider ignores the ray
Our ray and hit dot disappeared over the seek slider, and pinching did nothing. The slider had Is Focusable turned off. A slider only counts as interactable when it’s focusable, and the Widget Interaction Component only hovers and presses interactable widgets.
Fix: select the slider in the Widget Blueprint and tick Is Focusable. For a seek bar, seek on release (On Mouse Capture End), not on every value change, so playback doesn’t fight the drag.
Fix 2: scroll boxes can’t be scrolled
A pinch is a click. It can’t drag a scroll bar, and there’s no mouse wheel in VR. Add Up and Down buttons next to each list and move the scroll offset yourself:
float NewOffset = ScrollBox->GetScrollOffset() + Steps * ScrollStep;
NewOffset = FMath::Clamp(NewOffset, 0.f, ScrollBox->GetScrollOffsetOfEnd());
ScrollBox->SetScrollOffset(NewOffset);
The same nodes exist in Blueprints.
Fix 3: the pinch ray jumps when you click
We first aimed the ray from the thumb tip. The thumb tip moves as you pinch, so the ray swung off the button just as it clicked. Aim from the midpoint between the thumb and index knuckles (the proximal joints) instead. They barely move during a pinch.
Two more details:
- When tracking is lost, clear the hovered widget. Otherwise the first pinch after tracking returns clicks whatever was hovered before.
- Make the ray visible. A 4 mm ray was too thin to see in the headset. A 1 cm ray plus a 2.5 cm dot at the hit point, shrinking while you pinch, worked well. Show it only when it’s over a panel.
Fix 4: panel size and placement
- Put the scale on the widget component, not the actor’s root. Our panel came out 8 metres wide, because placing an actor in a level resets its root scale to 1 and our 0.05 scale was on the root.
- Size for fingertips. A draw size of about 800 x 700 to 1000 x 860 pixels at a scale of 0.05 gave panels 40 to 50 cm wide with buttons you can hit with a fingertip. Our song rows were 104 px tall, about 5 cm.
- Use a minimum row height, not a fixed one, or a two-line description overlaps the next row.
- Check placement from the seat, in the headset or simulator. Our panels overlapped the pianist’s arms twice before we got it right.
Fix 5: things that hide or darken the UI
- Shadows: a widget component casts its shape into the sun’s shadow by default. Turn Cast Shadow off. It’s a floating panel, and the shadow only costs time.
- Particles: snow between your eyes and a panel 60 to 100 cm away becomes large white blobs. Fade particles out close to the camera (we used fully faded within 1 m, fully visible from 2.5 m).
- Black panels: in a scene with physical light units, unlit widgets come out black. See this explanation and fix.
Fix 6: hands pressing things by accident
In a piano app your fingers pass over the keys all the time. We added a toggle for playing the keys with your hands. When you turn it back on, the first frame only records fingertip positions. Otherwise the first key gets a made-up speed from a stale previous position.
FAQ
Do I need Common UI? No. Plain UMG with the Widget Interaction Component is enough.
Should both hands use the panels? Two rays close to neighbouring panels can click at the same time. We made the panels right-hand only by default, with a setting for left-handed players.