MULTIMODAL PERCEPTION / ADAPTIVE CONTROL / WEARABLE ROBOTICS
Context-Aware Exoskeleton Assistance
Connecting task context with the timing and magnitude of physical assistance.
A research project connecting egocentric vision and gaze tracking with adaptive assistance for an active back-support exoskeleton. The system brings information about the attended object and the handling task into the control loop, alongside the user’s posture and physical interaction.
The problem
During manual handling, useful support depends on both when assistance is activated and how it adapts to the task. Posture alone does not describe which object the user is attending to, whether it is being handled, or what load it introduces.
This project investigates how visual and gaze information can complement the signals available to a back-support exoskeleton, connecting scene understanding with physical assistance.
Approach
Attend
Egocentric vision and gaze
Tobii eye-tracking glasses provide a first-person view and gaze information. YOLO-based perception identifies objects and their visual grasp state, while gaze indicates the object receiving the user’s attention.
Interpret
Task-context cues
Visual cues inform task progression and assistance activation. A vision-language model provides object and payload estimates to complement the perception pipeline.
Assist
Adaptive physical support
A finite-state machine combines posture and perception cues. A variable-admittance controller adjusts assistance, stiffness, and damping across the handling task, with smooth transitions between control states.
Selected experimental results
Controlled stooping and lifting experiments evaluated assistance timing, perceived interaction quality, and muscular activation. These are separate evaluations: a single-participant gaze ablation, a 15-participant questionnaire study, and a preliminary 5-participant EMG study.
0.066 s
Payload-aware assistance onset
Gaze-conditioned pipeline
Without gaze: 1.892 s · 50 trials per condition
3.60 / 4
Physical-demand questionnaire
Q1 score · higher means less demand
Posture-only: 2.13 / 4 · no exoskeleton: 2.00 / 4
14.29%
Thoracic EMG RMS reduction
Envelope RMS · loaded lifting
Lumbar RMS reduction: 14.63% · both vs no exoskeleton
The gaze ablation used 50 randomized trials per condition. Mean onset delay was 0.066 ± 0.033 s with gaze and 1.892 ± 0.040 s without gaze, a mean anticipation of 1.826 s. The reference event was the first YOLO detection of the object as grasped, rather than a direct measurement of physical load transfer.
The questionnaire compared vision-gated assistance, posture-only assistance, and no exoskeleton. The preliminary EMG comparison evaluated loaded lifting with the complete vision-enabled system against no exoskeleton. It did not include a posture-only EMG condition, so the muscular-activation reductions cannot be attributed independently to gaze or vision gating.
Implementation
The public code integrates the Tobii perception application, a Moteus motor-controller node, and a ROS 2 admittance controller. Perception cues and joint-state information inform a finite-state assistance policy for pick-and-place handling.
The controller blends trunk and payload compensation, ramps stiffness and damping between interaction profiles, and publishes telemetry for inspecting the system’s behaviour. The perception models and control settings are tied to the experimental setup; this overview makes no claim of general deployment performance.
- Platform
- Active back-support exoskeleton
- Perception
- Egocentric vision · gaze tracking · YOLO
- Task context
- Visual grasp state · VLM-based payload estimation
- Control
- Variable admittance · finite-state assistance
- Integration
- ROS 2 Humble · Moteus · Tobii
- Implementation
- Python · ROS 2 interfaces
Resources
Explore the public control stack and the earlier arXiv preprint for further context.
The linked preprint is an earlier version and may not include all figures or experimental results summarized on this page.