📘 Resource Overview & Documentation Details

You are viewing the technical directory listing for Gesture Recognition for Collaborative Robots. This engineering resource is curated under our Collaborative Robots division to support automation specialists, control systems engineers, and manufacturing technicians looking for verified reference materials.

🔍 Contextual Abstract & Technical Scope

Share Summary:LinkedInEmail

Gesture recognition for collaborative robots can offer operators a direct way to issue commands without relying on a conventional control interface. But in an industrial setting, a gesture interface must do more than identify a hand shape: it must distinguish valid commands from incidental movements and reject uncertain or incomplete inputs. A vision-based approach using Faster R-CNN detection and a rule-based post-processing stage illustrates how these requirements can be addressed—and where further validation is needed.

Why gesture recognition needs clear rules

Collaborative robots are intended to work in environments shared with people, making communication between operator and machine an important part of the interaction. Natural human gestures, however, can be ambiguous. For this reason, the system uses a small set of deliberately defined gestures rather than trying to interpret unrestricted movement.

Each valid gesture consists of two detected hands. The left hand is held closed as an anchor, while the right hand indicates the command. The tested commands are:

  • START: both hands are closed.
  • STOP: the right hand is open.
  • LEFT and RIGHT: the right hand points in the corresponding direction.

The commands identify a robot action or path in the experiment; LEFT and RIGHT are not continuous directional controls. A command is considered invalid if the anchor is missing or the hands are not sufficiently close in both horizontal and vertical dimensions. This structured approach is intended to reduce the chance that an unrelated hand pose will be treated as an instruction.

How Faster R-CNN hand detection works with spatial filtering

The recognition pipeline has two main stages. First, a Faster R-CNN object detector processes an RGB image and returns hand-region predictions. Each prediction includes a region of interest (RoI), a class label for the hand pose and a confidence score. Faster R-CNN combines a convolutional neural network for image features with a Region Proposal Network that identifies candidate object regions.

Second, a Custom Prediction Function (CPF) checks whether the detected hands form a valid command. It applies several filters rather than accepting each hand classification in isolation:

  1. Predictions below the confidence threshold are discarded. The procedure retains predictions with confidence scores greater than 90% and checks for a closed-hand candidate.
  2. Distances between pairs of detected hand regions are calculated. Pairs outside predefined close and far distance boundaries are rejected.
  3. When two regions overlap, the prediction with the lower confidence score is removed.
  4. The leftmost remaining prediction is checked as the anchor. The connected hand predictions are evaluated as possible command pairs, and the candidate pair with the highest combined confidence score is selected.

This post-processing matters because a detector may return multiple candidate regions, including overlapping or unrelated detections. The CPF uses confidence, position and proximity to decide whether those detections make up a complete two-hand gesture.

Training data designed for varied conditions

The system was evaluated using RGB images captured with a Kinect v2 camera. The datasets covered several visual conditions: operators in casual clothing, clothing similar to skin or background colors, light-blue gloves, and a camera positioned close to the operator. A combined Complete dataset brought these conditions together. The operators moved around the test area and performed gestures while facing the camera or standing at an angle.

For the combined dataset, images from each individual dataset were shuffled and divided into training and test portions, with 80% selected for training and 20% for testing. The portions were then shuffled again so that training batches included samples from different conditions. This approach was intended to preserve representation from each source dataset in both splits.

Two evaluation setups were used. In the first, test images came from operators who were also represented in the corresponding training data. In the second, images of one chosen operator were excluded from training and used only for testing. The latter provides a check on performance with an operator not seen during training, though results for an individual operator can depend on hand size, gesture style and image quality.

Results and limits to consider

On the Complete dataset, the first evaluation achieved a 97.13% F1-score and 95.01% recognition accuracy. When the chosen operator was held out from training, the Complete dataset achieved a 95.11% F1-score and 92.12% accuracy. These results were comparable across the two experiments, but they do not establish performance for every operator or operating environment.

Errors were associated with several conditions: operators viewed from the side, cluttered backgrounds, low image resolution and inaccurate bounding boxes. A separate Errors dataset, made mostly of images that should not produce a command, exposed a significant challenge. The system generated 55 false positives among 166 images, most often in images containing multiple operators. Its positive predictive value on this dataset was 1.79%, underscoring that strong results on a gesture test set do not by themselves demonstrate reliable rejection of non-command scenes.

Measured online inference time was 0.13 seconds per image, including image acquisition and display-related processing. The offline figure was 0.11 seconds. These measurements indicate that the tested implementation could process images at a pace suitable for real-time interaction in the reported setup. They are not, on their own, evidence that the interface meets a particular industrial safety or control requirement.

Practical implications for automation teams

For engineers and integrators considering vision-based robot commands, the key lesson is to design the gesture vocabulary and validation logic together. A clear anchor gesture, spatial constraints and confidence filtering can help distinguish intended commands from isolated hand poses. Training data should also reflect the expected variation in clothing, camera distance, operator position and background.

Testing should include not only correct gestures but also scenes in which the system should issue no command, including multiple people and incidental movements. The reported false positives make this particularly important. A gesture interface should be evaluated within the intended application and treated as one component of a broader control and safety design, not as a substitute for required safeguards.

Conclusion

Faster R-CNN hand detection combined with spatial post-processing offers a structured approach to gesture recognition for collaborative robots. The Complete dataset produced strong recognition results, and the measured processing time supported real-time interaction in the tested configuration. At the same time, performance varied with image conditions, and false positives in multi-operator scenes remain a concern. For practical adoption, automation teams should validate both command recognition and command rejection under representative site conditions before relying on gestures in robot operation.

Access Official Technical PDF