Skip to content

Latest commit

 

History

History
88 lines (82 loc) · 3.14 KB

README.md

File metadata and controls

88 lines (82 loc) · 3.14 KB
Raw

ss18_hand

Simulation and control of Roboy's hand model and gesture recognition

YOU CAN:

  • control the hand model by directly publishing joints angles to rostopic.
  • let the hand model mimic arbitrary hand movement with Leap Motion.
  • train neural network to recognize gestures (see gesture_recognition).

Here begins the story:

Installation

This project is built on Ubuntu 16.04. It is based on ROS Kinetic. If you don't have it yet, install it.

1. Clone the project:

$ git clone https://github.com/Roboy/ss18_hand.git
$ git submodule init
$ git submodule update

2. Install packages for gesture recognition (follow steps here)

3. Install Leap Motion SDK (only if you want to use this function)

Install the version for Linux: https://developer.leapmotion.com/sdk/v2

Build the project

$ cd /path/to/ss18_hand
$ catkin_make
$ source devel/setup.bash	# add this to your ~/.bashrc
$ export CONFIG_PATH="PATH_TO_THIS_RESPOSITORY/ss18_hand/src/roboy_moveit_configs" # add this to your ~/.bashrc

And link the hand model to Gazebo:

$ ln -s PATH_TO_THIS_RESPOSITORY/ss18_hand/src/roboy_models/left_hand ~/.gazebo/models/left_hand

Play with it

1. Execute gestures with predefined joint angles

There are some .bash files in the repository ss18_hand with predefined joint angles for certain gestures. To use it, you need to:

  • Launch the hand model
$ roslaunch roboy_hand LeftHand.launch
  • Start another terminal and source the .bash file, e.g.:
$ source close_hand.bash
2. Mimic gesture with Leap Motion
  • Start Leap Core Service
$ sudo leapd
  • Launch the hand model
$ roslaunch roboy_hand LeftHand.launch
  • Start detecting joints
$ rosrun leap_motion sender.py
  • See the detected joints angles
$ rostopic echo /roboy/middleware/FingerCommand

3. Execute gesture with look-up-table

  • Launch the gesture_execution.launch file to start the necessary nodes
$ roslaunch roboy_hand gesture_execution.launch model_name:=left_hand

to terminate it: press ctrl+c, q, enter, y, enter

  • Open a new terminal and start rviz for visualization of the model
$ rviz
  • If you have started the gesture recognition program, the detected gesture will be actuated on the model. (to run the neural network, follow the steps here)
  • Or you can also publish message to the ROS topic /gesture_recognition to execute gesture from the look-up-table
$ rostopic pub /gesture_recognition std_msgs/Int64 "data: 0"

for now we have gesture id from 0 to 9 corresponding to gesture 1 to 10 in the picture below:

Further Info

For more info on the project visit our Confluence Page: https://devanthro.atlassian.net/wiki/spaces/SS18/pages/235601936/Hand