The goal of this project is to demonstrate how to create a real-time object detection autonomous robot with relatively inexpensive components. By training your own machine learning model and pairing Intel's Neural Compute Stick 2 with a Raspberry Pi 3 B+, you'll be able jump-start your next real-time object detection project!
- Update History
- Project Overview
- Reading This Guide
- Hardware List
- Software List
- Hardware Configuration
- Train Object Detection Model with TensorFlow
- Optimize Model for Intel Neural Compute Stick 2
- Integrate the Optimized IR Model
- Testing
- 🐷 Deploy Porky
- Observations
- Feedback
- References and Acknowledgements
2019/05/09: Initial Release
This project will guide you on how to:
- Train your own model in TensorFlow using a transfer learning technique to save time and money
- Optimize the resulting TensorFlow model so that it can be used with Intel's Inference Engine/Neural Compute Stick
- Implement the optimized Intermediate Representation model into a OpenCV/Python program
- Deploy the program with real-time performance and feedback loops
The goal of this guide is to provide as many steps as possible in order to create an identical robot project. However, not everyone will have an identical development environment as I do/did (a Dockerfile will be provided in the future to help alleviate this issue) nor will they have identical hardware components.
As a result, please regard the following tips:
- Refer to the links provided inline and the References and Acknowledgements for further explanations and examples.
- When prompted with code/terminal examples that contain expressions that start with 'Path' and finish in camel case format, it's expected that you replace this expression with your own path.
For example PathToYourImageDirectory and PathToYourPictureLabel are intended to be changed to reflect your working environment:
pi@raspberrypi:~/Porky/dataset $ python3 image_capture.py -picture_directory=~/PathToYourImageDirectory -picture_label=PathToYourPictureLabel- Notice how the first section of the terminal example above provides the user information within the terminal: pi@raspberrypi:~$. This section is provided only as an example, your actual environment will probably differ.
Please take note of the 'Optional Hardware' list, this is provided only if you want to create a robot that is identical to the one this project demonstrates. This does not mean you are restricted to these components. Feel free to swap, subtract, and/or add components. However, for the best initial results (if your intention is to follow this guide), I highly suggest acquiring the components within the 'Required Hardware' section at the very least. This will enable you to train a customized machine learning model and perform real-time object detection with just a Raspberry Pi and the Intel Neural Compute Stick 2. My personal favorite sites for finding robotic components are Adafruit, RobotShop, eBay, and Amazon. The possibilities are endless!
- 💻 Raspberry Pi 3 B+ w/ MicroSD Card and a way to power the device (battery or AC wall adapter)
- Intel Neural Compute Stick 2 (NCS2)
- 📹 USB or Pi Camera This project uses the PS3 Eye Camera which can be found on eBay for about $6 USD each.
Disclaimer: Feel free to swap out any of these parts.
- Development PC (Linux, Windows, MacOS) Development for this project was performed on a Windows 10 platform which this guide reflects. While it's recommended to utilize a dev PC (and to not develop directly on the Raspberry Pi) for sake of speed, it's not necessary. I also recommend utilizing Linux (Ubuntu 16.04) because much of the machine learning documentation out there is geared towards Linux. Future updates for this project will be performed within the Linux environment to save some frustrations while trying to follow along with documentation.
- Display Monitor w/ HDMI Output and cable Helpful for debugging and testing within Raspberry Pi environment.
- Robot Chassis Kit w/ Motors This project uses the Lynxmotion 4WD1 Rover Kit. You can purchase this kit directly from RobotShop or find a used kit on eBay.
- Servos x2 w/ Mounting Hardware This project uses the Lynxmotion Pan and Tilt Kit.
- PWM/Servo Controller This project uses this one from Amazon.
- Motor Controller This project uses the Sabertooth 2X12 Regenerative Dual Channel Motor Controller which can be found at RobotShop.
- Li-Po Battery To power the motor controller. This project uses an HRB 3S 11.1V 6000 mAh LiPo battery with an XT60 Plug. Search on Amazon or eBay for deals. These batteries are known to cause fires, so please be aware of the risks and proper handling procedures.
- Balance Charger/Discharger To charge/discharge your Li-Po Battery safely. I use the SKYRC iMAX B6 Mini. Note: this charger requires the charging unit itself and a wall adapter/power supply, ensure you're purchasing both.
- Portable Powerbank Be aware that not all portable chargers are compatible for Raspberry Pi projects (battery sleeping features). This project uses this RAVPower Portable Charger.
- Mounting Arm For holding the Pan and Tilt Kit, this project uses the VideoSecu 1/4" Security Camera Mount and the SMALLRIG Super Clamp w/ 1/4" and 3/8" Thread.
- iFixit Toolkit I use the iFixit Pro Tech Toolkit (highly recommended if you do a lot of tinkering).
- Velcro Tape (for modular prototype mounting) I used a 2" Adhesive Black Hook and Loop Tape.
- Assorted Electrical Components (switches, buttons, wires, breadboards, etc) Check out Adafruit for great deals on electrical components.
The following list can be determined on your own by following the OpenVINO Toolkit documentation.
Please visit the following link: OpenVINO Windows Toolkit for installing OpenVINO on your platform.
The following bullet points reflect the requirements based on a Windows 10 environment:
- Python 3.6.5 with Python Libraries, 64-bit
- Microsoft Visual Studio with C++ 2019, 2017, or 2015 with MSBuild
- CMake 3.4+
- OpenCV 3.4+
- OpenVINO 2019.R1+
- TensorFlow
PS C:\> pip install tensorflow- Glob for Python 3
PS C:\> pip install glob3- Pandas
PS C:\> pip install pandas- Pillow
PS C:\> pip install PillowPlease visit the following link: OpenVINO Toolkit for Raspberry Pi for installing OpenVINO on your Raspberry Pi.
The following bullet points reflect the basic requirements to
-
Python 3.5+ (included with Raspbian Stretch OS)
-
OpenVINO 2019.R1+
-
Python Libraries (non-standard):
- OpenCV 4.1+ (included with OpenVINO Toolkit)
- Adafruit ServoKit
pi@raspberrypi:~$ pip3 install adafruit-circuitpython-servokit
pi@raspberrypi:~$ pip3 install pysabertooth
The wiring diagrams contained within this section were created with Fritzing, a great open-source tool.
To train your own machine learning model, you will need to gather the data to train and validate/test your model on. The idea for this project was to train a model based on images captured with an identical camera that was eventually going to be deployed live.
This setup consists of:
- Raspberry Pi 3 B+ w/ MicroSD Card
- PS3 Eye USB Camera eBay Search: ps3 eye camera
- Portable Powerbank RAVPower Portable Charger
- Mini Button Tactile Button Switches from Adafruit
- Breadboard Tiny Breadboard from Adafruit
- USB C to MicroUSB Cable Amazon Link To connect the Raspberry Pi to the Powerbank.
- 2 Female/Male Wires Female/Male 'Extension' Wires from Adafruit
Wire Diagram of the Button setup for the Raspberry Pi:
Note: the USB Camera and Powerbank are missing from the diagram above.
Image of the configured setup:
Please see the Capture Images with the Image Capturing Setup section to capture your own images for your dataset using this hardware configuration.
This hardware configuration serves the purpose for testing your hardware components (motors, servos, etc) and software integrations (debugging, testing, sandbox). This setup is geared towards using AC wall adapters to save batteries and keeping moving components as stationary as possible. Having a proper testing setup can potentially save lots of frustration and money. It is strongly suggested to test your own project before deploying it into the wild.
This setup consists of:
- Raspberry Pi 3 B+ w/ MicroSD Card
- Intel Neural Compute Stick 2 (NCS2)
- PS3 Eye USB Camera eBay Search: ps3 eye camera Outer case has been removed to save weight and help with mounting.
- Display Monitor w/ HDMI Output and Cable Helpful for debugging and testing within Raspberry Pi environment.
- Robot Chassis Kit w/ Motors This project uses the Lynxmotion 4WD1 Rover Kit. You can purchase this kit directly from RobotShop or find a used kit on eBay.
- Servos x2 w/ Mounting Hardware This project uses the Lynxmotion Pan and Tilt Kit.
- PWM/Servo Controller This project uses this one from Amazon.
- Mounting Arm For holding the Pan and Tilt Kit, this project uses the VideoSecu 1/4" Security Camera Mount and the SMALLRIG Super Clamp w/ 1/4" and 3/8" Thread.
- Motor Controller This project uses the Sabertooth 2X12 Regenerative Dual Channel Motor Controller which can be found at RobotShop.
- Li-Po Battery To power the motor controller. This project uses a 3S 11.1V 6000 mAh LiPo battery with an XT60 Plug. Search on Amazon or eBay for deals. These batteries are known to cause fires, so please be aware of the risks and proper handling procedures.
- Wiring Harness w/ Switch To connect Motor Controller to Li-Po Battery. This project uses XT60 Plugs. This may come with your rover kit. If one doesn't, you'll need to pick this up or something similar and replace the installed plug with the appropriate plug type for your battery.
- 5V 2.5A Switching Power Supply w/ MicroUSB Connector Adafruit Link. To power Raspberry Pi directly.
- 5V 2A Power Supply w/ 2.1mm Jack Adafruit Link. To power PWM/Servo Controller directly.
- Female DC Power Adapter - 2.1mm Jack Adafruit Link. To connect Power Supply to PWM/Servo Controller.
- USB Adapters Amazon Link To mount the NCS2 sticks onto the Raspberry Pi. Process used: rotated the adapters into desired position and used hot glue to secure the positioning.
Robot top plate partially off to display the motor controller and Li-Po battery inside:
Top-view of the robot in the testing/tweaking setup:
Robot on top of books in the testing/tweaking setup to restrict base movement:
Wire connection to PWM/Servo Controller:
Wire connection from PWM/Servo Controller to Raspberry Pi:
Sabertooth DIP switch settings (1, 2, 3 - DOWN and 4, 5, 6 - UP):
Full Top-View of Porky with the top plate off (shows: Sabertooth motor controller connection to Raspberry Pi):
Note: wire diagrams will be added in the future.
After performing adequate hardware and software tests, you'll be ready to release your autonomous robot without its leash. This section will show you how to configure your robot to be deployed live.
This setup consists of:
- See Tweak and Test Setup for bulk of components (minus the wall power supplies and adapters).
- Portable Powerbank Be aware that not all portable chargers are compatible for Raspberry Pi projects. This project uses this RAVPower Portable Charger.
- 4 x AA Battery Holder /w On/Off Switch Adafruit Link. To power the PWM/Servo Controller.
- 4 x AA Batteries
- USB C to MicroUSB Cable Amazon Link To connect the Raspberry Pi to the Powerbank.
Pretty much the tweak and test setup, but without the wall supplies/adapters and additional portable power delivery devices:
Note: wire diagrams will be added in the future.
The goal of this section is to use TensorFlow to train your custom model using transfer learning. While creating your own machine learning model from scratch can be extremely rewarding, that process typically involves much more configuration, troubleshooting, and training/validating time... which can be a costly process (1.5 hours with my training pipeline on Google Cloud Platform cost ~$11 USD). However, with transfer tearning, you can minimize all three fronts by choosing an already proven model to customize with your own dataset.
The following guides were used as reference for the machine learning sections:
- TensorFlow Object Detector API Readme
- How to train your own Object Detector with TensorFlow’s Object Detector API
- Creating your own object detector
Please read the above links to fill in missing gaps while this guide is updated and to get more examples of how you can use TensorFlow's Object Detection API.
First, you'll want to create your own dataset. You can do this by utilizing popular public datasets or by creating your own. I chose to create my own dataset for this project in an attempt to create a more unique classification. This process basically follows two steps: gather your data into a collection (with proper filenames to help organization, ie: piggy-1.png, piggy-2.png, etc) and label/annotate your data (label the regions of interest, ie: drawing a rectangle on the object you're classifying in the image and label it appropriately).
Capture Images with the Image Capturing Setup
This step isn't absolutely necessary to follow verbatim, you can also use images from a public dataset like ImageNet. Configure the hardware as described within the Image Capturing Setup and find the image_capture.py script within the dataset folder.
- Navigate to the dataset directory:
pi@raspberrypi:~$ cd ./Porky/dataset- Run the image capturing Python script:
pi@raspberrypi:~/Porky/dataset $ python3 image_capture.py -picture_directory=~/PathYourImageDirectory -picture_label=PathYourPictureLabel-
Capture images by pointing the camera at a subject and pressing the mini-button (which is connected to the breadboard) to take the picture. The pictures will be saved within the directory that was specified and will automatically increment the image label based on the number of images already contained within the folder.
-
After you're satisfied with the amount of images you've taken, create two folders: /train and /test within your image directory and place about 80% of your total images within the /train directory and the remaining images within the /test directory. Click this StackOverflow link to find out more about the 80/20 split.
This process consists of labelling/annotating your images in a format readable by TensorFlow (this project utilizes the Pascal VOS format).
- Install and launch LabelIMG. GitHub Link
- Click 'Change default saved annotation folder' in Menu -> File and choose the directory you want your 'train' annotations to be saved in.
- Click 'Open Dir' and choose the directory that contains your 'train' images.
- Click on an image to annotate.
- Click 'Create RectBox'
- Click and drag a rectangular box over the portion of the image you want to classify and release the mouse button when you've outlined the region of interest.
- A pop-up window will display that will prompt you to input a label for the region of interest that you outlined. Input the label and press the 'Ok' button or hit the 'Enter' key on your keyboard.
- Repeat steps 4 through 7 until you've labelled all of the images within the directory.
- Repeat steps 2 through 8 for the 'test' portion of your dataset.
Once you've gathered and labelled your dataset, install TensorFlow onto your dev PC (if you haven't already).
From PowerShell (if developing from a Windows PC) install TensorFlow to your Python Environment (virtual preferred - will be updated in the future):
PS C:\> pip install tensorflowNow install the TensorFlow models repository to your Dev PC:
PS C:\> cd PathToPreferredDirectory
PS C:\> git clone https://github.com/tensorflow/models.gitThe TensorFlow models repository will contain useful configuration scripts to configure your machine learning pipeline.
See Dat Tran's repository for the xml_to_csv.py script utilized for this step. The modified version of this script is contained within the dataset directory of this project's repository. See Gilbert Tanner's article on how those modifications came to be.
After the modifications are made, use the following command:
PS C:\Porky\dataset> py xml_to_csv.pyThis well create two CSV files: train_labels.csv and test_labels.csv.
This step requires two things to be done: your captured images need to be seperated into a two directories (train and test) and you'll need two corresponding csv files that contain your labels/annotations in Pascal VOC format.
First, modify the script, generate_tfrecord.py (Dat Tran's repository) to fit your labels. See the modified version of this script contained within the dataset directory of this project's repository. After the script has been modified, run the following commands:
To convert your 'train' images and labels to TFRecord format:
PS C:\Porky\dataset> py generate_tfrecord.py --csv_input=PathToLabelsCSVFile\train_labels.csv --image_dir=PathToImageDirectory\train --output_path=train.recordTo convert your 'test' images and labels to TFRecord format:
PS C:\Porky\dataset> py generate_tfrecord.py --csv_input=PathToLabelsCSVFiles\test_labels.csv --image_dir=PathToImageDirectory\test --output_path=test.recordTo save some cost and time, you can pick out an already trained machine learning model to use for your customized dataset. The following two bullet points will help you in the process of choosing an appropriate model:
This project uses the ssd_mobilenet_v2_coco model. It's not listed as an officially supported Myriad model (which I learned after the fact), but I was lucky in the case that it actually worked with the Myriad plugin.
If you have access to a capable GPU, I suggest performing Machine Learning locally. However, if you're like me and don't have immediate access to a capable GPU, you can use a cloud compute service to perform your Machine Learning for you. For this project, I used the Google Cloud Platform to perform the TensorFlow training.
Please follow the following link to guide you through this process. Be aware that there are frustrations dealing with depcrecated functions and bash commands within the Windows platform. In a future update, I will thoroughly detail the process I used via Windows 10 in this guide. In the meantime, feel free to provide some feedback on any issues you incur and I will attempt to help you as best as I can.
Another useful guide from TensorFlow: Quick Start: Distributed Training on the Oxford-IIIT Pets Dataset on Google Cloud
Once you're satisfied with accuracy of your machine learning session, you can kill the TensorFlow process and extract the latest checkpoints for your trained model. If you used the Google Cloud Platform, the checkpoint files will be contained within your storage bucket.
A checkpoint will typically consist of three files:
- model.ckpt-${CHECKPOINT_NUMBER}.data-00000-of-00001
- model.ckpt-${CHECKPOINT_NUMBER}.index
- model.ckpt-${CHECKPOINT_NUMBER}.meta
After training the machine learning model with TensorFlow, you're now ready to prepare the model and convert it to an Intermediate Representation (IR). This will allow the model to be utilized with the MYRIAD Plugin (Intel NCS2) and therefore be deployed live with a combination of a Raspberry Pi 3 B+ and an Intel Neural Compute Stick.
Please read the following guides as a precursor to the next steps:
- TensorFlow Guide on Exporting Models
- OpenVINO Guide for Converting TensorFlow Models to Intermediate Representation
- Copy the latest checkpoint to the cloned TensorFlow models\research directory
- Execute the following command within PowerShell:
PS C:\models\research> py .\object_detection\export_inference_graph.py `
>> --input_type image_tensor `
>> --pipeline_config_path C:\PathToYourPipelineConfigFile
>> --trained_checkpoint_prefis model.ckpt-PREFIXNUMBER `
>> --output_directory PathToOutputDirectoryThis command will output multiple files to your specified output directory, we will be using the frozen_inference_graph.pb file for our next step.
If you haven't done so already, install OpenVINO on your Dev PC.
Navigate to the installed Intel OpenVINO directory and execute the following command within PowerShell:
PS C:\Intel\computer_vision_sdk_2018.5.456\deployment_tools\model_optimizer> py .\mo_tf.py `
>> --input_model C:\PathToYourFrozenTFModel\frozen_inference_graph.pb `
>> --tensorflow_use_custom_operations_config C:\Intel\computer_vision_sdk_2018.5.456\deployment_tools\model_optimizer\extensions\front\tf\ssd_v2_support.json `
>> --tensorflow_object_detection_api_pipeline_config C:\PathToYourPipelineConfigFile
>> --data_type FP16Take note of the line: --data_type FP16, the Myriad VPU (Neural Compute Stick (2)) currently only supports 16-bit precision. If the line is left out, the converted model will not work with your compute stick(s).
Executing this script will output 3 files into the directory you ran the command from:
- frozen_inference_graph.bin (model weights)
- frozen_inference_graph.mapping
- frozen_inference_graph.xml (model configuration)
We're primarily looking for the model weights (.bin) and config (.xml) files for deployment.
Now that you have your IR Model, you can now deploy it into a script by using OpenCV and/or the OpenVINO SDK.
If you're unfamiliar with the Raspberry Pi platform, follow this official guide to set your Pi up. Be sure to download Raspberian for your OS.
The next step is to install OpenVINO on your Raspberry Pi, please follow this guide to do so.
You should see the following message within your Raspberry Pi terminal once you've completed the install:
[setupvars.sh] OpenVINO environment initialized
Connect to your Raspberry Pi (via SSH, RealVNC, or locally) and navigate to your preferred directory to store projects in. Then perform a git clone within the terminal:
pi@raspberrypi:~$ git clone https://github.com/keith-E/Porky.gitIf you've trained your own machine learning model, replace the frozen_inference_graph.bin and frozen_inference_graph.xml files within the src directory with your own. If you didn't train your own model, you can utilize the provided model... but be aware that the provided model was trained on a stuffed piggy 🐖 and may not give you the best results.
During the lifecycle of your robot project, it's a good idea to develop and maintain some sort of testing strategy. This section demonstrates how to use the provided testing scripts and their purpose.
To test if the camera is powering on correctly:
- Ensure USB cable is properly connected to the Raspberry Pi.
- Provide power to the Raspberry Pi.
- Wait a couple seconds.
- The blue LED on the right of the camera (facing the lens/microphone array) should light up.
- Type the following command within your Raspberry Pi terminal:
pi@raspberrypi:~$ lsusb- You should see a USB device listing similar to the following if you're using a PS3 Eye Camera:
Bus 001 Device 006: ID 1415:2000 Nam Tai E&E Products Ltd. or OmniVision Technologies, Inc. Sony Playstation EyeTo test if the camera is providing good feedback:
- Ensure the USB camera is connected.
- Navigate to the Porky/tests/ directory.
- Run the following script:
pi@raspberrypi:~/Porky/tests $ python3 camera_test.py- A window should eventually pop up if you're accessing the Raspberry Pi's display.
- Press the 'q' key on your keyboard to quit the script.
To test the servos:
- Ensure the PWM/Servo Controller is connected to the Raspberry Pi properly and external (battery or wall) is being delivered to the controller.
- Navigate to the Porky/tests/ directory.
- Run the following script:
pi@raspberrypi:~/Porky/tests $ python3 pan_and_tilt_test.py- The terminal will display the test status.
- While the test is running, observe the servos and ensure they are moving to the appropriate positions.
To test the motors:
- Ensure all of the connections are properly wired.
- Ensure the power switch is turned in the 'on' position and the Sabertooth motor-controller LEDs are lit up.
- Navigate to the Porky/tests/ directory.
- Run the following script:
pi@raspberrypi:~/Porky/tests $ python3 motor_test.py- The terminal will display the test status.
- While the test is running, observe the motors and ensure they are moving in the correct directions.
Configure your robot (see: Live Deployment Setup) and ensure the following if you've built something similar:
- All bolted connections are properly tightened.
- The Raspberry Pi is powered on via the portable powerbank.
- The PWM/Servo Controller is powered on via the 4xAA Battery Holder.
- The Motor-controller is switch is turned on and powered via the Li-Po battery.
If you've built an identical robot, issue the following command via a terminal:
pi@raspberrypi:~$ cd ~/Porky/src/
pi@raspberrypi:~/Porky/src $ python3 run.pyIf your robot does not utilize a Pan and Servo Kit and/or Motors, you can run the program without those processes:
pi@raspberrypi:~/Porky/src $ python3 run.py --pantiltstate 0 --motorstate 0
- Development
- The most complex process was the machine learning portion by far.
- Frustration was mostly observed while attempting to configure and train a model on TensorFlow and the Google Cloud Platform. This is most likely due to the guides I followed being outdated and using Linux as their platform.
- Performance
- The object detection for the first iteration of the piggy model is not as accurate and fast as I would prefer. This is most likely due to the lack of diversity within the dataset. I'll need to be more observant about gathering/capturing more lighting situations and overall environments in the future.
- The piggy detection model works best where more natural light is present.
- The object detection in the current iteration is very rebound-y and which causes the motors (and therefore the whole bot) to become very jittery.
I tried my best to detail all of the processes I used to get this project off the ground, but I may have missed some key steps along the way or you may have experienced some frustrations trying to follow along. With that being said, please don't hesitate to drop me any comments, questions or concerns. I promise to do my best to address your issues.
Professor Becker and CS390 For guiding and permitting this class project.
leswright1977/Rpi3_NCS2: leswright1977's bottle-chasing robot introduced me to the Intel NCS2 and its ability to integrate machine learning models for real-time applications.
PINTO0309: PINTO0309's MobileNet-SSD-RealSense project introduced me to using multiprocessing with OpenCV and Intel's CNN backend in order to achieve faster results.
Fritzing - An Open Source Diagram Design Tool
TensorFlow Object Detector API Readme
How to train your own Object Detector with TensorFlow’s Object Detector API: This article was great for providing a great easy-to-follow reference for creating the dataset(s) for this project. The xml_to_csv.py and generate_tf_record.py scripts were also utilized from the author's github repository
Creating your own object detector: This article provided a good reference and filled some blanks for preparing a dataset to be utilized for TensorFlow training. With a combination of the official TensorFlow documentation (Object Detector API Readme), Dat Tran's article (provided above), and this article, I was able to successfully train a customized machine learning model with TensorFlow and transfer learning.
Running TensorFlow on the Cloud
OpenCV Docs: The official documentation for OpenCV. Necessary for gaining a strong foundation of using OpenCV to build your application.
Adafruit Pixy Pet Robot: Adafruit's guide on creating color vision following robot using a Pixy CMUCam-5 vision system and Zumo robot platform. This guide was very helpful for learning how to integrate a PID (Proportional-Integral-Deravitive) control feedback loop for the motion mechanisms.
PyimageSearch Adrian provides great tutorials that are in-depth and easy to follow. This website is a great resource to learn about applying computer vision to your next project.