IRBs and Questionnaires

This week was dedicated to filling out IRB forms and working on the questionnaire that will be given after the experiment. This marks the beginning of the serious thinking about experiment design. I decided that every subject should perform the obstacle course three times: once with on-screen controls, once with tilt-based controls, and once with hand gestures.

The questionnaire aims to explain why subjects performed the way they did during the study. There is a short section about video game fluency, especially console games, to figure out which subjects are gamers and which are not. Most of the questions will be changed to 5-level Likert scale questions and the questionnaire will be put online via SurveyMonkey or Google Docs (link to come).

In terms of the system itself, I purchased an arm band for the experiments, and will hopefully try it out over the next week. My hope is that being able to attach the phone in some way to the subject will reduce the awkwardness of handling the controls, since the phone is very large/flat.

Tune Belt armband

The hand gesture based controls are not complete, but it is more important to decide on which gestures will be used. Many of them will probably be pulled from the 1987 Army Visual Signals Field Manual.

Post-Break Update

There have been quite a few developments since Winter Break. First, on the administrative side, the proposal was submitted, including a draft of an abstract and title:

Using smartphones for gesture-based control of robotic systems
Robotic control systems are becoming more common, especially in the military. With military applications, there are lives at stake, so having the most efficient, intuitive control system can make a large difference in the success of a mission and the safety of the soldiers involved. Arm and hand gestures are typical human forms of communication, so applying that to a robotic control system can yield a more intuitive system. Varcholik et. al. describe a gesture based control system that uses the Nintendo Wiimote to determine arm/hand gestures to control a robot. In this thesis, I propose the use of smartphones for gesture-based control of robotic systems. The proposed controller will be evaluated by performing a set of carefully designed human factors experiments and computing a set of metrics (e.g. time taken to complete tasks) to measure the efficacy of the gesture-based control system.

The thesis committee will consist of Prof. Badler, Prof. Lane, and Prof. Daniilidis. I also took Winter Break as an opportunity to purchase a Roomba 560, RooTooth dongle, XBOX 360 controller, and a Samsung Galaxy II for use with the project.

Samsung Galaxy II

Roomba 560

Next, the robotics side! After several days of working through the Roomba Open Interface Specification, the Roomba finally accepts commands through the RooTooth Bluetooth dongle. It can now be controlled through the XBOX controller, keyboard arrow keys and command line, using scripts that I wrote with the RoombaSCI python library. I also downloaded the Cellbots Android app so there are several more phone-based direct control schemes that are working out of the box (see first part of the video).

Cellbots for Android. I plan to use the D-Pad and Tilt controls from Part 1.
Cellbots provides an API for connecting Android phones to robots like the Roomba via Bluetooth, so I will almost certainly be using it to write the gesture-based app. I will also look into using this framework, which provides gesture recording and recognition as an Android service.

Once the "hand waving" app is complete, I can start to develop experiments involving several control methods to determine which scheme people would rather use.

Experiment Design

This week, I hope to write my own usability study, similar to the study in "Interactions and Training with Unmanned Systems and the Nintendo Wiimote". Here are some of the basic ideas/changes I would implement:

-The obstacle course would involve tasks like moving blocks from place to place, and steering around specially marked areas.
-The participants would still receive training, and then perform the obstacle course 3 times per control scheme.
-Participants would navigate the robot through the obstacle course using 2 (rather than 3) control schemes: iPhone and "controller" (some sort of standard controls for the specific type of robot being driven).
-The questionnaire would include the following questions (5-level Likert scale):

1. Instructions on how to perform gestures were clear

2. Gestures were easy to learn

3. Gestures were easy to remember/recall

4. Gestures were easy to perform

5. The Gestures performed did not cause my hands, arms, shoulders and/or wrists to become fatigued

6. The iPhone was easy to hold and use

7. The system recognized the gestures accurately

8. Controlling the vehicle through gestures was easy

9. Controlling the vehicle through the iPhone was easy

10. Controlling the vehicle through the "controller" was easy

The study will be very similar to the one from the paper. Eventually, there will be more detailed outlines of the study's procedures. Stay tuned, and as always, any suggestions are welcome!

Lauren

Paper: Daniel Barber

Eventually (hopefully this semester), there will come a time when the iOS Gesture Controls become a fully-fledged, standalone system. At that time, we will hopefully reach out to our friends at UCF to do some user studies in order to gauge the effectiveness of the system. There has been a study already in Wiimote-based Gesture Controls, called "Interactions and Training with Unmanned Systems". This study, because it is very similar to the iOS project, will be the basis for my study when I design it.

"Interactions and Training with Unmanned Systems" used a Wiimote, rather than an iPhone to control the robot. There were three different control schemes implemented: Traditional Joystick Control, Accelerometer/Motion Steering Wheel Style Control, and Hand and Arm Gestural Control. The joystick control utilizes only the D-pad, for thrust/steering. The accelerometer control is similar to the the controls from Mario Kart Wii (using the Wiimote sideways like a steering wheel, pressing 2 to accelerate). The main difference is that there is an added axis of rotation: tilting the remote along the Z-axis controls the speed at which the robot moves. The gestural control is the most similar to the iOS controls. Their system uses a machine learning algorithm comprising a linear classifier and 29 features based on a 2D pen-based gestures. They used 4 gestures: Move Forward, Turn Left, Turn Right, Stop.

The usability study that the researchers conducted involved using each of the three control styles to navigate a robot through an obstacle course. The runs were timed, and then the participant complete the following questionnaire (5-level Likert scale):

1. Instructions on how to perform gestures were clear

2. Gestures were easy to learn

3. Gestures were easy to remember/recall

4. Gestures were easy to perform

5. The Gestures performed did not cause my hands and/or writes to become fatigued

6. The Wiimote was easy to use

7. The system recognized the gestures accurately

8. Controlling the vehicle through gestures was easy

9. Controlling the vehicle through the Wiimotes's motion tracking was easy

10. Controlling the vehicle through the Wiimote's directional pad was easy

They also ranked their preferred controls, and gave additional feedback about their experience.

Controlling PR2

Hello, everyone! Last week, my aim was to control a PR2 model in Unity with my iPhone. The model moved forward at a set pace and as directions ("N", "SW", etc.) were received from the phone, the model adjusted its rotation to match. See the video below for more details.

There are several problems with this approach. One is that there is no GPS data currently being sent from the iPhone to the server. The data that is currently being sent is heading (0-360 degrees), and the data from the phone's 3 accelerometers. Another problem is that the data is being sent to the server over WiFi, which is sometimes slow and unreliable. This is a problem because we aim to control the robot in real time, so lag or lost  commands could be very dangerous. The third problem is that the "commands" are currently determined by a certain amount of acceleration in any direction. There are no "gestures", merely a recognition when the phone accelerates above a certain threshold.

I hope to fix the first two issues for this week by sending more information (like GPS coordinates) to the server, and by using Bluetooth communication instead of WiFi. The third issue (gestures) will be a larger portion of the project, done over the coming weeks.

Shake It Like a High Pass Filter

Hey everyone,

Last week, I got my feet wet with the iPhone accelerometer and compass data. To get started, I checked out some sample code from Apple's iOS Developer Library. I decided to use the AccelerometerGraph sample code as a basis for my code. 

The AccelerometerGraph app in action. AccelerometerGraph shows two graphs, one of the raw data , and one of the filtered data.

AccelerometerGraph samples the X, Y, and Z accelerometers at 60Hz (displayed as Red, Green, and Blue, respectively). The acceleration values are measured in units of g-force, with a value of 1.0 approximately equal to gravity acting on that axis of the phone.There was no minimum acceleration threshold in the original app; every second exactly 60 measurements were logged and graphed. The app provides options for Low Pass/High Pass and Standard/Adaptive filtering. Low pass filters allow the lower values to go through, while filtering out the higher values. In terms of acceleration, this means that quick, sharp movements will be filtered out, while steady sources of acceleration like gravity or larger movements will show up in the graph. High pass filters do the opposite. They allow the higher values to go through while filtering out the lower values, which means that the effects of gravity will be filtered out, leaving just the shorter, sharper movements in the graph. An adaptive filter is one that self-adjusts based on an error signal to attenuate noise while leaving the original signal intact.

When modifying AccelerometerGraph for gesture recognition, I added a minimum threshold. Any measurement where none of the accelerometers recorded a value higher than 0.05 was thrown out. The Adaptive High Pass Filter worked the best for filtering out both the effects of gravity and of small incidental movements like shifting from side to side. The image above shows the Adaptive High Pass Filter in action; the erratic measurements in the top graph are "toned down" to become the measurements in the bottom graph.

When an acceleration passes the threshold, I also get the current heading from the compass. The heading is returned in the form of a float from 0-360 degrees, where 0 is North, 90 is East, etc. These directions are usually based on true North, but if that data is not available then magnetic North is used. Heading is measured with respect to the top of the phone, so some trigonometry may be required in the future if the phone is tilted.

Once the heading measurement is recorded, I used the TouchJSON library to encode the three accelerations and heading as a JSON string. Using the ASIHTTPRequest library, I sent the JSON to a Node.js backend (currently running on my laptop), where the heading value is converted to a string value like "N", "NW", "NNW", etc. For next week I hope to replace the Node backend with Unity/C#. In the long term, I also hope to develop more rigorous ways of detecting a gesture in addition to the threshold and filters. I will be reading up on the iOS shake gesture recognition code.

First Post

Hello, everyone! This is the inaugural post in my Master's Thesis blog. I'll be detailing my progress every week, and maybe even answering a few questions.

The high-level goal for the project is to use the iPhone accelerometers and compass to read certain gestures (e.g. pointing in a direction) from the motion of a person holding the phone. Those gestures, once parsed, can be sent to a server and used to control a variety of robotic vehicles.

The three iPhone accelerometers.

To start the project off, I'll be pulling data from the iPhone accelerometer/compass and streaming it to the computer. I'll also be spending a lot of time looking at Apple's iOS Developer documentation and Unity to see if there are additional technologies that we can utilize.

That's all for now, tune in next week.