Experiment Design

This week, I hope to write my own usability study, similar to the study in "Interactions and Training with Unmanned Systems and the Nintendo Wiimote". Here are some of the basic ideas/changes I would implement:

-The obstacle course would involve tasks like moving blocks from place to place, and steering around specially marked areas.
-The participants would still receive training, and then perform the obstacle course 3 times per control scheme.
-Participants would navigate the robot through the obstacle course using 2 (rather than 3) control schemes: iPhone and "controller" (some sort of standard controls for the specific type of robot being driven).
-The questionnaire would include the following questions (5-level Likert scale):

1. Instructions on how to perform gestures were clear

2. Gestures were easy to learn

3. Gestures were easy to remember/recall

4. Gestures were easy to perform

5. The Gestures performed did not cause my hands, arms, shoulders and/or wrists to become fatigued

6. The iPhone was easy to hold and use

7. The system recognized the gestures accurately

8. Controlling the vehicle through gestures was easy

9. Controlling the vehicle through the iPhone was easy

10. Controlling the vehicle through the "controller" was easy

The study will be very similar to the one from the paper. Eventually, there will be more detailed outlines of the study's procedures. Stay tuned, and as always, any suggestions are welcome!

Lauren

Paper: Daniel Barber

Eventually (hopefully this semester), there will come a time when the iOS Gesture Controls become a fully-fledged, standalone system. At that time, we will hopefully reach out to our friends at UCF to do some user studies in order to gauge the effectiveness of the system. There has been a study already in Wiimote-based Gesture Controls, called "Interactions and Training with Unmanned Systems". This study, because it is very similar to the iOS project, will be the basis for my study when I design it.

"Interactions and Training with Unmanned Systems" used a Wiimote, rather than an iPhone to control the robot. There were three different control schemes implemented: Traditional Joystick Control, Accelerometer/Motion Steering Wheel Style Control, and Hand and Arm Gestural Control. The joystick control utilizes only the D-pad, for thrust/steering. The accelerometer control is similar to the the controls from Mario Kart Wii (using the Wiimote sideways like a steering wheel, pressing 2 to accelerate). The main difference is that there is an added axis of rotation: tilting the remote along the Z-axis controls the speed at which the robot moves. The gestural control is the most similar to the iOS controls. Their system uses a machine learning algorithm comprising a linear classifier and 29 features based on a 2D pen-based gestures. They used 4 gestures: Move Forward, Turn Left, Turn Right, Stop.

The usability study that the researchers conducted involved using each of the three control styles to navigate a robot through an obstacle course. The runs were timed, and then the participant complete the following questionnaire (5-level Likert scale):

1. Instructions on how to perform gestures were clear

2. Gestures were easy to learn

3. Gestures were easy to remember/recall

4. Gestures were easy to perform

5. The Gestures performed did not cause my hands and/or writes to become fatigued

6. The Wiimote was easy to use

7. The system recognized the gestures accurately

8. Controlling the vehicle through gestures was easy

9. Controlling the vehicle through the Wiimotes's motion tracking was easy

10. Controlling the vehicle through the Wiimote's directional pad was easy

They also ranked their preferred controls, and gave additional feedback about their experience.

Controlling PR2

Hello, everyone! Last week, my aim was to control a PR2 model in Unity with my iPhone. The model moved forward at a set pace and as directions ("N", "SW", etc.) were received from the phone, the model adjusted its rotation to match. See the video below for more details.

There are several problems with this approach. One is that there is no GPS data currently being sent from the iPhone to the server. The data that is currently being sent is heading (0-360 degrees), and the data from the phone's 3 accelerometers. Another problem is that the data is being sent to the server over WiFi, which is sometimes slow and unreliable. This is a problem because we aim to control the robot in real time, so lag or lost  commands could be very dangerous. The third problem is that the "commands" are currently determined by a certain amount of acceleration in any direction. There are no "gestures", merely a recognition when the phone accelerates above a certain threshold.

I hope to fix the first two issues for this week by sending more information (like GPS coordinates) to the server, and by using Bluetooth communication instead of WiFi. The third issue (gestures) will be a larger portion of the project, done over the coming weeks.

Shake It Like a High Pass Filter

Hey everyone,

Last week, I got my feet wet with the iPhone accelerometer and compass data. To get started, I checked out some sample code from Apple's iOS Developer Library. I decided to use the AccelerometerGraph sample code as a basis for my code. 

The AccelerometerGraph app in action. AccelerometerGraph shows two graphs, one of the raw data , and one of the filtered data.

AccelerometerGraph samples the X, Y, and Z accelerometers at 60Hz (displayed as Red, Green, and Blue, respectively). The acceleration values are measured in units of g-force, with a value of 1.0 approximately equal to gravity acting on that axis of the phone.There was no minimum acceleration threshold in the original app; every second exactly 60 measurements were logged and graphed. The app provides options for Low Pass/High Pass and Standard/Adaptive filtering. Low pass filters allow the lower values to go through, while filtering out the higher values. In terms of acceleration, this means that quick, sharp movements will be filtered out, while steady sources of acceleration like gravity or larger movements will show up in the graph. High pass filters do the opposite. They allow the higher values to go through while filtering out the lower values, which means that the effects of gravity will be filtered out, leaving just the shorter, sharper movements in the graph. An adaptive filter is one that self-adjusts based on an error signal to attenuate noise while leaving the original signal intact.

When modifying AccelerometerGraph for gesture recognition, I added a minimum threshold. Any measurement where none of the accelerometers recorded a value higher than 0.05 was thrown out. The Adaptive High Pass Filter worked the best for filtering out both the effects of gravity and of small incidental movements like shifting from side to side. The image above shows the Adaptive High Pass Filter in action; the erratic measurements in the top graph are "toned down" to become the measurements in the bottom graph.

When an acceleration passes the threshold, I also get the current heading from the compass. The heading is returned in the form of a float from 0-360 degrees, where 0 is North, 90 is East, etc. These directions are usually based on true North, but if that data is not available then magnetic North is used. Heading is measured with respect to the top of the phone, so some trigonometry may be required in the future if the phone is tilted.

Once the heading measurement is recorded, I used the TouchJSON library to encode the three accelerations and heading as a JSON string. Using the ASIHTTPRequest library, I sent the JSON to a Node.js backend (currently running on my laptop), where the heading value is converted to a string value like "N", "NW", "NNW", etc. For next week I hope to replace the Node backend with Unity/C#. In the long term, I also hope to develop more rigorous ways of detecting a gesture in addition to the threshold and filters. I will be reading up on the iOS shake gesture recognition code.

First Post

Hello, everyone! This is the inaugural post in my Master's Thesis blog. I'll be detailing my progress every week, and maybe even answering a few questions.

The high-level goal for the project is to use the iPhone accelerometers and compass to read certain gestures (e.g. pointing in a direction) from the motion of a person holding the phone. Those gestures, once parsed, can be sent to a server and used to control a variety of robotic vehicles.

The three iPhone accelerometers.

To start the project off, I'll be pulling data from the iPhone accelerometer/compass and streaming it to the computer. I'll also be spending a lot of time looking at Apple's iOS Developer documentation and Unity to see if there are additional technologies that we can utilize.

That's all for now, tune in next week.