Control Helicopoter with your hand glove

Introduction



The purpose of our project is to control a toy helicopter using flex sensors attached to a glove. The flex sensors are intended to replace the remote control that is generally used to fly the helicopter. Additionally we also created another mode which will allow us to use an accelerometer to control the forward and backward, and left and right movements, while using a flex sensor to control the throttle of the helicopter. We have two gloves each with three flex sensors attached to it. One of the gloves has an accelerometer attached to it.    

High Level Design

1. Rationale
One of our group members Zack ordered a helicopter from eBay for his own personal use. Once we started playing with it, we thought it would be a cool idea to try to control the helicopter without the controller and fly it using one's own body movements. Our initial idea was to attach electrodes to our arm to detect the flexing of the muscles on the forearm. Using the signals from the electrodes we thought that we would be able to send these signals to the helicopter and control it. However, Professor Land informed us that it might be dangerous to conncet electrodes to our arm and additionally the signals received from the electrodes might too small to feasibly use. Instead we decided an alternative would be to attach flex sensors to the hand and by bending our fingers we would be able to control the movements of the helicopter. This was how the Flexicopter was born!   

2. Background Information

Flex Sensors 

Flex sensors are basically variable resistors. As seen from the Figure 1 below, we will use a voltage divider circuit we to obtain the voltage across the flex sensor. As you bend the flex sensor, its resistance gets higher and hence the voltage across it gets higher. We will use the changing voltage to control the helicopter.

Figure 1: Flex Sensor Circuit

Accelerometer

An accelerometer measures how much you accelerate the device relative to the acceleration of free fall which is gravitational acceleration, g. Based on the relative acceleration of the device, it outputs a voltage. So as you move the accelerometer left, right, back and forward, the voltage changes. Similar to the flex sensors, we will use the varying voltage to control the helicopter.

Figure 2: Accelerometer

Helicopter

The way the helicopter works is that the Light Emitting Diode (LED) transmits bit streams repeatedly from the controller to the helicopter through RF waves. The receiver on the helicopter obtains these bit streams and moves the helicopter. Our setup mimicked this.

Figure 3: Helicopter

3. Logical Structure

Figure 4: Overall structure of our design

4. Hardware/Software TradeOffs

Our budget restricted the quality of helicopter we were able to use. We used a low quality fifteen dollar helicopter ordered from Hong Kong. This meant that we had no information about what kind of bit stream pattern the controller of the helicopter was using and we had to figure it out ourselves. Additionally, the low quality meant that the helicopter could easily be damaged. Our worst fears came true when our helicopter's stabilizer broke off two days before the final project was to be demoed because we had crashed it often when testing it. For the original helicopter, the bit stream pattern was too complicated for us to figure out any patterns they were following when we moved the helicopter backwards, forwards, left and right because there was some kind of error correction occurring and we could not figure out the algorithm. Hence we had to limit the motion of the helicopter to ten levels of throttle, and for each level of throttle we limited the helicopter to five levels of right,left, forward, backward, forward-right, forward-left, backward-right and backward-left. This still meant that we had to record 400 different bit streams, which included taking 400 pictures on the oscilloscope and translating them into a pattern that our code could translate send out. However, for the helicopter we ended up using after the original one broke, the bit stream patterns were much simpler and we were able to observe obvious patterns.

Software and Hardware Design

1. Hardware Design

As mentioned in our introduction, we allow the helicopter to be controlled in two modes. The first mode is the flex sensor mode, where only the flex sensors control the movement. In this mode, on the right hand, the flex sensor on the thumb controls the throttle, the flex sensor on the index finger controls the 'left' movement, the flex sensor on the middle finger controls the 'right' movement. On the left hand, the flex sensor on the index finger controls 'forward' motion while the flex sensor on the middle finger controls the 'backward' motion. The second mode is the accelerator mode, where the accelerator combined with on flex sensor controls the movement of the helicopter. Tilting the accelerator on the right hand up, down, to the left and to the right moves the helicopter forward, backward, left and right respectively. The flex sensor on the thumb of the left hand controls the throttle of the helicopter. These are summarized in Figure 5 shown below.

You can see the following components of  the glove from Figure 6 below. We sewed rectangular pockets of cloth on each individual finger of the glove in order to insert the flex sensors. A voltage divider circuit was built on a protoboard, which was then sewn on to the lower portion of the glove. Output wires from the protoboard are then connected to the microcontroller as inputs.

Figure 6: The Right Glove

The voltage divider circuit constructed on the protoboard is shown below in Figure 7.

   Figure7: Flex Sensor Circuits on the hands

The accelerometer and its associated circuit was placed on another protoboard and this was sewn onto the back of glove, on the palm of the hand as shown in Figure 2. We had to add additional circuitry to the accelerometer because the maximum voltage it could take was 3.6V and the VCC that we were sending to it from the microcontroller was 5V. Hence we needed a circuit that would step down the voltage from 5V to about 3V. We knew that the transistor posed a certain amount of impedence and we also knew that we had to have a voltage less than 3.6V at the emitter of the transistor which was then connected to the VCC input of the accelerometer. Instead of using theoretical equations to determine the resistances on the voltage divider circuit,  we used trial and error and determined that we needed 10 kiloOhms and 3 kiloOhms resistors. As shown below in Figure 8, the voltage at the base of the transistor will be about 3.84V. This gives us approximately 3.1V at the emitter of the transistor and this gets sent to the VCC of the accelerometer. 

Figure 8: AccelerometerCircuit

Since we did not want to make use of the STK500 board, we used a prototype board for the Mega644 and attached the Mega644 chip on it. We soldered all the diffferent components necessary onto the prototype board and then sewed it onto a piece of cloth which we then fashioned to wrap around our arm as shown in Figure 9. This way all the hardware necessary is on our body and nothing is detached. 

We also have the circuit which takes the bit stream pattern that has been indicated by the code based on which flex sensors we have bent and how much they have been bent, and sends it to the LED. Since we need the signal to be sent to the helicopter to be strong, we have to amplify the current going through the LED. The port pins limit the amount of current drawn to about 30 Milliamperes. We need at least 150 Milliamperes to be sent through the LED in order for the signal to be strong enough. The circuit we constructed is shown below in Figure 10. The basic idea of this circuit is that the current will be drawn from the 12 V voltage source that is connected to the LED. 

Figure 9: LED Circuit

2. Software design

The code is set up to operate in two modes: one which is controlled solely by the flex sensors and one which is controlled by the accelerometer and one flex sensor for throttle.  To choose between the two modes a digital input is read off of PORTC0, if the input is high the code operates in the accelerometer mode, when the input is low to this port the code operates in the flex sensor mode.  The code breaks down into two main sections for both modes: a section which chooses what signal to be sent, and a section that actually creates and sends the signal.   The signal to the helicopter consisted of high portions and low portions, of which, the high portions oscillate at a 37kHz rate.  After checking several bit streams, it became apparent that the signal was separated into six segments: start bits, 8-bits representing throttle, 4-bits representing right/left motion, 4-bits representing forward/backward motion, 2-bits representing channel a,b, or c, and the last three bits representing trim.  The start bits were a high signal, lasting 3280us and a low signal lasting 2120us.  These bits were consistent over all ranges of motion.  The next 8-bits in the signal were determined to be throttle.  The throttle increased linearly from 0x00 to 0xC9 based on how much the throttle was increased.  The four bits following that determined right and left motion.  If the controller was signalling maximum right, the bits would be 0x0 and if it was maximum left the bits would be 0xF.  The forward and back, which are the next four bits, work in a similar manner; when the controller was signaling maximum forward the bits were 0x0 and when it was maximum backward the signal was 0xF.  The two bits following that signify which channel the helicopter is operating on.  If the pattern was 0b00 then it meant channel a, 0b01 was channel b, and 0b10 was channel c.  Of the three trim bits the third to last bit was always a 0 and the second to last turned to one when left trim was pressed, and the last bit turned to one when the right trim was pressed.  To compile the signal from the inputs of the flex sensors and accelerometer, the flex sensors and accelerometer were tested and the maximum and minimum voltages were set in the code.  We set a linear relationship between the voltage coming in and the intensity of the throttle, right-left, and forward-back motion.  The flexData() and accellData() functions were called on according to the mode select, and these functions acquired the analog data from all the input ports which applied.  The input data was then used in the functions accelSignal() and flexSignal(), which actually compiled the signal to be sent in an array.  These functions were called on 100 times a second giving the system a signal to output.  

  

The function signalSender() combined with a compare match ISR on timer 0 were used to send the signal to the helicopter.  The ISR triggered at 74kHz, which is twice as fast as the 37kHz we were sending out. so we could set it up to toggle on match.  The signalSender() function goes through the 41 bit signal composed by accelSignal() or flexSignal(), sending the 37kHz signal when a high value was being sent and a zero when a low value was being sent.  When the full signal has finished sending it sends a low value for 10000 ticks to mimic the signal sent from the controller of the helicopter.  Only when this has completed does it start over, sending a new signal.  

3. Things tried that did not work out

The main problem that we had was that we broke our first helicopter.  This helicopter had a very complicated bit stream that had some sort of error correction code that was too difficult to figure out because we could not even tell what was representative of one's and zero's in the code.  Because of this issue we took 400 screen shots on the oscilloscope and then translated them into a bit stream that we compiled and sent out in our code.  The 400 hundred values consisted of doing 11 throttle values (no throttle to maximum throttle) and then within each of these ten throttles (zero throttle was the same for everything because it was off) we had 10 values going from forward to back, 10 values from right to left,  10 values from forward-left to back-right, and 10 values from forward-right to back-left.  Because we were already at 400 we were not able to go into changing the trim, which is an offset for the helicopter for spinning right or left.  The code was set up to choose out of a matrix the values based on the inputs from the flex sensors and the accelerometer.  We had this design set up and working with the flex sensors and accelerometer, however the helicopter was offset with  a spin and we could not correct this with our set up due to the lack of trim settings.  This spinning made it almost uncontrollable in the forward-back and left-right directions, although the throttle was easily controllable.  We did check the signal we were sending out on the oscilloscope against the pictures we had taken, and we were sending the correct signals out for the hand motions we were making.  Unfortunately, two days before we checked off our project the helicopter we were using crashed and the stabilizer broke off, completely inhibiting any control.  Luckily we were able to get another helicopter and decode its bit stream, which was significantly more simple than the previous helicopter's.  

Results of Design

    We compared the performance of flying our helicopter with the flex sensors and accelerometer with how the helicopter flies when we control it using the remote controller. It works almost exactly the same way. Our flex sensors are able to attain the maximum throttle speed as the controller does. While flexing our thumb does not provide as much range in the different levels of throttle as the controller does, we are still able to raise the helicopter up and down easily. When we use the controller to move the helicopter left or right, it mostly spins in the left-direction or right-direction and slowly moves to the left and right respectively. Our flex sensors display the same functionality. The flex sensors are able to move the helicopter forward and backwards visibly. When comparing the functionality of the accelerometer and the flex sensors, the flex sensors are able to control the helicopter better than the accelerometer. However, we feel that using the accelerometer is more intuitive than using the flex sensors, since tilting the accelerometer left and right moves the helicopter left and right, and tilting it up and down moves the helicopter forward and backward. The biggest difference between controlling the helicopter using the flex sensors and controlling it using the remote control would have to be distance that we are able to control the helicopter. We can only control the helicopter within a limited distance from the LED with our circuit. A more distinct disadvantage using our circuit is that we can only start the helicopter if the LED can be seen visibly from the helicopter. This means  for example we cannot have the starting position of the helicopter to be from the floor unless the LED is on the floor as well. We assumed that this was probably due to some additional changes in the bit stream patterns that we were not able to detect using the oscilloscope. 

Here is a video one of the team members Zack flying the helicopter using the flex sensors:

Conclusions

1. Expectations vs. Actual Results

    We did not meet our expectations that were previously set out at the beginning of this project.  We had assumed that we would be able to decode the bit stream quickly, however, with our first helicopter this was not the case.  We spent hours acquiring the different bit streams, only to have them prove useless when we broke the helicopter.  Our code ended up becoming a lot less complicated when the new helicopter was included in the project.  We were able to still utilize both an accelerometer mode and a flex sensor mode, and allow the user to choose between them by setting PORTC0 to zero (for flex sensor mode) and one (for accelerometer mode).  It was more difficult to control than originally anticipated because it takes a lot of concentration to keep the motions of your fingers separate.  

    One of the biggest problems we encountered was because of the flex sensors.  When we initially used them in our circuit, based on their resistances, they gave us a voltage range of ~0-3V.  However, over the course of the project, mostly at very unexpected times, flex sensors would lose their ability to change over the same range.  Our voltage ranges usually changed to ~0.4-0.6V making them practically unusable for our application.  One of the flex sensors ended up randomly working, but apart from that it was a very cumbersome problem.  Since we decided to only use six flex sensors it turned out fine in the end, since we ended up having about three flex sensors that no longer worked properly.  Our code was set up to allow for quick changes in calibration values depending on what flex sensor we are using.  We developed some code that was able to calibrate the flex sensors however we had to rewrite a lot of our code to make up for the fact that towards the last 3 days of our project several unfortunate events happened causing us to have to redo several aspects of the hardware and code.  We ended up leaving the calibration to be a pre-compile set up.

2. Usability

    We feel that our flexicopter can be used by most people with good hand coordination. It takes a while to get used to controlling the helicopter using the flex sensors and accelerometer. It is a steep learning curve, but we feel that once that curve has been overcome, users will thoroughly enjoy the feeling of controlling a helicopter using their own hands!

3. Appendix 

Code

#include <avr/pgmspace.h>

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <stdint.h>

#include <inttypes.h>

#include <avr/io.h>

#include <avr/interrupt.h>

#include <math.h>

#include <util/delay.h>

#include <avr/sleep.h>

 

//I like these...me too

#define begin {

#define end   }   

//The various sensors

unsigned char sensor1,sensor2,sensor3,sensor4,sensor5,sensor6,sensor7,sensor8;

//The pointers to the sensors

char *throttleP, *forwardP, *backP, *leftP, *rightP, *throttleAccelP, *rightleftP, *forwardbackP; //P for pointer

//Increment for which sensor to pick

char sensorCounter;

volatile unsigned int count;

unsigned int signalPointer; //pointer to count through one signal

char bitType;

char bitSent, turnSel;

//volatile unsigned int inputCount; //count for signalSender to be called

//accelerometer declarations

#define accelSampTime 768

volatile unsigned int accelCount;

//Declaration of variables for flex sensors high and low values

unsigned int throttleHighAccel, throttleLowAccel; //Throttle accelerometer  high/low

unsigned int throttleHighFlex, throttleLowFlex; //Throttle flexsensor high/low

unsigned int leftHigh, leftLow; //Left high/low

unsigned int rightHigh, rightLow; //Right high/low

unsigned int forwardHigh, forwardLow; //Forward high/low

unsigned int backwardHigh, backwardLow; //Back high/low

unsigned int leftHighAccel, leftLowAccel; //Left high/low for accelerometer

unsigned int rightHighAccel, rightLowAccel; //Right high/low for accelerometer

unsigned int forwardHighAccel, forwardLowAccel; //Forward high/low for accelerometer

unsigned int backwardHighAccel, backwardLowAccel; //Back high/low for accelerometer

//Bits within the bit stream decode

#define sm0 3 //Small 0 bit

#define sm1 0 //Small 1 bit

#define bg1 1 //Big 1 bit

#define bit1 5  //first 1 bit 

#define bit2 6  //second bit 0 bit

//Bit lengths in half periods of 37.0kHz

#define sm0len 37  //~520us

#define sm1len 61  //~860us

#define bg1len 123 //~1720us

#define OneBit 234 //~3280us for first bit 1

#define TwoBit 151 //~2120us for second bit 0 

//Define Time between sampling flex sensors

//#define flexSampTime 3840 //allows for the data to be read 20times/sec using timer0

//#define flexSampTime 1536 //allows for the data to be read 50times/sec using timer0

#define flexSampTime 768 //allows for the data to be read 100times/sec using timer0

//Time count down to trigger reading the values from the flex sensors 20times/sec

volatile unsigned int flexCount; 

char allBits[41] = {5,6,0,3,0,3,0,3,0,3,0,3,0,3,0,3,0,3,1,3,0,3,0,3,0,3,1,3,0,3,0,3,0,3,1,3,0,3,0,3,0};

//*********************************************************** 

//Singnal Sender sends the signal to the IR emitter

void signalSender(void)

begin

//When the signalPointer is not 20 then we are transmitting a bit stream

if (signalPointer < 41)

begin

//If you finished sending the last bit you can move on

if (bitSent)

begin

bitType = allBits[signalPointer];

bitSent = 0; //Mark the bit as unsent

if ((bitType == sm0) || (bitType == bit2)) //if its zero or 

begin

TCCR0A = 0b00000010; // turn off OC0A and clear-on-match because sending a low value

PORTB = 0x00; 

end

else

begin

TCCR0A = 0b01000010; // turn on toggle and clear-on-match because sending a high value

end

end

//check if bit has completed 

else if (((bitType == sm0) && (count == sm0len)) ||

((bitType == sm1) && (count == sm1len)) ||

((bitType == bg1) && (count == bg1len)) ||

((bitType == bit1) && (count == OneBit)) ||

((bitType == bit2) && (count == TwoBit)))

begin

count = 0; //reset count

bitSent = 1; //update bitSent to true 

signalPointer++;

end // if bit complete

end // if signalPointer less than 41

else //if signalPointer 41

begin

if (count >= 10000)//8646)

begin

signalPointer = 0;

count = 0;

TCCR0A = 0b01000010; // turn on toggle and clear-on-match because we have held the signal low for 10000

end

else 

begin

TCCR0A = 0b00000010; // turn off OC0A and clear-on-match because sending low signal for 10000

PORTB = 0x00;    //turn off output between signals

end

end

end

//***********************************************************

//timer 0 comare match ISR for the signalSender

ISR (TIMER0_COMPA_vect) 

begin

//Count how many half periods the 37kHz signal has gone through

count++; 

signalSender();

flexCount--; //Decrement flexCount each time

accelCount--;

end  

//***********************************************************  

//***********************************************************        

void flexSignal()

begin

///////////////////

////THROTTLE///////

///////////////////

double throttlTemp1,throttleTemp2;

int integerThrottle;

integerThrottle = ((201/(throttleHighFlex - throttleLowFlex)) * (*throttleP)) + (201 - ((201/(throttleHighFlex - throttleLowFlex))*throttleHighFlex));

if (*throttleP < throttleLowFlex)

begin

allBits[2] = 0; 

allBits[4] = 0; 

allBits[6] = 0; 

allBits[8] = 0; 

allBits[10] = 0; 

allBits[12] = 0; 

allBits[14] = 0;

allBits[16] = 0; 

end

else 

begin

allBits[2] = (integerThrottle & 0b10000000) >> 7; 

allBits[4] = (integerThrottle & 0b01000000) >> 6; 

allBits[6] = (integerThrottle & 0b00100000) >> 5; 

allBits[8] = (integerThrottle & 0b00010000) >> 4; 

allBits[10] = (integerThrottle & 0b00001000) >> 3; 

allBits[12] = (integerThrottle & 0b00000100) >> 2; 

allBits[14] = (integerThrottle & 0b00000010) >> 1;

allBits[16] = (integerThrottle & 0b00000001); 

end

//////////////////

////LFT-RT////////

//////////////////

char offRightLeft;

int integerRightLeft; 

int integerLeft = (((200/(leftHigh - leftLow)) * (*leftP)) >> 5) + 8;

int integerRight = ((200/(rightHigh - rightLow)) * (*rightP)) >> 5 ;

//Choosing which result to use for right-left motion

if ((integerLeft - 8) > integerRight) 

begin

integerRightLeft = integerLeft;

offRightLeft = (*leftP < leftLow); 

// PORTD = PORTD ^ 0x02;

end

else

begin 

integerRightLeft = integerRight;

offRightLeft = (*rightP < rightLow); 

// PORTD = PORTD ^ 0x04;

end

if (offRightLeft)

begin

allBits[18] = 1; 

allBits[20] = 0; 

allBits[22] = 0; 

allBits[24] = 0; 

// PORTD = PORTD ^ 0x01;

end

else 

begin

allBits[18] = (integerRightLeft & 0b1000) >> 3; 

allBits[20] = (integerRightLeft & 0b0100) >> 2;

allBits[22] = (integerRightLeft & 0b0010) >> 1; 

allBits[24] = (integerRightLeft & 0b0001); 

end

////////////////////

////FWD-BACK////////

////////////////////

int result3, result4;

char offForwardBack;

int integerForwardBack; 

int integerForward = (((200/(forwardHigh - forwardLow)) * (*forwardP)) >> 5) + 8;

int integerBack = ((200/(backwardHigh - backwardLow)) * (*backP)) >> 5 ;

//Choosing which result to use for right-left motion

if ((integerForward - 8) > integerBack) 

begin

integerForwardBack = integerForward;

offForwardBack = (*forwardP < forwardLow); 

// PORTD = PORTD ^ 0x02;

end

else

begin 

integerForwardBack = integerBack;

offForwardBack = (*backP < backwardLow); 

// PORTD = PORTD ^ 0x04;

end

if (offForwardBack)

begin

allBits[26] = 1; 

allBits[28] = 0; 

allBits[30] = 0; 

allBits[32] = 0; 

// PORTD = PORTD ^ 0x01;

end

else 

begin

allBits[26] = (integerForwardBack & 0b1000) >> 3; 

allBits[28] = (integerForwardBack & 0b0100) >> 2;

allBits[30] = (integerForwardBack & 0b0010) >> 1; 

allBits[32] = (integerForwardBack & 0b0001); 

end

end

//***********************************************************

//***********************************************************

//Task for getting ADC data from flex sensors

void flexData(void)

begin

flexCount = flexSampTime;

//Grab analog data off flex sensors PORTA0-4

if (sensorCounter == 0x00)

begin

sensor1 = ADCH;

ADMUX =  (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00001;

ADCSRA |= (1<<ADSC);

sensorCounter = 0x01;

end

else if (sensorCounter == 0x01)

begin

sensor2 = ADCH;

ADMUX =  (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00010; 

ADCSRA |= (1<<ADSC);

sensorCounter = 0x02; 

end

else if (sensorCounter == 0x02)

begin

sensor3 = ADCH;

ADMUX =  (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00011;

ADCSRA |= (1<<ADSC);

sensorCounter = 0x03;

end

else if (sensorCounter == 0x03)

begin

sensor4 = ADCH;

ADMUX =  (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00100;

ADCSRA |= (1<<ADSC);

sensorCounter = 0x04;

end

else

begin

sensor5 = ADCH;

ADMUX =  (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00000;

ADCSRA |= (1<<ADSC);

sensorCounter = 0x00;

end

//When we have read from all the flexsensors update signal being sent

if (sensorCounter == 0x00)

begin

flexSignal(); 

end

end  

//***********************************************************

   

//***********************************************************        

void accelSignal(void)

begin

allBits[38] = (PINC & 0x02) >> 1; //trim left

allBits[40] = (PINC & 0x04) >> 2; //trim right

PORTD = allBits[40];

////////////////////

////THROTTLE////////

////////////////////

int integerThrottle;

integerThrottle = ((201/(throttleHighAccel - throttleLowAccel)) * (*throttleAccelP)) + (201 - ((201/(throttleHighAccel - throttleLowAccel))*throttleHighAccel)) +24;

//PORTD = ~(integerThrottle);

if (*throttleAccelP < throttleLowAccel)

begin

//PORTD = (PORTD ^  0x01);

allBits[2] = 0; 

allBits[4] = 0; 

allBits[6] = 0; 

allBits[8] = 0; 

allBits[10] = 0; 

allBits[12] = 0; 

allBits[14] = 0;

allBits[16] = 0; 

end

else 

begin

//PORTD = (PORTD ^  0x02);

allBits[2] = (integerThrottle & 0b10000000) >> 7; 

allBits[4] = (integerThrottle & 0b01000000) >> 6; 

allBits[6] = (integerThrottle & 0b00100000) >> 5; 

allBits[8] = (integerThrottle & 0b00010000) >> 4; 

allBits[10] = (integerThrottle & 0b00001000) >> 3; 

allBits[12] = (integerThrottle & 0b00000100) >> 2; 

allBits[14] = (integerThrottle & 0b00000010) >> 1;

allBits[16] = (integerThrottle & 0b00000001); 

end

////////////////////

////LEFT-RIGHT////////

////////////////////

 int integerLeftAccel;

 int integerRightAccel;

integerLeftAccel = (((200/(rightLowAccel - rightHighAccel)) * (*rightleftP))  >> 5) - 2 ;

//PORTD = ~(integerLeftAccel);

allBits[18] = (integerLeftAccel & 0b1000) >> 3; 

allBits[20] = (integerLeftAccel & 0b0100) >> 2;

allBits[22] = (integerLeftAccel & 0b0010) >> 1; 

allBits[24] = (integerLeftAccel & 0b0001); 

////////////////////

////FORWARD-BACK////////

////////////////////

 int integerForwardAccel;

 int integerBackAccel;

integerBackAccel =~( (((200/(backwardLowAccel - backwardHighAccel)) * (*forwardbackP))  >> 5) - 3 );

//PORTD = ~(integerBackAccel);

allBits[26] = (integerBackAccel & 0b1000) >> 3; 

allBits[28] = (integerBackAccel & 0b0100) >> 2;

allBits[30] = (integerBackAccel & 0b0010) >> 1; 

allBits[32] = (integerBackAccel & 0b0001); 

end

//***********************************************************

//***********************************************************

//Task for getting ADC data from accelerometer and flex sensor

void accelData(void)

begin

accelCount = accelSampTime;

if (sensorCounter == 0x00)

begin

sensor6 = ADCH;

ADMUX = (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00110; 

ADCSRA |= (1<<ADSC);

sensorCounter = 0x01;

end

else if (sensorCounter == 0x01)

begin

sensor7 = ADCH;

ADMUX = (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00111; 

ADCSRA |= (1<<ADSC);

sensorCounter = 0x02;

end

else if (sensorCounter == 0x02)

begin

sensor8 = ADCH;

ADMUX = (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00101;

ADCSRA |= (1<<ADSC);

sensorCounter = 0x00;

end

//When we have read from accelerometer and throttle flexsensor update signal being sent

if (sensorCounter == 0x00)

begin 

accelSignal();

end  

end

//***********************************************************

//***********************************************************        

//Main loop

int main(void)

begin

/*********************************************************************/

/********************************* MCU INITIALIZATIONS BEGIN*******************************************/

/**********************************************************************

//Port B.3 gives the output to the IR emitter 

   DDRB = 0x08;

PORTB = 0x08;

//Port C.0 for switch mode select

DDRC = 0x00;

PORTC = 0xff;

//LED D.0-D.7

DDRD = 0xff;

  PORTD = 0xff;

//Turn on Timer 0 to control the signal to the IR emitter

TIMSK0 = 2; //turn on timer 0 cmp match ISR 

TCCR0A = 0b01000010; // turn on toggle and clear-on-match

  TCCR0B = 0b00000001; // no clock prescalar and turn on counter

  OCR0A = 216;   //set the compare reg to 216 time ticks for 37.0kHz square wave pulse

//********ADC*************

   //init the A to D converter 

   //channel zero/ left adj /EXTERNAL Aref

   //!!!CONNECT Aref jumper!!!!

   ADMUX =  (1 << REFS1) | (1 << REFS0) | (1<<ADLAR) + 0b00000;  

    

   //enable ADC and set prescaler to 1/128*16MHz=125,000

  //and clear interupt enable

  //and start a conversion

   ADCSRA = (1<<ADEN) | ((1<<ADSC) + 7) ; 

/*********************************** MCU INITIALIZATIONS END*******************************************/

/****************************************************************/

/****************************************************************/

/**************************** VARIABLE INITIALIZATIONS BEGIN*******************************************/

/*******************************************************************/

//Initialize values 

bitSent = 1;

signalPointer = 0;

count = 0;

flexCount = flexSampTime; 

accelCount = accelSampTime;

sensorCounter = 0x00;

//Sensor definitions

throttleP = &sensor1;

leftP = &sensor2;

rightP = &sensor3;

forwardP = &sensor4;

backP = &sensor5;

throttleAccelP = &sensor6; 

rightleftP = &sensor7;

forwardbackP = &sensor8;

//range for throttle, left-right and forward backward for flex sensors

throttleLowFlex = 15; 

throttleHighFlex = 90; 

leftHigh = 180; 

leftLow = 70;

rightHigh = 255; 

rightLow = 15;

forwardHigh = 200; 

forwardLow = 90;

backwardHigh = 200; 

backwardLow = 90;

//range for throttle, left-right and forward backward for accelerometer

throttleLowAccel = 15;

throttleHighAccel = 90;

leftHighAccel = 200; 

leftLowAccel = 165;

rightLowAccel = 134; 

rightHighAccel = 95;

forwardHighAccel = 200; 

forwardLowAccel = 150;

backwardLowAccel = 130; 

backwardHighAccel = 94;

/****************************** VARIABLE INITIALIZATIONS END*******************************************/

/*************************************************************/

  //crank up the ISRs

  sei();

  //main task scheduler loop 

  while(1)

  begin   

//If the flexCount is zero read the values from the flexsensors

   if (flexCount == 0)

begin

if ((PINC & 0x01) == 0x00)

begin

flexData();

// allBits[39] = (PORTC & 0x02) >> 1; //trim left

// allBits[40] = (PORTC & 0x04) >> 2; //trim right

end

end

if (accelCount == 0)

begin

if ((PINC & 0x01) == 0x01)

begin

accelData();

end

end

  end

end  

//***********************************************************

Cost Details

Parts	Unit Price	Quantity	Total Price
Mega644 MCU	$8	1	$8
Custom PC Board	$4	1	$4
Helicopter	$17.50	1	$17.50
Header Socket	$0.05	115	$5.75
Flex Sensors*	$10	10	Free
Accelerometer	$20	1	$20
Pair of Gloves	$11.68	1	$11.68
Cloth	$3.77	1	$3.77
			Total: $70.70

4. References 

Data Sheets

Accelerometer (A7260):       http://www.freescale.com/files/sensors/doc/data_sheet/MMA7260QT.pdf

NPN transistor (2N3904):      http://www.fairchildsemi.com/ds/2N/2N3904.pdf

NPN transistor (TIP31A):      http://www.datasheetcatalog.org/datasheet/PowerInnovations/mXuqyry.pdf

Mega644 datasheet:             http://courses.cit.cornell.edu/ee476/AtmelStuff/mega644full.pdf 

Prototype Board for Atmel Mega644:    http://www.nbb.cornell.edu/neurobio/land/PROJECTS/Protoboard476/index.html

Control Ipod

Austin Lu and Albert Ren
Source: http://instruct1.cit.cornell.edu/
Introduction

Have you ever imagined, "What does that cable I plug into my iPod every day actually do, and how do I take advantage of it for myself?" We did too, and that's what we aimed to do with our 476 project. The iPod is, of course, the popular digital music player developed by Apple. The heart of the project is to use the Mega32 microcontroller to transmit and receive commands from the iPod using the built-in Dock Connector, as opposed to cracking the iPod open and wiring it up. The Dock Connector is the little port at the bottom of the iPod that you would normally connect a white USB or FireWire sync cable into. Commands that are currently implemented include play, stop, fast forward and other basic commands that allow the user to enjoy music from the iPod, as well as the ability to display the current track's Title, Artist, and Album on an external LCD display.
This document is written for, primarily, the 476 graders! Just as importantly though, this document is intended for future 476 classes looking for interesting project ideas to tackle, because as far as we could tell, no one had attempted a project dealing with the iPod before. This document is also intended for those searching on the Internet for help on hacking their iPod.




High-Level Design

At first we were worried about the feasibility of the project. We had decided early on that we wanted a project with obvious user feedback. After speaking to Bruce Land about the concept of an iPod dock, we decided that it would be our final project -- what is more obvious than music starting or stopping?

A huge amount of our time involved researching the iPod communication protocol. The iPod's communication protocol through the Dock Connector is not actually publicly available from Apple. However, thanks to the reverse engineering efforts by many hobbyists, there is a lot of information about the protocol (termed the "Apple Accessory Protocol") available on the Internet.

It is possible that the protocol isn't public due to the "Made for iPod" licensing program, which lets accessory makers in the "iPod ecosystem" connect to the iPod and display certain logos on their packaging after licensing. This could present certain legal issues, as it may be considered some form of protected intellectual property. However, as the protocol has been obtained through reverse engineering methods and not through a breach of contract, we do not anticipate this to be a big factor. Additionally, this is only an educational exercise and not in any way intended to be a commercially viable product.

The structure of our project is to have the Atmel Mega32 controller talk to the iPod in both sending and receive mode through the iPod's Dock Connector. We would send commands over the serial pins on the Mega32 and receive on the appropriate iPod pins as well. We can verify that the commands worked by simply observing the iPod and checking to see that the iPod responded appropriately, e.g. with a volume change, skipped track, etc.

The only relevant standards that we use are 8-N-1 serial communication for talking to the iPod. The actual Apple Accessory Protocol is an Apple creation, so not quite a "standard" in a usual sense. Otherwise, we do not make any use of any other standards, such as IEEE or ISO standards. The iPod sends out responses for strings in ASCII format.





Design Aspects

Research

A large amount of work went into determining whether this idea was actually feasible. No one appears to have made any similar efforts in 476 final projects before, and though there is a similar idea in ECE 313, that course was designed around a standard CD player and controlled the player by shorting the buttons.

We initially found the iPod Dock Connector pin-out, which is available from pinouts.ru After that, we eventually found the Apple Accessory Protocol, which is detailed best on the iPod Linux wiki.



Important Pins on the iPod

Pin Use

1 Ground

11 Serial Ground

12 iPod RX; microcontroller transmits on this pin

13 iPod TX; microcontroller receives from this pin

21 iPod Accessory resistor: We use 500k

It's important to note which pins are the receive and transmit pins. On the pinouts.ru page, pin 13 is "Serial RxD," which means serial receive from the iPod's point of view. Mixing it up could confuse you unnecessarily for a few hours.

Once we had a pinout and what looked like usable, if unofficial, documentation of the protocol, we started looking at actual connectivity to the iPod. There were a few efforts done by other people to create their own docks, but these were not quite what we were looking for.

After a bit of searching, we managed to find a hobbyist's website that had the actual iPod connectors available for sale. Even better, they also had a custom PCB available for sale to break out all of the individual signals from each of the pins, which was very useful because the pins themselves are rather small.

A short while later, we ended up discovering a second source for our PCB, which cost about the same as the original source but came with the dock connector pre-soldered, which was very useful as it was probably going to be a better soldering job than we could hope to accomplish by ourselves. Additionally, unlike the other source, it would ship within the United States. Using First Class Mail, we received our component in about 2 days (shipped on Thursday from CO, received on Saturday).

Next, we had to determine what commands it is we wanted to implement. We decided that the basic control functions we're all familiar with - Play/Pause, Volume Up and Down, Skip Forward and Backward - must be implemented. We decided next that the "good to have" functions would be receiving data from the iPod and displaying it on the LCD screen, namely the current track's Title, Artist, and Album.

All of the control details can be reverse engineered using a breakout board with a pass-through connector (in other words, a female iPod connector) and a commercial accessory that implements the same feature. All you'd have to do is plug the accessory into the pass-through connector, the pass-through into the iPod, and listen in on the serial pin.

It turns out that the iPod has a set of "modes" that it categorizes the functions in. The basic set of functions all belong in Mode 2, while the "good to have" receiving functions are all Mode 4. We also use functions from Mode 0 in order to change the iPod mode. This is important, because it changes how you expect to send and receive the data packets with the iPod.

Communication with the iPod is conducted over a serial protocol with a standard 8N1 setting at 19200 baud. This means that each packet has 10 bits. The packet starts with a low start bit, 8 bits (1 byte) of information, no parity, and a high stop bit. Each packet has a very ordered structure, which is as follows:

Section Size of Section (in bytes) Value

Header 2 0xff 0x55

Length of packet 1 Indicates the number of bytes for mode, command, and parameter

Mode 1 Mode for this command

Command 2 Command we're sending or receiving

Parameter 0 to n Optional - Some commands use this to supply more info

Checksum 1 (0x100 - [actual values of length + mode + command + parameter]) & 0xff

This table is modified slightly for clarity from the iPodLinux page on the subject. This is the case for other tables that appear in this section as well.

Incidentally, each 8-bit chunk of data in a serial packet is transmitted in Little Endian mode. The serial protocol also idles at VCC-high when not sending.

The command codes for the functions we have selected are shown below:



Mode 0 Commands

Command Function

0x01 0x02 Switch to Mode 2

0x01 0x04 Switch to Mode 4



Mode 2 Commands

Command Function

0x00 0x00 Button Release

0x00 0x01 Play

0x00 0x02 Volume Up

0x00 0x04 Volume Down

0x00 0x08 Forward

0x00 0x10 Backward



Mode 4 Commands

Command Parameter (in bytes) Function

0x00 0x1e 0 Get Current Position in playlist

0x00 0x1f 4 Current Position Result

0x00 0x20 4 - current position Get title of current song

0x00 0x21 string - unknown length Title result

0x00 0x22 4 - current position Get artist of current song

0x00 0x23 string Artist result

0x00 0x24 4 - current position Get album of current song

0x00 0x25 string Artist result

We were led to believe that Mode 2 commands needed to be sent 66 times per second until button release. However, from our experience this is not necessary. The command only needs to be sent once for the iPod to act upon it. Once the command is sent though, it is necessary to sent a button release command to indicate, well, the button's release. Otherwise, the iPod is going to assume that the button is being held and will do what it normally does in such situations. For example, pressing Play without a Button Release will turn your iPod off (a tidbit that confused us for a little bit).

The Mode 4 commands that we implemented seem basic, but as far as we could tell, very few accessories currently on the market take advantage of the mode 4 commands. As far as we know, only iPod car-kits use the mode 4 commands; these car kits scroll similar information on the dashboard.

Mode 4 commands require that the iPod be switched into Mode 4 using a Mode 0 command, when the iPod will display "OK to disconnect" and cut off interaction with the physical on-iPod controls until the dock connector is removed. With exception of volume control, the standard Mode 2 commands are still accepted, even though it is currently in Mode 4. Annoyingly, switching into Mode 4 also pauses the currently playing track.

The current position numbering is transmitted in 4 bytes, with each byte in Big Endian notation. Do not confuse this with the Little Endian transmit inside the individual bytes. The returned strings for Title, Artist, and Album are null terminated.



Program Design

The state machine operates on a single assumption: the iPod talks back only when we want it to. If the iPod were to send a message when we were not expecting it, we would ignore the iPod. Additionally, we have helper functions that wait for input from the user which set flags to alert the state machine.

The state Command is the initial state of the microcontroller. In Command, we wait for the validcommand flag to be set from the button debouncer. This flag alerts the state machine to proceed to DecodeCommand. Otherwise, Command checks to see if the releaseflag has been set and whether the mode is set to 0x02. If it is, send the state machine to the button_release state.

The next state is DecodeCommand. DecodeCommand prepares the character array t_buffer to be transmitted through the UART. In this state, the two modes are differentiated -- if the button pressed was of mode 0x02, the length of the command is typically shorter. If the command was of mode 0x04, the counter mode4flag is set to 1, which will alert the state machine that the iPod will be responding. The state proceeds to Send.

In Send, mode4flag is checked. All the mode 0x04 commands require that the iPod switch to a mode 4 state. We discovered that the iPod readily accepts both mode 0 and mode 2 commands, but will not recognize mode 0x04 commands. Thus, there is a sequence of commands to execute before sending a mode 4 command. This sequence is embodied in the mode4sequence counter. We will send commands based on what mode4sequence is.

The first command in mode4sequence is a switch to mode 4 command. Then we wait several milliseconds before transmitting a "current song" command. We check to see if the command was successful (the iPod sends back the current track if the command was received, and sends back a failure if the command was incorrect). Finally, we send the mode 0x04 command that has the parameter currentsong to determine characteristics of the song. Again, we wait for the iPod to respond. The last command in mode4sequence is switching the iPod back to mode 2. We made this decision because in mode 2, we can use volume up and down. Additionally, the iPod screen is ugly when it enters mode 4. Whenever we wait for the iPod to respond, we will be in the "Receive" state. Upon a successful response, the state will return to the Send state. If the command was of mode 0x02, the state machine will transition to Command. To transmit a command, we call on the helper function puts_int().

We should only enter the Receive state if the command is of mode 4. Depending on what mode4sequence is, the state machine may re-enter the send state (switching iPod modes do not require responses), or the state machine waits for the Receive interrupt to set the r_ready flag. Upon receiving a buffer, the buffer is dissected and the iPod response is determined. Upon a successfully sending a command, the state machine changes mode4sequence, resets the failure counter and returns to the Send state to send the next command in the sequence. Upon a failure, the state machine increments the failure counter and returns to the send state without changing the sequence. This means that the command that failed will be sent again. Once the failure counter hits the failure threshold, or if we have exceeded our timeout limit, the state machine returns to the command state.

The button_release state should occur when the user has finished sending a mode 0x02 command. The t_buffer is set to the button release command, which is of mode 0x02, and is transmitted to the iPod. The state then proceeds to Send.





Helper functions we used:

initialize() sets up the LCD display and determines the baud rate that we use. Since the iPod transmits at 19200 bits per second, we set UBRRL to 51. Additionally, an interrupt is set to trigger every millisecond, which provides the clock for the state machine.

buttons() is called every millisecond, and checks to see if a button has been pressed or not. If the previous button (lastpress) is 0, and there is a button press (press != 0), then debouncecounter begins to increment. When debouncecounter reaches 40 (40 ms have passed), the validcommand flag is set high, and the debouncecounter is reset to -460. Thus, the debouncecounter also functions as a hold timer. After 500ms have passed, the validcommand flag is set high again, and the button press is sent again.

As it turns out, when an iPod receives a mode 0x02 command, it continues performing that operation until a button release command is sent. However, the iPod does not send any acknowledgment regarding mode 2 commands, unlike mode 4 commands. Thus, we decided to err on caution and simplicity, and send the command multiple times if the button is held down noticeably. It would be an interesting question to ask if the iPod functions similarly for internal button presses (e.g. on the click wheel). Clearly, volume up/down is handled differently on the iPod due to the wheel motion.

Finally, we have a helper function called lcd_scroll(). This helper function is called every 300 ms, and is used to scroll the title, artist, and album fields on our little 16x2 LCD screen. These fields need to be scrolled if the fields are longer than 16 bits, which would not display nicely on our LCD screen. It works by copying over the relevant part of r_buffer, namely the part that includes the string fields, to another array, lcdbuffer. That array is then copied over into a "scratch" LCD buffer, displaybuffer, which includes exactly what 16 characters are supposed to be displayed on the LCD. Thus, every 300 ms, we move the LCD display buffer over by 1 character and update the LCD screen. Additionally, we also loop around to the beginning when we hit the end of our string field, so the scroll itself loops too.

puts_int() starts transmitting t_buffer through the UART. It resets all through t_index and t_ready, then calls put_char() to transmit the first character of t_buffer. Afterwards, the interrupt transmits each character at a time.


Tricky Stuff
To send and receive, we initially used Bruce Land's serial transmission code and modified the stop conditions to checksums instead of null terminators. Additionally, the transmit interrupt would not stop at the appropriate place -- we rewrote the code so it was logically the same, but now would stop at the appropriate spot.

Within the receive interrupt, we realized that a parameter within an iPod response could erroneously match the checksum. For example, when sending back the album title "Good News for People Who Like Bad News," one of our implementations stopped at the space character (' ') because its ASCII code, 0x20, matches the incomplete computed checksum. The old implementation detected this match and considered it to be the end of the transmission, while the actual checksum was quite a few bytes later. After finding several checksum collisions, we changed the end condition of the receive interrupt to look at the length of r_buffer to prevent the premature setting of r_ready flags.

How the Send and Receive states intertwined was easily the trickiest part of the code. The sources we used did not explicitly tell us every response an iPod would give us -- for example, mode switching was not well documented and we were not aware of its necessity. Originally, we were planning to ignore responses that we did not care about. After spending some time in lab, we decided that this was not the correct approach, and spent time determining all the responses.

When the iPod leaves mode 4 and enters mode 2, the iPod can ignore commands! When the user presses the physical play button on the iPod, there are no issues. In contrast, the iPod ignores the serial Play command up to three times! We believe there is a command in the sequence missing that we are not aware of and/or is not documented. We cannot see what a "proper" commercial implementation does as we do not own an accessory that actually makes use of Mode 4 commands and, as a result, cannot intercept and spy on what other implementations do. Thus, to start playing a song after entering Mode 4, the user needs to hit the play button twice on our implementation.

Finally, John Sicilia showed us a clever way of checking the receive buffer using the LEDs. Light up the LEDs according to the character in the buffer, delay for several seconds, turn the LEDs off for some time, and then move on to the next character in the buffer. Prior to this method, we had been looking at the waveform on the oscilloscope, which works fine but is very time consuming. When there were issues like the checksum error, we would compare the oscilloscope waveform to the output from the LEDs. A sample of this technique is shown here; be sure to include delay.h:

for(k=0; k<11; k++)

{

PORTB = t_buffer[k];

delay_ms(5000);

PORTB = 0xFF;

delay_ms(500);

}

while(1);

Hardware Design

The microcontroller outputs a high and low voltage of 5V and 0V respectively. However, the iPod operates at a maximum of 3.3V for high. Thus, for the serial transmission line, we created a voltage divider, so that when the line is pulled to 5V, the iPod sees 3.3V. The high level schematic is below.

Additionally, the iPod only outputs a maximum of 3V when transmitting serially. We used an LF353 op-amp to increase the output voltage to 5V. The schematic for a non-inverting amplifier is shown below, where R2 is 1 kilo-Ohm and R1 is 2 kilo-Ohms. The Gain is 1+(R2/R1), where our values give a gain of 1.5.

Issues that we encountered

Everyone warns you about it, but somehow we got bit by it anyway. We're referring to, of course, having extra components for important things that you need. In our case, the critical piece was the PCB breakout board for the dock connector. It turns out that a pin inside the connector was no longer well connected with the pin connected to the board. This meant that our wire connection was good all the way from breadboard, to PCB, to the pin leading inside the dock connector, but the pin inside the dock connector was flaky. We had anticipated this and ordered another connector as a spare, but this took additional time and put time pressure on our schedule that we would've liked to do without.
Results

The serial commands sent to the iPod and the responses from the iPod were sent and received accurately, which we could verify by looking at the waveforms on an oscilloscope and also by the eventual display on the LCD. The LCD displays and scrolls correctly, and the iPod responds to each command appropriately and quickly, depending on how fast a person can push the button. Neither safety nor interference were much of a concern, as our design did not really pose much of an issue in these categories. Usability could possibly be improved, as the pushbuttons are currently on the STK and not in a particularly nice enclosure, for example. If this were a prototype for a commercial design, it would be easy to fix though.
Conclusions

The results of this project met our expectations. We wanted to both control and receive from an iPod with a microcontroller, and this was accomplished. Our design used the serial communication protocol 8-N-1, and for this we used Bruce Land's serial communication code as a jump-off point for our serial communication code. As mentioned before, the specifics of the iPod communication protocol came from researching the reverse engineering efforts of other hobbyists, and we communicated with a few of them if there we had specific questions about the protocol. The documentation of the protocol was fairly good but, as always with documentation and especially unofficial documentation, could have been improved and more clear. Hopefully our writeup here will clarify certain things that are confusing. We do not expect to have any patent opportunities with this project.

Future Design Aspirations

If we had more time available, we would have liked to implement an IR receiver with the microcontroller. Thus, a user could use an IR remote to send commands to the microcontroller, which would then be relayed to the iPod. We also could have implemented more mode 4 commands, with more pushbuttons being our main constraint. Additionally, future efforts could involve changing the 16x2 LCD for a more complicated and complex one that could allow displaying more information at once or even color images, such as album art or pictures. An audiophile may want to add something that uses the Line-Out component of the connector, as those pins are readily accessible as well.


Ethics

We believe that our project was consistent with the IEEE Code of Ethics. For example, we believe that our project is safe to individuals and the public at large, and we would promptly notify everyone if this ever changed. We believe that we have been honest about our project's performance and potential future efforts. We also hope that our project has made the iPod (and similar MP3 players for which control data can be obtained) an attractive option for future hobbyists and 476 students.

In the interest of full disclosure, one of us was formerly employed by Apple (then known as Apple Computer). It should be noted that none of the material referenced in this project or on this page resulted either directly or indirectly from any material obtained during employment or with any use of Apple-owned resources, such as hardware or documentation. We have made careful efforts to ensure that nothing done here would compromise any previous or (hopeful) future employment agreements and that we have respected intellectual property throughout.

No animals were harmed in the making of this project. Cornell students were merely deprived of sleep.