Learn how to play music using an accelerometer and gyroscope sensor with the Wekinator software platform. 

In this article, you'll learn how to generate music through an accelerometer and an MPU6050 gyroscope (also referred to as a "gyro sensor"), using the Wekinator open-source software platform to enable machine learning techniques. 

For a helpful introduction to the Wekinator platform, check out our Machine Learning for Makers: How to Get Started With Wekinator article. 

The first step of this project is to interface the MPU6050 sensor with an Arduino, which will send the output data to processing. 

In processing, we'll calculate the YPR (yaw, pitch, roll) values and draw a 3D model that mimics the movement of the sensor then sends the values to the Wekinator. Finally, the output data is sent to a drum machine made in processing that produces music. 

 

play_music_through_wekinator1.png

Directions to connect the MPU6050 gyroscope to an Arduino.

The Input

On the input side, the sensor will need to be connected to the Arduino. The directions for doing so are shown in the figure below. 

Installing the Arduino Libraries 

To properly interface the Arduino with the sensor, first download the Arduino library developed for that purpose. You'll also require the I2C library, which you can access through this file folder

Next, unzip the contents of the file, open the folder titled, "Arduino" and move the I2C and MPU6050 sensor folder to the Arduino library folder. 

play_music_through_wekinator14.png

Example indicating the selected locations of the I2C dev and MPU6050 folders.

Uploading the Code

Start by launching the Arduino IDE software. Navigate to examples under 'File', then MPU6050 and open the MPU6050_DMP6 file.

play_music_through_wekinator2.png

Example demonstrating how to navigate to MPU6050_DMP6 file in Arduino IDE.

Upload the code found in the Arduino IDE and it should display on the serial monitor. 

If it's showing the output data, then you can assume the sensor has successfully interfaced with the Arduino. 

play_music_through_wekinator4.png

Output data displayed in a window confirming the MPU605 sensor has successfully connected interfaced with the Arduino. 

To send the data to processing, some changes need to be made to the code. 

First, select the line 117 of the code and uncomment it, move up to line 100 and comment that segment of code. If you have trouble with this step, reference the figure below. 

play_music_through_wekinator3.png

An example underlining the lines of code that need to be changed.

If you upload the code again, it will display unreadable characters on the serial monitor. 

play_music_through_wekinator6.png

An example of unreadable character code on the serial monitor window.

Processing Code

In order to receive the data from the Arduino, you'll be using the 'toxiclibs' library

play_music_through_wekinator5.png

Reference image indicating the circled location of the 'toxiclibs' library file. 

Next, copy the entire folder into the zip file and paste it in the processing library folder. You'll navigate to 'yourProcessingFolder', modes, java, then the libraries category. 

Now paste the code provided below (a modified version of the example in the sensor's library) into processing and upload it. 

    import processing.serial.*;
import processing.opengl.*;
import toxi.geom.*;
import toxi.processing.*;
import oscP5.*;
import netP5.*;

OscP5 oscP5;
NetAddress dest;

ToxiclibsSupport gfx;

Serial port;                         // The serial port
char[] teapotPacket = new char[14];  // InvenSense Teapot packet
int serialCount = 0;                 // current packet byte position
int synced = 0;
int interval = 0;

float[] q = new float[4];
Quaternion quat = new Quaternion(1, 0, 0, 0);

float[] gravity = new float[3];
float[] euler = new float[3];
float[] ypr = new float[3];

void setup() {
    // 300px square viewport using OpenGL rendering
    size(300, 300, OPENGL);
    gfx = new ToxiclibsSupport(this);

    // setup lights and antialiasing
    lights();
    smooth();
  
    // display serial port list for debugging/clarity
    println(Serial.list());

    // get the first available port (use EITHER this OR the specific port code below)
    String portName = Serial.list()[0];
    
    // get a specific serial port (use EITHER this OR the first-available code above)
    //String portName = "COM4";
    
    // open the serial port
    port = new Serial(this, portName, 115200);
    
    // send single character to trigger DMP init/start
    // (expected by MPU6050_DMP6 example Arduino sketch)
    port.write('r');
    
    /* start oscP5, sending messages at port 9000 */
    oscP5 = new OscP5(this,9000);
    dest = new NetAddress("127.0.0.1",6448);
}

void draw() {
    if (millis() - interval > 1000) {
        // resend single character to trigger DMP init/start
        // in case the MPU is halted/reset while applet is running
        port.write('r');
        interval = millis();
    }
    
    // black background
    background(0);
    
    // translate everything to the middle of the viewport
    pushMatrix();
    translate(width / 2, height / 2);

    // 3-step rotation from yaw/pitch/roll angles (gimbal lock!)
    // ...and other weirdness I haven't figured out yet
    //rotateY(-ypr[0]);
    //rotateZ(-ypr[1]);
    //rotateX(-ypr[2]);

    // toxiclibs direct angle/axis rotation from quaternion (NO gimbal lock!)
    // (axis order [1, 3, 2] and inversion [-1, +1, +1] is a consequence of
    // different coordinate system orientation assumptions between Processing
    // and InvenSense DMP)
    float[] axis = quat.toAxisAngle();
    rotate(axis[0], -axis[1], axis[3], axis[2]);

    // draw main body in red
    fill(255, 0, 0, 200);
    box(10, 10, 200);
    
    // draw front-facing tip in blue
    fill(0, 0, 255, 200);
    pushMatrix();
    translate(0, 0, -120);
    rotateX(PI/2);
    drawCylinder(0, 20, 20, 8);
    popMatrix();
    
    // draw wings and tail fin in green
    fill(0, 255, 0, 200);
    beginShape(TRIANGLES);
    vertex(-100,  2, 30); vertex(0,  2, -80); vertex(100,  2, 30);  // wing top layer
    vertex(-100, -2, 30); vertex(0, -2, -80); vertex(100, -2, 30);  // wing bottom layer
    vertex(-2, 0, 98); vertex(-2, -30, 98); vertex(-2, 0, 70);  // tail left layer
    vertex( 2, 0, 98); vertex( 2, -30, 98); vertex( 2, 0, 70);  // tail right layer
    endShape();
    beginShape(QUADS);
    vertex(-100, 2, 30); vertex(-100, -2, 30); vertex(  0, -2, -80); vertex(  0, 2, -80);
    vertex( 100, 2, 30); vertex( 100, -2, 30); vertex(  0, -2, -80); vertex(  0, 2, -80);
    vertex(-100, 2, 30); vertex(-100, -2, 30); vertex(100, -2,  30); vertex(100, 2,  30);
    vertex(-2,   0, 98); vertex(2,   0, 98); vertex(2, -30, 98); vertex(-2, -30, 98);
    vertex(-2,   0, 98); vertex(2,   0, 98); vertex(2,   0, 70); vertex(-2,   0, 70);
    vertex(-2, -30, 98); vertex(2, -30, 98); vertex(2,   0, 70); vertex(-2,   0, 70);
    endShape();
    
    popMatrix();
    
    //Send the OSC message
    sendOsc();
}

void serialEvent(Serial port) {
    interval = millis();
    while (port.available() > 0) {
        int ch = port.read();

        if (synced == 0 && ch != '$') return;   // initial synchronization - also used to resync/realign if needed
        synced = 1;
        print ((char)ch);

        if ((serialCount == 1 && ch != 2)
            || (serialCount == 12 && ch != '\r')
            || (serialCount == 13 && ch != '\n'))  {
            serialCount = 0;
            synced = 0;
            return;
        }

        if (serialCount > 0 || ch == '$') {
            teapotPacket[serialCount++] = (char)ch;
            if (serialCount == 14) {
                serialCount = 0; // restart packet byte position
                
                // get quaternion from data packet
                q[0] = ((teapotPacket[2] << 8) | teapotPacket[3]) / 16384.0f;
                q[1] = ((teapotPacket[4] << 8) | teapotPacket[5]) / 16384.0f;
                q[2] = ((teapotPacket[6] << 8) | teapotPacket[7]) / 16384.0f;
                q[3] = ((teapotPacket[8] << 8) | teapotPacket[9]) / 16384.0f;
                for (int i = 0; i < 4; i++) if (q[i] >= 2) q[i] = -4 + q[i];
                
                // set our toxilibs quaternion to new data
                quat.set(q[0], q[1], q[2], q[3]);

                
                // below calculations unnecessary for orientation only using toxilibs
                
                // calculate gravity vector
                gravity[0] = 2 * (q[1]*q[3] - q[0]*q[2]);
                gravity[1] = 2 * (q[0]*q[1] + q[2]*q[3]);
                gravity[2] = q[0]*q[0] - q[1]*q[1] - q[2]*q[2] + q[3]*q[3];
    
                // calculate Euler angles
                euler[0] = atan2(2*q[1]*q[2] - 2*q[0]*q[3], 2*q[0]*q[0] + 2*q[1]*q[1] - 1);
                euler[1] = -asin(2*q[1]*q[3] + 2*q[0]*q[2]);
                euler[2] = atan2(2*q[2]*q[3] - 2*q[0]*q[1], 2*q[0]*q[0] + 2*q[3]*q[3] - 1);
    
                // calculate yaw/pitch/roll angles
                ypr[0] = atan2(2*q[1]*q[2] - 2*q[0]*q[3], 2*q[0]*q[0] + 2*q[1]*q[1] - 1);
                ypr[1] = atan(gravity[0] / sqrt(gravity[1]*gravity[1] + gravity[2]*gravity[2]));
                ypr[2] = atan(gravity[1] / sqrt(gravity[0]*gravity[0] + gravity[2]*gravity[2]));
    
                // output various components for debugging
                //println("q:\t" + round(q[0]*100.0f)/100.0f + "\t" + round(q[1]*100.0f)/100.0f + "\t" + round(q[2]*100.0f)/100.0f + "\t" + round(q[3]*100.0f)/100.0f);
                //println("euler:\t" + euler[0]*180.0f/PI + "\t" + euler[1]*180.0f/PI + "\t" + euler[2]*180.0f/PI);
                println("ypr:\t" + ypr[0]*180.0f/PI + "\t" + ypr[1]*180.0f/PI + "\t" + ypr[2]*180.0f/PI);
                
            }
        }
    }
}

void drawCylinder(float topRadius, float bottomRadius, float tall, int sides) {
    float angle = 0;
    float angleIncrement = TWO_PI / sides;
    beginShape(QUAD_STRIP);
    for (int i = 0; i < sides + 1; ++i) {
        vertex(topRadius*cos(angle), 0, topRadius*sin(angle));
        vertex(bottomRadius*cos(angle), tall, bottomRadius*sin(angle));
        angle += angleIncrement;
    }
    endShape();
    
    // If it is not a cone, draw the circular top cap
    if (topRadius != 0) {
        angle = 0;
        beginShape(TRIANGLE_FAN);
        
        // Center point
        vertex(0, 0, 0);
        for (int i = 0; i < sides + 1; i++) {
            vertex(topRadius * cos(angle), 0, topRadius * sin(angle));
            angle += angleIncrement;
        }
        endShape();
    }
  
    // If it is not a cone, draw the circular bottom cap
    if (bottomRadius != 0) {
        angle = 0;
        beginShape(TRIANGLE_FAN);
    
        // Center point
        vertex(0, tall, 0);
        for (int i = 0; i < sides + 1; i++) {
            vertex(bottomRadius * cos(angle), tall, bottomRadius * sin(angle));
            angle += angleIncrement;
        }
        endShape();
    }
}

void sendOsc() {
  OscMessage msg = new OscMessage("/wek/inputs");
  msg.add((float)ypr[2]); // x-axis 
  msg.add((float)ypr[1]);  // y -axis
  oscP5.send(msg, dest);
}

After uploading the code, a window should appear as shown below. 

play_music_through_wekinator8.png

Window that should appear after uploading code into processing.

The Output

In terms of the output for this project, we'll be using a processing sketch that receives the output from the Wekinator and produces music according to its instructions.

The specific sketch will receive three, continuous, outputs from the Wekinator and is found on the examples page on the Wekinator website.

Download the 'Simple continuously-controlled drum machine' file from the webpage and run the sketch in the processing window. 

play_music_through_wekinator7.png

Example of running the processing sketch in processing. 

Using Wekinator

In the Wekinator software, you'll need to set the settings to reflect those shown in the example below. 

Assign the inputs values to 2 and outputs to 3. Also, designate the output type to the default setting, 'All continuous.'

This will prompt the Wekinator to send the 3 different outputs required to processing and cause different music to play, depending on the values. 

play_music_through_wekinator11.png

If you click on the next button, you'll come to the 'New Project" window shown below. 

play_music_through_wekinator9.png

Set the output1 value to '1' and try tilting the sensor in any direction. Then, set the other two output boxes to 'o' and start the recording in order to create some audio samples. 

play_music_through_wekinator10.png

Tilt the sensor in the other direction and set output2 to '1' and the other two outputs areas to '0', before starting to record a sample again. 

play_music_through_wekinator12.png

Finally, tilt the sensor in another direction, of your choice and set output3 to '1' and the other two to '0'. 

Record a final sample to examine the result of those particular value settings. 

play_music_through_wekinator13.png

Feel free to experiment further by recording additional samples while tilting the sensor in different directions. 

After recording the sample, program the Wekinator and run it. It should produce music according to the sensor's movement. 

Muhammad Aqib
I am an embedded programmer with the expertise in Arduino, Raspberry pi, ESP8266, IOT and OpenCV.

Maker Pro Logo
Continue to site
Quote of the day

-