In this article, learn how to use the face tracker made in openFrameworks to control an FM synthesizer in ChucK. The data will then be trained by Wekinator machine learning software.
Inputs will be given to Wekinator from openFrameworks. The face tracker made in openFrameworks will send the x, y position and width of your face to Wekinator. Wekinator will be trained according to these inputs and will send the output to ChucK where the FM synthesizer will be controlled. The FM synthesizer will receive five continuous outputs from Wekinator and will make different sounds according to these.
We will record samples for the three gestures: the first will be with a face right in front of the camera, the second will be a face at a further distance from the camera, and the third will be with no face or other objects in front of the camera.
Setting Up openFrameworks
If you are new to openFrameworks, you need to first install it on your computer.
The executable file for Mac is available from Wedinator's examples page. If you use Windows, get the source code from the examples page and then you need to compile it.
Download the file according to your OS from there. Unzip the file and run it. It will use the laptop's webcam (or any webcam attached to your computer) to track your face. The output window will look like this:
Setting Up Outputs for ChucK
On the output side, we will use ChucK. ChucK will receive 5 continuous outputs from Wekinator and will make the sounds accordingly.
The project code for ChucK is available for download and is titled ‘FM Synthesis.’ Download for the operating system that you are using.
Once downloaded, open the terminal and go to the directory where you have placed the project code and write the following line:
ChucK will begin listening for outputs from Wekinator and, upon receiving the output, will change the pitch of sound.
Setting up the Inputs in Wekinator
Open Wekinator and make the settings match the ones shown in the below figure.
Set the inputs to 3 and the outputs to 5. Select the output type to “all continuous” and leave other settings as they are.
Wekinator will receive 3 inputs from openFrameworks and, after training, will send 5 different outputs to ChucK.
ChucK will produce different sounds according to Wekinator's outputs.
Once those settings are complete, click "next" and this window will appear:
Position your face in front of the camera and click "randomize." Set the output in the Wekinator window to 1 and start the recording for a second. It will record some samples.
Next, sit further from the webcam, select output 2, and click on randomize. Record for a second and to get some samples.
Now, select output 3, move away from the camera, and click on randomize. Start the recording for half a second.
After setting these three outputs, click "train" and then click "run." Now, when you will drag the green box in the processing window, ChucK will produce music according to the output.
The video below shows how to set up this project and get it working.