This project leverages the NXP FRDM-MCXN947 development board to detect epileptic seizures in real-time via video analysis. The board's NPU accelerates machine learning tasks, enabling efficient edge AI processing. The system captures and analyzes video to identify seizure patterns, providing immediate alerts and historical data to improve care.
Introduction
Epilepsy, a neurological disorder affecting approximately 65 million people worldwide, is often characterized by the recurrent seizures that leave both the subject and viewers in a state of shock. Historically, video feeds capturing an incident would be analyzed post hoc, and although valuable data and behavioral patterns were extracted that led to significant advancements, it lacks the real-time alerting and monitoring that modern advancements to computer vision and machine learning offer.
In this project, the NXP FRDM-MCXN947 is explored as a development board that serves as the basis for this system, as it's dual-core Arm Cortex M33 alongside the integrated Neural Processing Unit (NPU) enable efficient machine learning inference at the edge. With a range of expansion options and features, the dev board can be connected to an external camera as an input for live video feed while the ethernet (or WiFi) can be used to connect the device to the internet, allowing for remote management and alerting capabilities. Furthermore, the option to expand the system with additional components that may aid in the detection, analysis, or data collection aspects is always available. With that, time to dive into the design.
Machine Learning
Before we begin, a quick disclaimer. I am by no means a medical professional. That said, I strive for the truth, so if the information I present is not accurate, please reach out and I'll correct it. On with the show...
First, what is a seizure? A seizure is a sudden, abnormal burst of electrical activity in the brain. For the maker and electronic audience that Maker Pro attracts - think of it as a short circuit. These electrical bursts can cause a variety of symptoms, depending on the area of the brain affected.
Part of the seizure lifecycle includes several phases. To stay within the scope of the project, we'll focus on two stages that occur pre-seizure: prodromal and early ictal. In the prodromal phase, symptoms may include confusion, anxiety, tremors, spurts of anger, or other mood disturbances. The next phase, early ictal (also referred to as "aura"), includes visual symptoms such as muscle jerking, nausea and dizziness, difficulty speaking or understanding speech, or tics.
These stages are the primary target of our application, as a successful detection during these precursory stages will enable sufficient preparation for minimizing threats. This can include ensuring the person is in a safe place, away from objects that can cause injury, and alerting someone to be on standby.
Data Collection
To begin the design, data collection is a necessary prerequisite. Both video and image datasets should be collected. Video datasets of recordings of patients with seizures under different lightings and severity will help make the model more accurate. Additionally, clips of the pre-seizure symptoms discussed earlier, such as muscle twitching or slurred speech (leveraging microphone) can be an aid when training the model. Next, still photographs showcasing the emotional aspects that one might undergo, such as confusion or anger, can contribute to the growing dataset as well. Although a tedious process, datasets should be annotated appropriately with labels for identifying the different characteristics. Although often not publicly shared, several medical research institutions have data collected that they would be willing to share.
Model Training
With the data collected, we begin by selecting an architecture, such as CNN, RNN, or a combination of several models, like ConvLSTM. The model can then be trained using frameworks like TensorFlow in an iterative approach, where hyperparameters are adjusted to improve accuracy and reduce false positives. Once the model reaches a state of confidence, validate the model using a new dataset of unseen data. Post-training quantization can reduce the model size while also improving CPU and hardware accelerator latency, with little degradation in model accuracy. When converting a TensorFlow model too TensorFlow Lite format, an option for quantizing the model is included in the flow. Additionally, ensuring the model can fit within the constraints of the microcontroller and compressing accordingly using techniques like weight pruning can be employed during these stages. Now, let's get the model onto the dev board.
Model Integration
The eIQ NPU is a highly scalable acceccelerator core architecture providing machine learning acceleration. It offers support for a wide variety of neural network types, such as convolutional neural networks (CNN), recurrent neural network (RNN), long short term memory (LSTM), and more. Baked into the MCUXpresso SDK includes a comprehensive software suite that includes the TensorFlow Lite for Microcontroller (TFLM), designed to run ML models with just a few kilobytes of memory and being the perfect fit for low-power computing devices. To accelerate the model with the NPU, the model must first be converted by the Neutron Converter command-line tool that is included in the eIQ toolkit.
Converting model with the following options:
Input = seizures.tflite
Output = seizures_converted.tflite
Target = mcxn84x
Run dependencies:
The converted model graph can be viewed in the eIQ portal, displaying the custom NeutronGraph nodes, an example below for visual demonstration. Additionally, a comparison of the input and output file should show some notable differences with regards to file size.
We will reference this model shortly, but let's turn to the simple hardware setup for now.
Hardware
To keep things brief, particularly as this writeup consists of the design phase of the project, this section will scratch the surface regarding the components and it's usecase, but no deep dive.
First, the dev board itself, the NXP FRDM-MCXN947
Like was mentioned earlier, leveraging the NPU is key for this application. Ethernet or WiFi can be used for connectivity. I have been exploring the idea of creating a minimalist web interface/API that will enable the system to double as a live remote camera feed, such as when a loved one wants to check in on partner with epilepsy. This same API can serve as a backbone for the alerting functionality.
The camera model chosen, OV7670, is a cheap, but common camera module in the maker community.
While it should get the job done, there are other options which offer higher pixel resolution. Although self-explanatory, this module is being used for capturing the live feeds.
Finally, the microphone, SparkFun's Analog MEMS breakout board.
Like the camera, it's cheap, popular, and gets the job done. While the initial thoughts of its usage were targeted at recognizing auditory patterns associated with seizures, there is also a potential use case of word detection via speech-to-text. Consider a scenario where in the event the microphone picks up the word "pineapple", it'll alert those monitoring the application.
Software
Considering this is the design phase, the software explanation will be brief, however strongly urge any readers to reference the library of excellent documentation NXP provides. In short, the user must install the MCUxpresso SDK, select the appropriate board, and the eIQ middleware.
In one of the assembly files, the file path to the converted TensorFlow Lite model can be referenced. With the model now referenced (and assuming the code works as expected), building and flashing the board should result in our envisioned design running. With real-time video feeds coming in from the camera module, the model inference runs and software logic determines whether the need for alerting exists.
Several matters should be taken into consideration during the development of the application. First, low power modes should be implemented to conserve energy when the system is idle. Computer vision techniques such as downscaling, normalizing pixel values, and some additional filtering should optimize the system as well.
Conclusion
Building a real-time video feed analysis system for detecting seizures is no simple matter, but it's one that can potentially reap many benefits. Many systems the already exist for detecting seizures rely upon EEG, or electroencephalogram, signals. Powerful dev boards like the NXP FRDM-MCXN947 are changing the game, however, with onboard NPUs that bring ML/AI capabilities to the edge. Hopefully advancements in these fields result in meaningful, safe, and impactful benefits to those fighting medical challenges.
