Smart mirrors for fitness: pose estimation and multi-person tracking

In the rapidly evolving landscape of fitness technology, developers are constantly seeking innovative ways to enhance user experiences and outcomes. A significant challenge in this domain is improving the quality of fitness workouts without the constant need for a personal coach. The use of smart mirrors for fitness workouts is a prime example of how technology can help, and embedding AI inside the mirror offers a transformative solution to these challenges with several benefits:

Real-Time Analysis: Immediate feedback on workout quality and posture correction.

Cost-Effectiveness: Eliminates the need for expensive coaching fees.

24/7 Availability: Provides continuous support and feedback at any time.

Personalization: Tailors workout plans based on user profile and real-time data.

Autonomy: Operates without the need for a constant connection to the cloud, ensuring privacy, reducing latency, and enhancing reliability.

We’ll dig more into the implementation details in this use-case. In the meantime, you can learn more about the benefits of Edge AI here.

Application principle

The image of the scene is captured by an image sensor. On one end it’s processed to be displayed on the LCD screen. On the other end, the image is downscaled to be sent to the neural network.

Approach

The most effective way today to perform pose estimation applications is using convolutional neural networks (CNN).
We used one of the models available in the STM32 Model zoo. The model is the Yolov8n_pose, provided by our partner Ultralytics. This is the state-of-the-art model for pose estimation application, providing accurate keypoints with high frame rate.
Such model requires high computational capabilities, and the most suitable MCU option is the STM32N6 which embeds a neural processing unit, the Neural-ART accelerator.
To easily optimize and deploy this model on the STM32N6, we used the scripts provided in the STM32 Model zoo and the STM32Cube.AI code optimizer.
These technologies are part of a broader range of Edge AI solutions. Check out the ST Edge AI Suite by clicking here.

Sensor

RGB Image sensor.

In the STM32 Model Zoo, we provide the code example with two different image sensors: Sony IMX335 5Mp RGB (part of the STM32N6570-DK) or ST’s VD66GY 1.5 Mp RGB Global shutter (ideal for precise movement capture).

Dataset and model

Dataset: Internal (ST-modified Coco 201 Person)
Model: Yolov8n_pose (implemented in Pytorch by Ultralytics and quantized in int8 format using tensorflow lite converter). Input size: 256 x 256 x 3

Results

Weights: 3.2 MB
Activations: 1.1 MB
Inference time: 36 ms
Inference per second: 28
Pose mAP50 (mean Average Precision): 51.1 %

The yolov8 is a state of the art model for multi-pose estimation. Different tradeoff can be achieved by using other dataset, model or resolution. Balancing these factors is crucial for optimizing AI models to meet specific performance and hardware resource constraints. Find more models in the STM32 Model zoo.

Get started now!

Easily recreate this use-case with the following resources:
– Step by step tutorial
– Yolov8n_pose pre-trained model
– Application code example

Additional resources:
– How to install the STM32 Model zoo

Author: Vincent RICHARD / Last updated: January, 2025