Pose Estimation of Deformed AR Markers

Overview

In recent years, 2D codes have become widely used for various applications such as cashless payments and robot localization.
However, recognition and pose estimation become challenging when these codes are deformed.

To address this issue, we propose a fast and accurate 3D pose estimation method for deformed AR markers.
Our approach improves speed and accuracy through:
modifying the object detector, extending the Augmented Autoencoder (AAE), introducing regression, and alternating optimization of two models.
We also validated its practicality on an embedded board.

Proposed Method

Flow of Pose Estimation for Deformed AR Markers

The proposed method estimates the position and orientation as follows:

1. Detect deformed AR markers using NanoDet-Plus
2. Extract latent variables from detected markers using the encoder
3. Estimate pose from the latent variables using an MLP
4. Estimate position based on bounding box and predicted pose

Introduction of NanoDet-Plus

The object detector was changed from the conventional Single Shot Multibox Detector (SSD) to NanoDet-Plus.
NanoDet-Plus is a lightweight and fast anchor-free object detector.
By using this, we achieved both high-speed and high-precision detection of deformed AR markers.

Extension of AAE

We introduce an Augmented Variational Autoencoder (AVAE), which integrates a Variational Autoencoder (VAE) into the conventional AAE.
Combining AAE’s capability of removing noise from images with VAE’s ability to learn latent structures behind the data,
we aim to capture the continuity of poses while eliminating background and deformation from deformed markers.

Alternating Optimization of Two Models

The AVAE removes deformations, and the Multi-Layer Perceptron (MLP) estimates poses.
Since it’s difficult to optimize both models simultaneously, we alternate the optimization steps.
The AVAE’s encoder and decoder are trained to minimize the reconstruction error,
while the encoder and MLP are trained to minimize pose estimation error.
Repeating these steps enables the encoder to output latent variables that contain only essential information for pose estimation while excluding noise and deformation.

Experiments and Results

We compare our method with a conventional method using SSD + AAE + Nearest Neighbor for pose estimation.
Our method reduced the average position error by 26.9% and average orientation error by 62.7%.
Moreover, processing time per frame improved by 99.3%.

Pose Estimation on Deformed AR Markers in Real Environment

The following shows the pose estimation results for deformed AR markers in real environments.
The color of the wireframe represents the marker ID, and its size and orientation represent the estimated position and pose.

Related Publications