Deep Reinforcement Learning + Image Prediction Model

Overview

Reinforcement learning aims to maximize the value of the observed state up to the present. Since value represents the expected reward that can be obtained in the future, predicting future states enables a more accurate estimation of the current state’s value.

In this study, we introduce a time-series model to predict future states when calculating value. To achieve this, we employ a deep learning model that predicts future images based on sequential image data. By incorporating predicted future states, we expect to achieve higher rewards at an earlier stage.

Proposed Method

Reinforcement Learning with an Image Prediction Model

We propose a reinforcement learning method based on a value function that incorporates an image prediction model. To accurately estimate the difference between the current and future values, we integrate the predicted future state’s value into the loss function of the Q-network. This allows the agent to make decisions that take future states into account, promoting long-term optimization rather than merely maximizing immediate rewards.

Predictive Image Generation

For predictive image generation, we adopt Convolutional Dynamic Neural Advection (CDNA). CDNA generates a predicted image one frame ahead by using a sequence of consecutive frames and conditioning on the observed movements and postures of objects in those frames.

Evaluation Experiments and Results

Line-Tracing Task

To evaluate the performance of the proposed method, we conducted experiments using a line-tracing task. Specifically, we analyzed how the agent learns in environments with different numbers of prediction frames and how rewards change accordingly.

The results confirmed that introducing an appropriate number of prediction frames enables the agent to achieve higher rewards in the early stages of learning. In particular, when the number of prediction frames is set appropriately, learning stability improves, and the time required to reach high rewards is reduced. However, when the number of prediction frames is excessively large, error accumulation increases, leading to lower rewards.

CartPole Task

We also conducted similar evaluation experiments using the CartPole task. The results showed that selecting an appropriate number of prediction frames improved the rewards. However, compared to the line-tracing task, the degree of reward improvement varied, suggesting that the optimal number of prediction frames differs depending on the task.

Related Publications

— Yoshiki Kato, Tomohiro Nishikata and Yuji Yamauchi, “Incorporating Image Prediction Model in Value Function for Deep Reinforcement Learning”, Journal of the Japan Society for Precision Engineering, vol. 91, no.4, pp.518-525, 2025. link