Action-Conditioned Hamiltonian Generative Networks (AC-HGN) for Supervised and Reinforcement Learning

Abstract

This paper introduces Action-Conditioned Hamiltonian Generative Networks (AC-HGN), a physics-informed neural network architecture which learns Hamiltonian dynamics in environments subject to state-dependent external forces. AC-HGN embeds control inputs of any form into an abstract phase space, extending abstract Hamiltonian dynamics with learned external forces. In a supervised setting, results show that AC-HGN surpasses the prediction accuracy of state-of-the-art Lagrangian Neural Networks when trained on a static dataset. Furthermore, AC-HGN can be readily used as a physics-informed world model in a Model-Based Reinforcement Learning (MBRL) setting by embedding policy actions as external forces. Due to the autoencoder structure of AC-HGN, this marks the first Physics-Informed MBRL algorithm which is not reliant on any domain knowledge and is not limited to specific input modalities. Experimental results demonstrate that AC-HGN achieves competitive sample efficiency and asymptotic performance in simple environments, with minimal degradation in more complex environments, while significantly outperforming an uninformed world model. We conclude that the proposed architecture can accurately and efficiently capture environment dynamics and external forces in a Hamiltonian fashion while requiring no domain-specific knowledge, improving the applicability of physics-informed neural networks in supervised and reinforcement learning settings.