mmGAN: Semi-Supervised GAN for Improved Gesture Recognition in mmWave ISAC Systems

Abstract

Integrated sensing and communication (ISAC) has gained significant traction in recent years, primarily because it allows existing communication infrastructure to support sensing applications with minimal additional costs. In particular, millimeter-wave (mmWave) ISAC has the potential to offer improved sensing performance in applications such as pose estimation and gesture recognition. For complex sensing tasks and environments, data-driven sensing, which relies on deep learning, is becoming increasingly popular and has shown promising results. However, deep learning models for these tasks require large labeled datasets to achieve high accuracy. Dataset collection and labeling are labor-intensive and time-consuming. Consequently, there is growing interest in leveraging unlabeled data to overcome these challenges. To address this, we propose mmGAN, a semi-supervised method for ISAC-based gesture recognition. We propose a novel loss function for mmGAN based on softplus, feature matching, and manifold regularization to significantly improve gesture recognition performance. We evaluate mmGAN on a 5G Orthogonal Frequency Division Multiplexing (OFDM) mmWave dataset comprising power per beam pair measurements. When training both mmGAN and the supervised baseline with only 0.6% of the labeled data, mmGAN demonstrates up to 25 percentage points higher accuracy than the supervised baseline. Our method serves as a strong foundation for cross-subject transfer learning, demonstrating the significant value of leveraging unlabeled data to enhance cross-domain sensing performance in ISAC systems. Our results demonstrate that the proposed loss function achieves superior performance across diverse subjects. Further, mmGAN significantly narrows the performance gap between semi-supervised and fully supervised models on the publicly available Widar dataset. Moreover, we provide an interpretable analysis of mmGAN performance through saliency maps and ablation studies, revealing key insights into the model’s behavior and generalization. This work is the first to evaluate gesture recognition performance in 5G OFDM mmWave ISAC systems using a semi-supervised learning approach, covering the entire pipeline from testbed implementation to model evaluation.