Abstract: Objective To address the problems of texture loss, viewpoint drift, and unstable estimation in shipborne vision-based wave measurement under nighttime low illumination, rain-fog occlusion, strong specular reflection, and vessel roll-pitch-yaw motions, this study investigates an all-weather ocean wave perception method based on multispectral monocular vision, aiming at robust inversion of key wave parameters such as dominant wave direction. Methods A shipborne multispectral acquisition system integrating visible-light and long-wave infrared imaging with an inertial measurement unit (IMU) was constructed. Image and attitude data were time-synchronized and organized into temporal samples using a sliding window. Contrast Limited Adaptive Histogram Equalization (CLAHE) and U-Net segmentation were employed to enhance images and extract effective sea-surface regions while removing ship structures, sky-sea horizon, wake foam, and occlusions. A ResNet-18 encoder with transfer learning was used for frame-wise feature extraction, and multimodal fusion was achieved via trigonometric mapping of attitude angles and multilayer perceptron embedding. A lightweight multi-head self-attention mechanism was introduced to model inter-frame correlations and regress the two-dimensional unit vector of wave direction, followed by angle recovery using the atan2 function. A multi-camera geometric consistency verification and fusion strategy was further designed to improve engineering reliability. Results Field experiments conducted on an engineering vessel in the Zhuhai coastal waters show that, under visible-light conditions, the mean absolute error of wave direction estimation is 0.41° with a standard deviation of 0.28° and a success rate of 99.90%. Under long-wave infrared conditions, the mean absolute error is 1.14°, the standard deviation is 0.89°, and the success rate is 98.94%. Stable and smooth directional trends can still be obtained in nighttime and low-light environments, and temporal modeling effectively suppresses output fluctuations caused by transient occlusion and short-term loss of sea-surface regions of interest.Conclusions The proposed multispectral monocular vision-IMU multimodal temporal perception framework achieves high accuracy and robustness under complex sea states and dynamic shipborne conditions. Long-wave infrared imaging significantly enhances all-weather observation capability, providing technical support for intelligent shipborne sea-state perception and the engineering deployment of ocean observation systems.