Abstract: Objectives Aiming at the problem of complex vibration modes and multiple modal test points of the stern structure of underwater vehicles, a sensor optimization layout method based on genetic algorithm and binary discrete particle swarm hybrid algorithm (GA-BPSO) is proposed in this paper. Methods Firstly, a finite element model of a typical stern structure is established and structural parameters are extracted, and a combined objective function of three-dimensional redundancy elimination index and modal confidence criterion is constructed. Then, the modal measurement points of the stern structure are optimally arranged based on the GA-BPSO algorithm. To verify the effectiveness of the optimization method, modal experiments with uniform and optimal arrangement of measurement points of the stern structure are carried out. Results The results show that, the optimized placement reduces the number of test points from 840 to 200. The maximum off-diagonal element in the modal confidence matrix of the optimized placement is reduced to 0.0333, with frequency errors controlled within 1% and high modal shape correlation. Conclusions The proposed method effectively balances the linear independence of modal shapes and the visualization quality, making it suitable for modal testing of underwater stern structure.