Abstract: Objectives Aiming at the problems of easy switching of tracking ID and low accuracy caused by complex scenes such as platform shaking and small target caused by long distance during the working process of cleaning ships, a multi-target tracking method based on improved YOLOv5-Byte was proposed. Methods Firstly, the Byte data association model was fused with the YOLOv5 detector to construct the multi-target tracking algorithm. Secondly, aiming at the problem that CIoU in YOLOv5 is sensitive to small targets, the normalized Wasserstein distance metric is proposed for the boundary box Gaussian modeling. Then, the balance factor φ is introduced to adjust the contribution of CIoU and normalized Wasserstein distance measures to the loss function to adjust the sensitivity of the detector to small targets. Finally, IoU is introduced into the adjustable hyperparameter α in the form of a power exponent in the Byte data association model to reduce the risk of small targets being discarded due to low confidence.Results The experimental results on the surface floating garbage data set showed that IDF1 and MOTA increased by 11.5% and 8.7% respectively, and IDs decreased for 7 times compared with the improvement.Conclusions The algorithm achieves accurate tracking of multiple small targets on the surface of the water, and provides a reference for the autonomous fishing technology of clean ships.