Molecular motor proteins serve as an essential component of intracellular transport by generating forces to haul cargoes along cytoskeletal filaments. Two species of motors that are directed oppositely (e.g. kinesin, dynein) can be attached to the same cargo, which is known to produce bidirectional net motion. Although previous work focuses on the motor number as the driving noise source for switching, we propose an alternative mechanism: cargo diffusion. A mean-field mathematical model of mechanical interactions of two populations of molecular motors with cargo thermal fluctuations (diffusion) is presented to study this phenomenon. The delayed response of a motor to fluctuations in the cargo velocity is quantified, allowing for the reduction of the full model a single “characteristic distance”, a proxy for the net force on the cargo. The system is then found to be metastable, with switching exclusively due to cargo diffusion between distinct directional transport states. The time to switch between these states is then investigated using a mean first passage time analysis. The switching time is found to be non-monotonic in the drag of the cargo, providing an experimental test of the theory.