Distributed Training Instability

Let’s say we’ve built a neural network for image classification using NumPy, and we’re training it in a distributed environment with several workers. We notice that, even when using the same random seed, the loss curve looks erratic across different training runs, model weights diverge between workers, and training results aren’t consistent between single-node and distributed setups.

How would you go about diagnosing and resolving these distributed training inconsistencies to ensure stable and reproducible results?