Training Instability in Neural Networks

You’re AI engineer working on an energy forecasting system at Siemens that helps utilities predict short-term electricity demand to balance grid load and reduce outages. You train a recurrent neural network on long historical time series (weather, calendar effects, local demand signals) to forecast energy demand.

Early in training, the model appears to learn normally, but after several epochs the loss becomes highly unstable, oscillating wildly before turning into NaNs. At the same time, training slows dramatically and the model fails to recover even after restarting from recent checkpoints.

You’ve verified that the data pipeline is correct and labels are well-aligned with inputs. No changes were made to the dataset, but increasing the sequence length or learning rate makes the problem significantly worse.

How would you investigate the cause of this training instability? What signals would you look for during debugging, and how would those findings guide the changes you make to the model or training setup?