Backpropagation Explanation

What does the backpropagation algorithm do in the context of neural networks? What is the informal intuition behind the algorithm? What are some drawbacks of the algorithm compared to other optimization methods?

Bonus: Formally derive the backpropagation algorithm and prove that it does what it claims to do.