Optimizer

Backpropagation is the core algorithm behind how neural networks learn. Let's first do an intuitive walk through on what the algorithm is actually doing without any reference to the mathematics behind it. For those of you who do want to know how the math works, there is an appendix that goes through the calculus underlying backpropagation. So far we learnt how neural networks feeds forward information to predict an outcome. In our example of housing price prediction, each houses characteristics such as year built and renovated are fed into the first layer of the network. What we mean by learning is to find which weights and biases minimize a certain cost function. as a quick reminder, as a cost of a single training example, what you do is to take the output that the network gives, along with the observed output, and just add up the square of differences between each component. Doing this for tens of thousands of examples and averaging the results gives the total cost of the network. Then, we look for the negative gradient of this cost function which tells the network on how to change the weights and biases of all connections, so as to most efficiently decrease the cost. 
Backpropagation is an algorithm to compute the gradient. In effect, the cost function is optimized for each training example, and therefore, each training example has an effect on how weights and biases are adjusted. 

Metrics