This post is about automatic differentiation, a programming technique allowing to compute automatically the derivative of a function from its source code. This technique, in its reverse mode, is attributed to Seppo Linnainmaa who introduced the idea in his Master thesis in 1970.
How to compute the derivative of a numerical function? If we have a symbolic mathematical expression for it, then we can use the rules of differential calculus, and this can be automated: this is symbolic differentiation. If the function is a black box, then we can compute numerically the differential ratio by using evaluations for very close arguments: this is numerical differentiation. But actually, if we have the source code of the function, one could follow the code and create an augmented code in parallel that computes the effect of each elementary operation on the derivative: this is known as automatic differentiation. Automatic differentiation can be implemented by using object programming and operators overloading. Automatic differentiation can be in trouble in the presence of conditionals.
Nowadays automatic differentiation is fashionable, partly because of the development of deep machine learning and multi-layered neural networks. For more, take a look at the Wikipedia article. If you like the Julia programming language, you may take a look at JuliaDiff.
Leave a Comment