Autograd Is Slow Per Operation And Free Per FLOP, And That One Sentence Tells You When To Use It
The reverse-mode engine that taught the field how to build autodiff is under two hundred lines of Python. You can read all of it in an afternoon. What you cannot do is make it fast, and the reason is architectural, not a missing optimization.
