how much slower is this approach compared to automatic differentiation?
keenan
@pw123 It's probably much faster. Why? Because unlike automatic differentiation, you can (i) apply simplifications that reduce the complexity of your expressions, by doing global analysis of the expression tree (which automatic differentiation has no knowledge of), and (ii) once you have a simplified expression, you can translate it into low-level code, which can then benefit from further compiler optimizations (which might consider cache behavior, vectorization, etc., or run performance tuning based on actual execution time).
Automatic differentiation is not really used because it is fast, more because it is convenient, and can easily be worked into a lot of existing code. In general, I'd be wary of using off-the-shelf tools because that's what everyone else is doing. For instance: we recently tried using TensorFlow for a problem we were working, and discovered it was abysmally slow compared to symbolic differentiation code we wrote ourselves. Why? Simply because TensorFlow is written with a certain problem structure in mind (machine learning workloads involving "large" data vectors); our problem looked different (highly heterogeneous multi-objective optimization with a much smaller number of variables).
This image kinda reminds me of Bayes Nets
how much slower is this approach compared to automatic differentiation?
@pw123 It's probably much faster. Why? Because unlike automatic differentiation, you can (i) apply simplifications that reduce the complexity of your expressions, by doing global analysis of the expression tree (which automatic differentiation has no knowledge of), and (ii) once you have a simplified expression, you can translate it into low-level code, which can then benefit from further compiler optimizations (which might consider cache behavior, vectorization, etc., or run performance tuning based on actual execution time).
Automatic differentiation is not really used because it is fast, more because it is convenient, and can easily be worked into a lot of existing code. In general, I'd be wary of using off-the-shelf tools because that's what everyone else is doing. For instance: we recently tried using TensorFlow for a problem we were working, and discovered it was abysmally slow compared to symbolic differentiation code we wrote ourselves. Why? Simply because TensorFlow is written with a certain problem structure in mind (machine learning workloads involving "large" data vectors); our problem looked different (highly heterogeneous multi-objective optimization with a much smaller number of variables).
TL;DR: Think for yourself; question authority.