transformation AD Rahul Kharche Cranfield University, Shrivenham AD Fest 2005, Nice 14 th - 15 th April 2005 MSAD p.

Transcription

1 Progress with MATLAB Source transformation AD MSAD Rahul Kharche Cranfield University, Shrivenham AD Fest 2005, Nice 14 th - 15 th April 2005 MSAD p. 1/18

2 Agenda Project Goals Previous work on MSAD Further developments Test results from MATLAB ODE examples MINPACK optimisation problems bvp4cad Summary Future Directions References MSAD p. 2/18

3 Project Goals Enhance performance by eliminating overheads introduced by operator overloading in MAD [For04] Explore MATLAB a source analysis and transformation techniques to aid AD Create a portable tool that easily integrates with MATLAB based solvers Provide control over readability of generated code Provide an array of selectable AD specific optimisations a MATLAB is a trademark of The MathWorks, Inc. MSAD p. 3/18

4 Previous work on MSAD Was shown to successfully compute the gradient/jacobian of MATLAB programs involving vector valued functions using the forward mode of AD and source transformation [Kha04] Augmented code generated by inlining the fmad class operations from MAD the derivvec class continued to hold the derivatives and perform derivative combinations resulted in a hybrid approach analogous to [Veh01] Simple forward dependence based activity analysis Active independent variables and supplementary shape size information can be provided through user directives inserted in the code MSAD p. 4/18

5 Previous work on MSAD (contd.) Rudimentary size(shape) and type(constant, real, imaginary) inference Thus removed one level of overheads encountered in MAD giving discernible savings over MAD for small problem sizes but these savings grew insignificant as the problem size was increased MSAD p. 5/18

6 Further developments Now uses size, type inference to specialise and further inline derivvec class operations Optionally generates code for holding and propagating sparse derivatives Incorporated sparsity inference (propagating MATLAB sparse types for derivative variables) if S implies a sparse operand and F full, then rules such as S + F F, S F F S. F S, S&F S T = S(i,j) T is sparse, if i,j are vectors T(i,j) = S T retains its full or sparse storage type are applied MSAD p. 6/18

7 Further developments (contd.) Run-times are obtained using MATLAB 7.0 on a Linux machine with a 2.8GHz Pentium-4 processor and 512MB of RAM. MSAD p. 7/18

8 previous Results - Brusselator ODE CPU(JF)/CPU(F) (log scale) BrusselatorODE CPU(JF)/CPU(F) Vs n NUMJAC (full) MSAD (full) MAD (full) NUMJAC (comp) MSAD (comp) MAD (comp) n (log scale) Jacobian Sparsity Brussode (n = 32) nz = % improvement over MAD for small n down to 4% for large n, with compression performance matches that of finite-differencing, numjac(vec), only asymptotically MSAD p. 8/18

9 previous Results - Brusselator ODE BrusselatorODE CPU(JF)/CPU(F) Vs n CPU(JF)/CPU(F) (log scale) NUMJAC (full) MSAD (full) MAD (full) NUMJAC (comp) MSAD (comp) MAD (comp) CPU(JF)/CPU(F) BrusselatorODE CPU(JF)/CPU(F) Vs n NUMJAC (comp) MSAD (comp) MAD (comp) n n (log scale) 30% improvement over MAD for small n down to 4% for large n, with compression performance matches that of finite-differencing, numjac(vec), only asymptotically using sparse derivatives, performance converges asymptotically to that of MAD almost exponentially increasing savings over full evaluation with increasing n MSAD p. 8/18

10 Results - Brusselator ODE CPU(JF)/CPU(F) (log scale) BrusselatorODE CPU(JF)/CPU(F) Vs n NUMJAC (comp) MSAD (comp) MAD (comp) Jacobian Sparsity Brussode (n = 32) nz = n (log scale) 91% 30% speedup over MAD with increasing n using compression outperforms numjac(vec) n = 640 onward, with gains upto 25% MSAD p. 9/18

11 Results - Brusselator ODE BrusselatorODE CPU(JF)/CPU(F) Vs n CPU(JF)/CPU(F) (log scale) NUMJAC (comp) MSAD (comp) MAD (comp) CPU(JF)/CPU(F) BrusselatorODE CPU(JF)/CPU(F) Vs n NUMJAC (comp) MSAD (comp) MAD (comp) MAD (comp) n n (log scale) 91% 30% speedup over MAD with increasing n using compression outperforms numjac(vec) n = 640 onward, with gains upto 25% decreasing relative speedup, but a small constant saving, over MAD using sparse derivatives MSAD p. 9/18

12 Results - Burgersode ODE CPU(JF)/CPU(F) (log scale) BurgersODE CPU(JF)/CPU(F) Vs n NUMJAC (comp) MSAD (comp) MAD (comp) Jacobian Sparsity Burgersode (n = 32) nz = n (log scale) 87% 37% speedup over MAD with increasing n, using compression outperforms numjac n = 64 onward, with gains between 28% and 45% MSAD p. 10/18

13 Results - Burgersode ODE CPU(JF)/CPU(F) (log scale) BurgersODE CPU(JF)/CPU(F) Vs n NUMJAC (comp) MSAD (comp) MAD (comp) CPU(JF)/CPU(F) BurgersODE CPU(JF)/CPU(F) Vs n NUMJAC (comp) MSAD (comp) MAD (comp) n n (log scale) 87% 37% speedup over MAD with increasing n, using compression outperforms numjac n = 64 onward, with gains between 28% and 45% decreasing relative speedup, but a small constant saving, over MAD using sparse derivatives MSAD p. 10/18

14 Results - Data Fitting problem 10 4 Data Fitting CPU(JF)/CPU(F) Vs n (m = 4) CPU(JF)/CPU(F) (log scale) NUMJAC (full) MSAD (full) MAD (full) n outperforms both MAD and numjac in direct evaluation of the Jacobian by > 60% MSAD p. 11/18

15 Results - Data Fitting problem 10 4 Data Fitting CPU(JF)/CPU(F) Vs n (m = 4) Vandermonde matrix Jacobian Sparsity DataFit (n = 10, m = 4) CPU(JF)/CPU(F) (log scale) NUMJAC (full) MSAD (full) MAD (full) n nz = 30 outperforms both MAD and numjac in direct evaluation of the Jacobian by > 60% if we take note of the sparsity in the Jacobian of the intermediate Vandermonde matrix [For04] and use sparse derivatives, we get an order of magnitude improvement over numjac, but a decreasing relative improvement over MAD MSAD p. 11/18

16 Observations Significantly better performance using Jacobian compression compared to other methods and to numjac, MAD and the previous approach using compression, even for large n MSAD using full evaluation of the Jacobian performs well compared to MAD and numjac using full Decrease in relative performance with increasing n, when using sparse derivatives. MSAD p. 12/18

17 Observations Significantly better performance using Jacobian compression compared to other methods and to numjac, MAD and the previous approach using compression, even for large n MSAD using full evaluation of the Jacobian performs well compared to MAD and numjac using full When using the full or the compressed mode, the generated code contains only native data-types qualifying it for any MATLAB JIT-Acceleration Decrease in relative performance with increasing n, when using sparse derivatives. This can be attributed to the larger overheads in manipulating the internal sparse representation of a matrix, making any savings relatively small MSAD p. 12/18

18 Results - MINPACK problems MINPACK DGL2 CPU(gf)/CPU(f) Vs n MINPACK DSSC CPU(gf)/CPU(f) CPU(gf)/CPU(f) (log scale) NUMJAC (full) MSAD (full) MAD (full) CPU(gf)/CPU(f) (log scale) NUMJAC (full) MSAD (full) MAD (full) n (log scale) Results from 2-D Ginzburg-Landau and Steady-state combustion problems n (log scale) using full derivatives to evaluate the gradient shows 80% 50% improvement over MAD, and outperforms numjac by a similar margin over medium and large n MSAD p. 13/18

19 Results - MINPACK problems MINPACK DGL2 CPU(gf)/CPU(f) Vs n MINPACK DSSC CPU(gf)/CPU(f) CPU(gf)/CPU(f) (log scale) NUMJAC (full) MSAD (full) MAD (full) CPU(gf)/CPU(f) (log scale) NUMJAC (full) MSAD (full) MAD (full) n (log scale) Results from 2-D Ginzburg-Landau and Steady-state combustion problems n (log scale) using full derivatives to evaluate the gradient shows 80% 50% improvement over MAD, and outperforms numjac by a similar margin over medium and large n using sparse derivatives shows a vast improvement over MAD here, 75% 85% MSAD p. 13/18

20 Results - MINPACK problems MINPACK DGL2 CPU(gf)/CPU(f) Vs n MINPACK DSSC CPU(gf)/CPU(f) CPU(gf)/CPU(f) (log scale) NUMJAC (full) MSAD (full) MAD (full) CPU(gf)/CPU(f) (log scale) NUMJAC (full) MSAD (full) MAD (full) n (log scale) Results from 2-D Ginzburg-Landau and Steady-state combustion problems n (log scale) using full derivatives to evaluate the gradient shows 80% 50% improvement over MAD, and outperforms numjac by a similar margin over medium and large n using sparse derivatives shows a vast improvement over MAD here, 75% 85% caused by redundant computations involving some inactive intermediates treated as active in MAD [For04, 7-8] MSAD p. 13/18

21 Results - Smaller problems Ratio CPU(Jf )/CPU(f) Problem n numjac MSAD MAD Coating Thickness Standardization Pollution ODE a Combustion of Propane - Full Human Heart Dipole Chemical AzkoNobel Combustion of Propane - Reduced Amplifier DAE Enzyme Reaction Robertson ODE Smaller sized problems from MINPACK, Test set for IVPs and MATLAB ODE examples almost all cases show an order of magnitude speedup over MAD performance is fairly close to that of finite-differencing(numjac), in four cases better a function vectorised to the advantage of numjac MSAD p. 14/18

22 Results - bvp4cad 6 5 Absolute run times to obtain solutions of BVP problems bvp4c FD bvp4c MSAD bvp4c MAD bvp4cfd bvp4cmsad bvp4cmad absolute time (s) CPU time in sec Test problems 0 I II III IV V VI VII VIII problem Results from bvp4c using MSAD (previous results on the right) significant speedup over previously adopted hybrid approach in MSAD performance better than using numjac a in six of eight cases, and comparable otherwise a note the improved speed using numjac compared to earlier results (previously MATLAB 6.5 was used) MSAD p. 15/18

23 Summary MSAD shows definite improvement in full and compressed Jacobian evaluation over MAD and numjac order of magnitude speedup in small and medium sized test cases In problems with sparsity in the derivatives of results or intermediates, using sparse derivatives in MAD and MSAD shows a large saving over the full evaluation of gradients/jacobian In general, MSAD shows only a constant saving over MAD using sparse derivatives. In certain cases larger gains may be obtained Use of only native data types in the output code allows MATLAB JIT to perform some run-time optimisations MSAD p. 16/18

24 Future Directions Feature enhancement Support for branching constructs involving active variables Handle cells and structures Incorporate exception handling to trap non-differentiability and syntactic errors Improving performance Optimising generated code using dependency analysis (CFG, call-graphs) Use more refined shape inference techniques Apply constant folding Testing Include a mechanism for systematic testing Construct a comprehensive test suite MSAD p. 17/18

25 References [ASU86] A.V. Aho, R. Sethi, and J.D. Ullman. Compilers, Principles, Techniques, and Tools. Computer Science. Addison-Wesley, Reading, Massachusetts, [Eat02] [For04] [Gri00] [Kha04] [SKF03] [Veh01] [Ver98] J.W. Eaton. GNU Octave a high-level language for numerical computations S.A. Forth. An efficient overloaded implementation of forward mode automatic differentiation in MATLAB. Technical report, Cranfield University (RMCS Shrivenham), Swindon, UK, June A. Griewank. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. Frontiers in Applied Mathematics. SIAM, Philadelphia, R. Kharche. Source transformation for AD in MATLAB. Masters thesis, AMOR, Cranfield University, Shrivenham, UK, L.F. Shampine, R. Ketzscher, and S.A. Forth. Using AD to solve BVPs in MATLAB. Technical report, Cranfield University (RMCS Shrivenham), Swindon, UK, A. Vehreschild. Semantic augmentation of MATLAB programs to compute derivatives. Diploma thesis, Institute for Scientific Computing, Aachen University, Germany, A. Verma. Structured Automatic Differentiation. PhD thesis, Cornell University, May MSAD p. 18/18