Advanced Analytics: Plant a (decision) TREE and save the world*!

Transcription

1 Advanced Analytics: Plant a (decision) TREE and save the world*! Vivek Nair North Carolina State University vivekaxl@gmail.com vivekaxl.com * Configure software using less resources

2 Most Valuable Point Information is a source of learning. But unless it is organized, processed, and available to the right people in a format for decision making, it is a burden, not a benefit -- Dr. William Pollard

3 Decision Trees - Use Cases TAR(ZAN)2 [] 22 [] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 Treatment learner." Artificial Intelligence Review 25.3 (26):

4 TAR(ZAN)2 Learns Small Theories Problem: Find picture on a page from features

5 TAR(ZAN)2 Learns Small Theories Problem: Find picture on a page from features

6 TAR(ZAN)2 Learns Small Theories Find picture on a page from features

7 TAR(ZAN)2 Learns Small Theories Find picture on a page from features (34 height 86) (3.9 mean_tr < 9.5)

8 TAR(ZAN)2 Learns Small Theories Find picture on a page from features (34 height 86) (3.9 mean_tr < 9.5)

9 Decision Trees - Use Cases TAR(ZAN)2 [] 22 [] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 Treatment learner." Artificial Intelligence Review 25.3 (26):

10 Decision Trees - Use Cases TAR(ZAN)2 [] 22 SWAY[2] 26 Performance Optimization[3] XTREE[4] [] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 Treatment learner." Artificial Intelligence Review 25.3 (26): [2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 26. [3] Guo et al. Variability-aware performance prediction: A statistical learning approach." ASE-. [4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-

11 Decision Trees - Use Cases TAR(ZAN)2 [] 22 Optimization SWAY[2] 26 Software Variability Performance Optimization[3] Planning XTREE[4] [] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 Treatment learner." Artificial Intelligence Review 25.3 (26): [2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 26. [3] Guo et al. Variability-aware performance prediction: A statistical learning approach." ASE-. [4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-

12 Decision Trees - Use Cases TAR(ZAN)2 [] 22 Optimization SWAY[2] 26 Software Variability Performance Optimization[3] Planning XTREE[4] [] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 Treatment learner." Artificial Intelligence Review 25.3 (26): [2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 26. [3] Guo et al. Variability-aware performance prediction: A statistical learning approach." ASE-. [4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-

13 Configurable Systems and Variability System

14 Configurable Systems and Variability input.y4m System

15 Configurable Systems and Variability input.y4m System output.x264

16 Configurable Systems and Variability Configuration options c c2 c3 c4 input.y4m... System cn output.x264

17 Configurable Systems and Variability Configuration options c c2 c3 c4 input.y4m... System cn output.x264 Non-functional behavior: response time, throughput, etc.

18 Configurable Systems and Variability Configuration options c c2 c3 c4 input.y4m... System cn output.x264 Non-functional behavior: response time, throughput, etc.

19 Performance Optimization is Necessary!

20 Performance Optimization is getting more Complex! Necessary [] Xu et. al.. Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software.fse [2] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM,.

21 Performance Optimization is required since Default Configuration is Bad! Necessary Complex [] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM,. [2] Herodotou, Herodotos, et al. "Starfish: A Self-tuning System for Big Data Analytics." CIDR

22 Performance Optimization can be Expensive! Evaluation of single instance of software/hardware co-design problem can take weeks[] Necessary Complex Rolling Sort use-case required 2 days, within a total experimental time of about 2.5 months[2] Test suite generation using Evolutionary Algorithm can take weeks[3] Default is bad Image recognition workload and speech recognition workload, jobs ran for many hours or days[4] [] Zuluaga, Marcela, et al. "Active learning for multi-objective optimization." International Conference on Machine Learning.. [2] Jamshidi, Pooyan, and Giuliano Casale. "An uncertainty-aware approach to optimal configuration of stream processing systems."mascots-26 [3] Wang, Tiantian, et al. "Searching for better configurations: a rigorous approach to clone evaluation." FSE- [4] Venkataraman, Shivaram, et al. "Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics." NSDI. 26.

23 Is it pervasive? Cloud Computing - Ernest - Cherrypick - PARIS Necessary Complex Default is bad Expensive Machine Learning - Hyperparameter Tuning - Random search - SMBO - Fabolas Database - otter-tune - ituned Software Engineering - Tuning or Default Values? - Tuning for Software Analytics - Tuning for Defect Prediction - Topic Modelling

24 Performance Optimization! Necessary Complex Default is bad Expensive Pervasive

25 Performance Optimization! Necessary Complex Default is bad Expensive Pervasive

26 Road Map

27 Road Map Residual-based Methods

28 Road Map Rank-based Method

29 Road Map Bayesian-based Method

30 Road Map Surrogate is a cheap(er) version of the actual system Request Request Surrogate APACHE http server APACHE http server Response Time Response Time

31 Progressive Sampling Guo, Jianmei, et al. "Variability-aware performance prediction: A statistical learning approach." ASE-.

32 Residual-based Methods Progressive Sampling How to find the best performing configuration for any given system?

33 Residual-based Methods Progressive Sampling Configuration Space

34 Residual-based Methods Progressive Sampling Testing Pool Training Pool 4% Configuration Space 2% Validation Pool 4%

35 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space

36 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space

37 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Train

38 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF MRE < : Exit ELSE: continue

39 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF MRE < : Exit ELSE: continue Mean Relative Error MRE = actual - predicted actual

40 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF MRE < : Exit ELSE: continue

41 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF MRE < : Exit ELSE: continue

42 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF MRE < : Exit ELSE: continue

43 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF MRE < : Exit ELSE: continue

44 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space

45 Residual-based Methods Progressive Sampling Testing Pool Training Pool Validation Pool Configuration Space Use the model to find the best configuration

46 Residual-based Methods Progressive Sampling - Limitation The stopping condition is arbitrary Cannot estimate cost required to build a surrogate

47 Projective Sampling Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems." ASE.

48 Residual-based Methods Projective Sampling Given an acceptable accuracy estimate (τ), how many samples is required to build a quality surrogate?

49 Residual-based Methods Projective Sampling Learning Curve

50 Residual-based Methods Projective Sampling Learning Curve

51 Residual-based Methods Projective Sampling Learning Curve

52 Residual-based Methods Projective Sampling Learning Curve Steep Incline

53 Residual-based Methods Projective Sampling Learning Curve Gradual Incline Steep Incline

54 Residual-based Methods Projective Sampling Learning Curve Pleatue Gradual Incline Steep Incline

55 Residual-based Methods Projective Sampling Optimal Sample Size Learning Curve Plateau Gradual Incline Steep Incline

56 Residual-based Methods Projective Sampling Estimates the Learning Curve Generate initial data points for the learning curve Find best fit projective function Calculate optimal training set Build surrogate

57 Residual-based Methods Projective Sampling Estimates the Learning Curve Generate initial data points for the learning curve Find best fit projective function Calculate optimal training set Build surrogate

58 Residual-based Methods Projective Sampling Estimates the Learning Curve Generate initial data points for the learning curve Find best fit projective function Calculate optimal training set Build surrogate Requirement: Initial samples should reflect relationship between all configuration options Intuition: Performance depends if configuration option is selected or deselected Heuristic: Feature Frequency - initial samples have each option selected or deselected, at least, δ times

59 Residual-based Methods Projective Sampling Testing Pool Training Pool Configuration Space

60 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Accuracy Configuration Space Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected

61 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Accuracy Configuration Space c c2 c3 c4 Train Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected

62 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Accuracy 5% Configuration Space c c2 c3 c4 Train Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected

63 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Accuracy 5% Configuration Space c c2 c3 c4 Train Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected

64 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Configuration Space c c2 c3 c4 Accuracy 5% 2 7% Train Feature frequency table (δ=2) c c2 c3 c4 Selected 2 Deselected

65 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Configuration Space c c2 c3 c4 Accuracy 5% 2 7% Train Feature frequency table (δ=2) c c2 c3 c4 Selected 2 Deselected

66 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Configuration Space c c2 c3 c4 Accuracy 5% 2 7% 3 29% Train Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected 2

67 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Configuration Space c c2 c3 c4 Accuracy 5% 2 7% 3 29% Train Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected 2

68 Residual-based Methods Projective Sampling Testing Pool Training Pool #Samples Configuration Space c c2 c3 c4 Accuracy 5% 2 7% 3 29% 4 35% Train Feature frequency table (δ=2) c c2 c3 c4 Selected Deselected

69 Residual-based Methods Projective Sampling Estimates the Learning Curve Generate initial data points for the learning curve Find best fit projective function Calculate optimal training set Build surrogate

70 Residual-based Methods Projective Sampling #Samples Accuracy 5% 2 7% 3 29% 4 35%

71 Residual-based Methods Projective Sampling Exponential Log Power

72 Residual-based Methods Projective Sampling Estimates the Learning Curve Generate initial data points for the learning curve Find best fit projective function Calculate optimal training set Build surrogate

73 Residual-based Methods Projective Sampling Penalty factor Taken from Sarkar et al. Coefficients of projective function Number of configurations whose performance value will be predicted

74 Residual-based Methods Projective Sampling Estimates the Learning Curve Generate initial data points for the learning curve Find best fit projective function Calculate optimal training set Build surrogate

75 Residual-based Methods Projective Sampling Testing Pool Training Pool Validation Pool Configuration Space Optimal Sample Size Use the model to find the best configuration

76 Residual-based Methods Projective Sampling - Limitation Assumes an accurate model can be built

77 Rank-based Method Nair, Vivek, et al. "Using Bad Learners to find Good Configurations." FSE

78 Rank-based Method

79 Rank-based Method

80 Rank-based Method

81 Rank-based Method

82 Rank-based Method What happens when an accurate model cannot be built?

83 Rank-based Method Core Insights

84 Rank Preserving Model

85 Rank Preserving Model

86 Rank Preserving Model

87 Rank Preserving Model

88 Rank Preserving Model

89 Rank Preserving Model

90 Rank Preserving Model

91 Rank Preserving Model Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF accuracy < : Exit ELSE: continue

92 Rank Preserving Model Testing Pool Training Pool Validation Pool Configuration Space Check Performance Train τ IF accuracy < : Exit ELSE: continue

93 Rank Preserving Model - Limitation Testing Pool Training Pool 4% Configuration Space 2% Validation Pool 4% Requires Testing Pool - 2% of configuration space

94 - Bayesian based Method Nair, Vivek et al. "FLASH: A Faster Optimizer for SBSE Tasks." preprint

95 Bayesian-based Method Can we avoid measurement of configurations in the testing pool?

96 Bayesian-based Method Bayesian Optimization Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

97 Bayesian-based Method Bayesian Optimization True Performance Distribution Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

98 Bayesian-based Method Bayesian Optimization True Performance Distribution Predicted Performance Distribution Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

99 Bayesian-based Method Bayesian Optimization True Performance Distribution Predicted Performance Distribution Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

100 Bayesian-based Method Bayesian Optimization True Performance Distribution Predicted Performance Distribution Acquisition Function Taken from Dr. Nando de Freitas (tiny.cc/4tgeny) Which configuration should I evaluate next?

101 Bayesian-based Method Bayesian Optimization True Performance Distribution Predicted Performance Distribution Acquisition Function Taken from Dr. Nando de Freitas (tiny.cc/4tgeny) Which configuration should I evaluate next?

102 Bayesian-based Method Bayesian Optimization True Performance Distribution Predicted Performance Distribution Acquisition Function Taken from Dr. Nando de Freitas (tiny.cc/4tgeny) Tradeoff between Exploration vs Exploitation

103 Bayesian-based Method Bayesian Optimization True Performance Distribution Predicted Performance Distribution Acquisition Function Taken from Dr. Nando de Freitas (tiny.cc/4tgeny) Surrogate of choice: Gaussian Processes (GP)

104 Bayesian-based Method Bayesian Optimization Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

105 Bayesian-based Method Bayesian Optimization Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

106 Bayesian-based Method Bayesian Optimization GP lose efficiency in high dimensional spaces i.e. number of features exceeds a dozen

107 Bayesian-based Method - FLASH GP

108 Bayesian-based Method - FLASH GP

109 Bayesian-based Method - FLASH

110 Bayesian-based Method - FLASH

111 Bayesian-based Method - FLASH

112 Bayesian-based Method - FLASH

113 Bayesian-based Method - FLASH

114 Bayesian-based Method - FLASH

115 Bayesian-based Method - FLASH Fast Effective Comprehensible

116 Conclusion ML Algorithms are not a black box How to use Decision Tree in Planning? Can I explain the results to the Decision Makers? Lazy is good Only do what is required Optimization does not require an accurate model Easy over Hard Try simplest first Tuning SVM outperforms DL

117 Rank-based Method: Flash: epal: Bayesian Optimization: "Look for me...beneath the tree...north!" Three-eyed raven