Scheduling and Optimization of Fault-Tolerant Embedded Systems

Transcription

1 Scheduling and Optimization of Fault-Tolerant Embedded Systems, Viacheslav Izosimov, Paul Pop *, Zebo Peng Department of Computer and Information Science (IDA) Linköping University * Department of Informatics and Mathematical Modelling Technical University of Denmark 1

2 We want systems to be fault tolerant! 2

3 We want systems to be fault tolerant! Safety critical 3

4 We want systems to be fault tolerant! QoS should not deteriorate (too much)! 4

5 Real-Time: Time constraints have to be satisfied even in the presence of faults 5

6 Permanent faults Transient/Intermittent faults 6

7 Permanent faults Transient/Intermittent faults 7

8 Permanent faults Transient/Intermittent faults Increasing susceptibility to transient faults: Shrinking transistor size Increasing frequency Increasing temperature Decreasing supply voltage... 8

9 Permanent faults Transient/Intermittent faults Increasing susceptibility to transient faults: Shrinking transistor size Increasing frequency Increasing temperature Decreasing supply voltage... Sources of transient faults: Electromagnetic interferences Crosstalk Ground bounce Radiation by - cosmic particles - packaging material Power supply fluctuations... 9

10 The potential number of transient faults is large! It is larger than that of permanent faults! 10

11 The potential number of transient faults is large! It is larger than that of permanent faults! Specific approaches are needed in order to achieve cost efficient fault tolerant systems in the context of such an increased number of transient faults! 11

12 In this talk System-level perspective Fault tolerant systems with real-time constraints Transient processor faults Fault tolerance techniques considered software replication re-execution checkpointing 12

13 In this talk System-level perspective Fault tolerant systems with real-time constraints Transient processor faults Fault tolerance techniques considered software replication re-execution checkpointing Problem focus: System-level optimization and scheduling - time constraints are satisfied - required fault tolerance is achieved - limited amount of resources available 13

14 Outline Overall Flow, Application and System Model Fault Tolerance Policy Assignment Transparency Re-execution and Checkpointing Generation of Fault-Tolerant Schedules Cross-layer Optimization with Hardening Alternatives Conclusions 14

15 Overall Flow m 2 m1 Process graphs Periods, deadlines Max. nr. of faults Overheads 15

16 Overall Flow m 2 m1 m 2 m 1 Fault tolerance policies Transparency 16

17 Overall Flow m 2 m1 m 2 m 1 Mapping of processes and replicas 17

18 Overall Flow m 2 m1 m 2 m 1 Fault-Tolerant Conditional Process Graph 18

19 Overall Flow m 2 m1 m 2 m 1 Schedule tables N 1 true F F F F P 1 F P F 1 F P F F 1 P1 P m m

20 Overall Flow m 2 m1 m 2 m 1 N 1 true F F F F P 1 F P F 1 F P F F 1 P1 P m m Optimized - Mapping&Schedule - Fault tolerance policy assignment - Checkpointing 20

21 An application is modelled as a set of process graphs: Application Model Γ 1 Period: T Γ1 Deadline: D Γ1 Γ 2 Period: T Γ2 Deadline: D Γ2 Γ 3 Period: T Γ3 Deadline: D Γ3 21

22 An application is modelled as a set of process graphs: p 0 Application Model p 2 p 6 p 1 p 3 WCETτ 3 δτ 3 p 12 p 11 p 13 p 4 p 5 p 7 p 8 p 9 p 10 p 14 p 15 p 16 p 17 p 32 Γ 1 Period: T Γ1 Deadline: D Γ1 Γ 2 Period: T Γ2 Deadline: D Γ2 Γ 3 Period: T Γ3 Deadline: D Γ3 22

23 System Model Fault tolerant TDMA bus 23

24 System Model Fault tolerant TDMA bus Max. k transient faults can occur during one operation cycle 24

25 System Model Fault tolerant TDMA bus Max. k transient faults can occur during one operation cycle Error detection overhead Overheads: Recovery overhead Checkpointing overhead 25

26 Fault Tolerance Policies k = 2 Error detection overhead Recovery overhead Re-execution 26

27 Fault Tolerance Policies k = 2 Error detection overhead Recovery overhead Re-execution Replication 27

28 Fault Tolerance Policies k = 2 Error detection overhead Recovery overhead Re-execution Replication Re-execution&Replication 28

29 Outline Overall Flow, Application and System Model Fault Tolerance Policy Assignment Transparency Re-execution and Checkpointing Generation of Fault-Tolerant Schedules Cross-layer Optimization with Hardening Alternatives Conclusions 29

30 Fault Tolerance Policy Assignment m 2 m 1 P2 P4 k = 1 N 1 N 2 N 1 N m 1 : 10 m 2 : 10 : 10 : 5 : in WCET 30

31 Fault Tolerance Policy Assignment m 2 m 1 P2 P4 k = 1 N 1 N 2 Replication N 1 N 2 N 1 N m 1 : 10 m 2 : 10 : 10 : 5 : in WCET 31

32 Fault Tolerance Policy Assignment m 2 m 1 P2 P4 N 1 N 2 k = 1 N 1 N 2 N 1 N 2 bus m 2 N 1 N m 1 : 10 m 2 : 10 : 10 : 5 : in WCET Re-execution 32

33 Fault Tolerance Policy Assignment m 2 m 1 P2 P4 N 1 N 2 k = 1 N 1 N 2 N 1 N 2 bus m 2 N 1 N m 1 : 10 m 2 : 10 : 10 : 5 : in WCET N 1 N 2 bus m 2 Replication&Re-execution 33

34 Fault Tolerance Policy Assignment The Problem: Find a fault tolerance policy assignment such that, with the available amount of resources, it is possible to generate a schedule which in the worst case satisfies the imposed deadlines. 34

35 Transparency To what extent should fault occurrences in a certain part of the application affect the schedule of other parts? Should a fault occurrence be hidden, or should the system schedules be changed such as to optimally adapt to the new situation? 35

36 Transparency Transparency is good Easier to test, debug, verify Less complex schedules 36

37 Transparency Transparency is good It s not for free Easier to test, debug, verify Less complex schedules But Performance overhead: Longer schedules 37

38 Transparency Transparency is achieved by frozen processes/messages: their start time is the same in all alternative schedules. 38

39 Transparency Transparency is achieved by frozen processes/messages: their start time is the same in all alternative schedules. m 2 m 1 No transparency 39

40 Transparency Transparency is achieved by frozen processes/messages: their start time is the same in all alternative schedules. m 2 m 1 m 2 m 1 No transparency Full transparency 40

41 Transparency Transparency is achieved by frozen processes/messages: their start time is the same in all alternative schedules. m 2 m 1 m 2 m 1 m 2 m 1 No transparency Full transparency Customised transparency 41

42 Transparency N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 42

43 Non-Transparent Schedule N 1 Fault free scenario N 2 bus m 1 m 2 N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 43

44 Non-Transparent Schedule N 1 Fault free scenario N 2 bus m 1 m 2 N 1 Worst case scenario N 2 bus m 1 m 2 N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 44

45 Non-Transparent Schedule N 1 Fault free scenario N 2 bus m 1 m 2 N 1 Worst case scenario N 2 bus m 1 m 2 N 1 N 2 N 1 N 2 Different start times in 30 X = 5 ms different scenarios. 20 X X 20 X 30 k = 2 m 2 m 1 45

46 Fully Transparent Schedule N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 46

47 Fully Transparent Schedule N 1 Fault free scenario N 2 bus m 1 m 2 N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 47

48 Fully Transparent Schedule N 1 Fault free scenario N 2 bus m 1 m 2 N 1 Worst case scenario N 2 bus m 1 m 2 N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 48

49 Fully Transparent Schedule N 1 Fault free scenario N 2 bus m 1 m 2 N 1 Worst case scenario N 2 bus m 1 m 2 N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 Same start times = 5 ms in all scenarios. k = 2 m 2 m 1 49

50 Fully Transparent vs. Non-Transparent Schedule N 1 Non-transparent: WCS N 2 bus m 1 m 2 N 1 Fully transparent: WCS N 2 bus m 1 m 2 50

51 Customized Transparency N 1 N 2 N 1 N 2 30 X 20 X X 20 X 30 = 5 ms k = 2 m 2 m 1 51

52 Customized Transparency N 1 Customized transparency: WCS N 2 bus m 1 m 2 52

53 Customized Transparency N 1 Non-transparent: WCS N 2 bus m 1 m 2 N 1 Fully transparent: WCS N 2 bus m 1 m 2 N 1 Customized transparency: WCS N 2 bus m 1 m 2 53

54 Customized Transparency The Problem: Generate an efficient schedule with a customized degree of transparency, according to the designer s specification. For example: Isolate fault occurrences from one processor to another by freezing interprocessor messages. 54

55 Re-execution with Checkpointing Error detection overhead k = 1 Recovery overhead Checkpointing overhead Re-execution 55

56 Re-execution with Checkpointing Error detection overhead k = 1 Recovery overhead Checkpointing overhead Re-execution Re-execution with checkpointing Only this segment is re-executed 56

57 Checkpoint Optimization k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms 57

58 Checkpoint Optimization 1 P1 P1 P1 Number of checkpoints k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms 58

59 Checkpoint Optimization 1 P1 P1 P1 Number of checkpoints k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms 59

60 Checkpoint Optimization 1 P1 P1 P1 Number of checkpoints k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms 60

61 Checkpoint Optimization 1 P1 P1 P1 Number of checkpoints k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms 61

62 Checkpoint Optimization 1 P1 P1 P1 Number of checkpoints k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms

63 Checkpoint Optimization 1 P1 P1 P1 Number of checkpoints k = 2 5 ms (checkpointing) 10 ms (error detect.) 15 ms (recovery) C 1 = 50 ms

64 Checkpoint Optimization The optimal number of checkpoints for both and, if considered in isolation, is three! k = 2 m C 1 = 50 ms C 2 =60 ms 64

65 Checkpoint Optimization checkpoints k = 2 m C 1 = 50 ms C 2 =60 ms 65

66 Checkpoint Optimization checkpoints checkpoints! k = 2 m C 1 = 50 ms C 2 =60 ms 66

67 Checkpoint Optimization The Problem: Find the number of checkpoints for each process using re-execution, such that, with the available amount of resources, it is possible to generate a schedule which in the worst case satisfies the imposed deadlines. 67

68 Given: The Big Problem Application (set of process graphs) + imposed deadlines System architecture (processors, buses) Maximum number of transient faults Overheads (error detection, re-execution, checkpointing) Transparency requirements 68

69 Given: The Big Problem Application (set of process graphs) + imposed deadlines System architecture (processors, buses) Maximum number of transient faults Overheads (error detection, re-execution, checkpointing) Transparency requirements Find: Fault tolerance policy assignment for each process Number of checkpoints for processes to be re-executed Mapping of processes and of their replica Schedule 69

70 Given: The Big Problem Application (set of process graphs) + imposed deadlines System architecture (processors, buses) Maximum number of transient faults Overheads (error detection, re-execution, checkpointing) Transparency requirements Find: Fault tolerance policy assignment for each process Number of checkpoints for processes to be re-executed Mapping of processes and of their replica Schedule Such that: Deadlines are satisfied in the worst case 70

71 Given: Find: Such that: The Big Problem Application (set of process graphs) + imposed deadlines System architecture (processors, buses) Maximum number of transient faults Overheads (error detection, re-execution, checkpointing) A combination of greedy heuristics and tabu search based design space exploration. Transparency requirements Fault tolerance policy assignment for each process Number of checkpoints for processes to be re-executed Mapping of processes and of their replica Schedule Deadlines are satisfied in the worst case 71

72 Outline Overall Flow, Application and System Model Fault Tolerance Policy Assignment Transparency Re-execution and Checkpointing Generation of Fault-Tolerant Schedules Cross-layer Optimization with Hardening Alternatives Conclusions 72

73 Generation of Fault-Tolerant Schedules m 2 m 1 73

74 Generation of Fault-Tolerant Schedules m 2 m 1 k = 2 1 F P P F P4 2 F P F P1 1 m 1 2 m 1 4 P F 4 P4 4 5 F P4 4 F P4 1 1 F 1 P1 F P1 1 F P1 1 F P1 2 F 1 2 F P1 2 F 2 P S m 2 m 2 S 4 6 F 1 P2 1 2 F P2 1 F P F P2 4 m3 F P2 2 3 F P2 2 S 1 F P3 1 F P

75 Generation of Fault-Tolerant Schedules No faults Scenario F P F 1 m P1 1 F P1 2 1 F P2 1 1 m 2 m 1 F P4 1 S m 2 m 2 S m3 S 1 75

76 Generation of Fault-Tolerant Schedules Fault in and F P F 1 m P1 1 F P1 2 1 F P2 1 2 F P4 1 1 F P m 2 m 1 F P4 2 S m 2 m 2 S m3 S 1 76

77 Generation of Fault-Tolerant Schedules m 2 m 1 k = 2 1 F P P F P4 2 F P F P1 1 m 1 2 m 1 4 P F 4 P4 4 5 F P4 4 F P4 1 1 F 1 P1 F P1 1 F P1 1 F P1 2 F 1 2 F P1 2 F 2 P S m 2 m 2 S 4 6 F 1 P2 1 2 F P2 1 F P F P2 4 m3 F P2 2 3 F P2 2 S 1 F P3 1 F P

78 Schedule Table N 1 true F F F P F FP F 1 F P F F 1 P1 P m m

79 Schedule Table N 1 true F F F P F FP F 1 F P F F 1 P1 P m m

80 Schedule Table N 1 true F F F P F FP F 1 F P F F 1 P1 P m m

81 Generation of Fault-Tolerant Schedules Trade-offs Worst case schedule length vs Schedule tables size vs Transparency vs Schedule generation time 81

82 Generation of Fault-Tolerant Schedules 60 processes k=1 k=2 k=3 Max. nr. of faults Transparency 100% % 50% 25% 0% Size of schedule tables (K) Increased transparency Smaller schedule table 82

83 Generation of Fault-Tolerant Schedules 60 processes k=1 k=2 k=3 Max. nr. of faults Transparency 100% 75% 50% 25% 0% Schedule length overhead (%) Increased transparency Longer schedule 83

84 Generation of Fault-Tolerant Schedules Maintain the same ordering in all fault scenarios 84

85 Generation of Fault-Tolerant Schedules Maintain the same ordering in all fault scenarios Extremely(!) small size of schedule tables Very low schedule generation time 85

86 Generation of Fault-Tolerant Schedules Maintain the same ordering in all fault scenarios Extremely(!) small size of schedule tables Very low schedule generation time ~15% longer schedules 86

87 Outline Overall Flow, Application and System Model Fault Tolerance Policy Assignment Transparency Re-execution and Checkpointing Generation of Fault-Tolerant Schedules Cross-layer Optimization with Hardening Alternatives Conclusions 87

88 Optimization with Hardening Alternatives Where comes that mystical k from? 88

89 Optimization with Hardening Alternatives Where comes that mystical k from? γ : maximum probability of system failure in time unit (hour). Reliability goal: ρ = 1- γ 89

90 Optimization with Hardening Alternatives Where comes that mystical k from? γ : maximum probability of system failure in time unit (hour). Reliability goal: ρ = 1- γ Mapped application Rel. goal: ρ Process failure probability: p i SFP k 90

91 Optimization with Hardening Alternatives For processors with different hardening level the p i is different. Mapped application Rel. goal: ρ Process failure probability: p i SFP k 91

92 Optimization with Hardening Alternatives Reliability goal: ρ = 1 - γ = N 1 Period: T = 360 ms Period T = 360 ms 92

93 Optimization with Hardening Alternatives Hardening versions of computation node N 1 N 1 N 1 h = 1 h = 2 h = 3 t p t p t p Period T = 360 ms Cost Increase in reliability Decrease in process failure probabilities 93

94 Optimization with Hardening Alternatives Worst-case execution times are increased Hardening performance degradation (HPD) N 1 N 1 h = 1 80 h = 2 h = 3 t p t p t p Period T = 360 ms Cost Cost is increased with more hardening! 94

95 Optimization with Hardening Alternatives N 1 h = 1 80 h = 2 h = 3 t p t p t p N 1 Cost k=6 k=2 k=1 Period T = 360 ms 95

96 Optimization with Hardening Alternatives N 1 h = 1 80 h = 2 h = 3 t p t p t p N 1 Cost k=6 k=2 k=1 Period T = 360 ms 1 N 1 /1 /2 /3 /4 /5 /6 /7 2 N 1 /1 /2 /3 3 N 1 /1 /2 96

97 Optimization with Hardening Alternatives N 1 h = 1 80 h = 2 h = 3 t p t p t p N 1 Cost k=6 k=2 k=1 Period T = 360 ms 1 N 1 /1 /2 /3 /4 /5 /6 /7 2 N 1 /1 /2 /3 3 N 1 /1 /2 97

98 Optimization with Hardening Alternatives The Problem: Find the hardening level of each node, the mapping, fault tolerance policy assignment and number of checkpoints for each process, and generate a system schedule, such that the deadlines, reliability goal, and cost constraint are satisfied. 98

99 Cross-layer Fault Tolerance Optimization Application Compiler Efficient fault-tolerance needs a crosslayer approach: Operating System Instruction set All layers are involved Application specific cross-layer trade-offs Micro-architecture Component Circuit 99

100 Design optimization of fault tolerant real-time systems: Conclusions Fault Tolerance Policy assignment Transparency Checkpoint optimization Schedule generation Optimization with hardening alternatives (cross-layer); 100

101 Design optimization of fault tolerant real-time systems: Conclusions Fault Tolerance Policy assignment Transparency Checkpoint optimization Schedule generation Optimization with hardening alternatives Efficient system-level design techniques can help to meet real-time and fault tolerance requirements in the context of limited amount of resources. 101

102 Fault Tolerance Policy Assignment Average deviation of schedule length (%) Mapping and policy assignment Number of processes 102

103 Fault Tolerance Policy Assignment Average deviation of schedule length (%) Mapping and re-execution Mapping and policy assignment Number of processes 103

104 Fault Tolerance Policy Assignment Average deviation of schedule length (%) Mapping and replication Mapping and re-execution Mapping and policy assignment Number of processes 104

105 Checkpoint Optimization Average deviation of schedule length (%) 40% 30% 20% 10% 0% Local Optimization of Checkpoints Number of processes 105

106 Checkpoint Optimization Average deviation of schedule length (%) 40% 30% 20% 10% 0% Global Optimization of Checkpoints Local Optimization of Checkpoints Number of processes 106

107 Design optimization of fault tolerant real-time systems. Conclusions Fault Tolerance Policy assignment Transparency Checkpoint optimization Schedule generation 107

108 Design optimization of fault tolerant real-time systems. Conclusions Fault Tolerance Policy assignment Transparency Checkpoint optimization Schedule generation Efficient system-level design techniques can help to meet real-time and fault tolerance requirements in the context of limited amount of resources. 108