Optimizing VM Checkpointing for Restore Performance in VMware ESXi Server

Transcription

1 Optimizing VM Checkpointing for Restore Performance in VMware ESXi Server Irene Zhang University of Washington Tyler Denniston MIT CSAIL Yury Baskakov VMware Alex Garthwaite CloudPhysics

2 Virtual Machine Checkpointing Provides snapshot of a running virtual machine. Taking a checkpoint is fast, but restoring a checkpoint is slow. Traditionally used to support fault tolerance applications.

3 Existing VM checkpointing systems cannot support applications that require fast restore. Dynamic VM allocation Energy conservation Virtual desktop infrastructure

4 Halite New checkpointing system for VMware ESXi Server that reduces restore time to seconds.

5 VM Checkpoint Restore VM VM t

6 VM Checkpoint Restore VM VM t

7 VM Checkpoint Restore VM VM t

8 VM Checkpoint Restore VM VM t

9 VM Checkpoint Restore VM VM t

10 VM Checkpoint Restore VM VM t Fast for small VMs less than 1GB, but does not scale to bigger VMs.

11 ESXi Checkpoint Restore VM VM t

12 ESXi Checkpoint Restore VM VM t

13 ESXi Checkpoint Restore VM VM t

14 ESXi Checkpoint Restore VM VM t

15 ESXi Checkpoint Restore VM VM t Starts the VM quickly, but the VM experiences too much performance degradation.

16 Faults to disk during checkpoint restore are inefficient. Checkpointed memory is organized in physical address order. Each fault is random 4KB disk read. No spatial locality, so bigger reads do not help.

17 Working Set Restore [Zhang, et al., VEE 2011] Reduces faults to disk by prefetching pages the VM may access. Found some VM workloads are too complicated to predict. Many sources of divergence: timing variations, background processes and external/user inputs.

18 Halite predicts access locality. Predict pages that are likely to be accessed together and store them together. Sufficient to provide spatial locality in the checkpointing file, enabling bigger reads. Easier to predict and more resilient to divergence.

19 Locality Blocks Block of checkpointed VM memory pages likely to be accessed together. Locality Block

20 Halite Checkpoint Restore VM VM t Locality Block

21 Halite Checkpoint Restore VM VM t Locality Block

22 Halite Checkpoint Restore VM VM t

23 Halite Checkpoint Restore VM VM t

24 Halite Checkpoint Restore VM VM t

25 Halite Checkpoint Restore VM VM t Locality blocks enable more efficient disk access during checkpoint restore.

26 Access Locality Prediction Two techniques: checkpoint save access tracking and virtual address locality. Done entirely during checkpoint save. Directly used to save memory pages into locality blocks.

27 ESXi Checkpoint Save VM VM t

28 ESXi Checkpoint Save t VM VM

29 ESXi Checkpoint Save Checkpoint t VM VM

30 ESXi Checkpoint Save Checkpoint VM VM t

31 ESXi Checkpoint Save Checkpoint t VM VM

32 ESXi Checkpoint Save Checkpoint t VM VM

33 ESXi Checkpoint Save Checkpoint VM VM t

34 ESXi Checkpoint Save Checkpoint VM VM Pages accessed together during t checkpoint save are likely to be accessed together again during checkpoint restore.

35 Halite Checkpoint Save VM VM t

36 Halite Checkpoint Save Checkpoint t VM VM

37 Halite Checkpoint Save Checkpoint VM VM t

38 Halite Checkpoint Save Checkpoint t VM VM

39 Halite Checkpoint Save Checkpoint t VM VM Locality Block

40 Halite Checkpoint Save Checkpoint t VM VM Locality Block

41 Virtual Address Space Locality Used for pages the VM does not touch during checkpoint save. Pages mapped together in the guest virtual address space are likely to be accessed together. Background thread saves pages into locality blocks in virtual address order.

42 Other optimizations Compression Threading Background thread throttling Zero page optimization

43 Other optimizations Compression Threading Background thread throttling Zero page optimization Please refer to the paper.

44 Evaluation Does Halite improve restore performance? How much do locality blocks improve performance? Could we use blocks without access locality? Does Halite affect checkpoint save performance?

45 Hardware 2.3 GHz 8-core AMD Opteron 24 GB 15K RPM Seagate ITB (VMFS-5)

46 Workloads sim Custom memory access simulator pgbench Built-in PostgreSQL benchmark worldbench Windows desktop benchmark apache webserver HTTP get benchmark

47 Workloads sim Custom memory access simulator pgbench Built-in PostgreSQL benchmark worldbench Windows desktop benchmark apache webserver HTTP get benchmark

48 VM Configuration OS sim 4 vcpu 2 GB pgbench 4 vcpu 2 GB worldbench 2 vcpu 1 GB apache webserver 2 vcpu 2 GB

49 Worldbench Test Setup Worldbench Test Run seconds

50 Checkpoint Worldbench Test Setup Worldbench Test Run seconds

51 Checkpoint Worldbench Test Setup Worldbench Test Run seconds ESXi Checkpoint Restore 841.3

52 Checkpoint Worldbench Test Setup Worldbench Test Run seconds ESXi Checkpoint Restore Halite Checkpoint Restore 817.8

53 Baseline Checkpoint Worldbench Test Setup Worldbench Test Run seconds ESXi Checkpoint Restore Halite Checkpoint Restore 817.8

54 Halite reduces restore overhead from tens of seconds to seconds. ESXi 25.2 seconds Halite 1.6 seconds Restore Slowdown (seconds)

55 Locality blocks provide more than half of Halite s performance improvement. ESXi 25.2 seconds Locality Blocks w/o Compression 10.7 seconds Halite 1.6 seconds Restore Slowdown (seconds)

56 Locality blocks provide more than half of Halite s performance improvement. ESXi 25.2 seconds Locality Blocks w/o Compression 10.7 seconds Halite 1.6 seconds Restore Slowdown (seconds)

57 Locality blocks provide more than half of Halite s performance improvement. ESXi 25.2 seconds Locality Blocks w/o Compression 10.7 seconds Halite 1.6 seconds Restore Slowdown (seconds)

58 Blocks with no locality hurt performance. ESXi 25.2 Physical Blocks 56.1 Locality Blocks 10.1 Halite Restore Slowdown (seconds)

59 Blocks with no locality hurt performance. ESXi 25.2 Physical Blocks 56.1 Locality Blocks 10.1 Halite Restore Slowdown (seconds)

60 Blocks with no locality hurt performance. ESXi 25.2 Physical Blocks 56.1 Locality Blocks 10.1 Halite Restore Slowdown (seconds)

61 Worldbench Checkpoint Test Setup Worldbench Test Run seconds

62 Worldbench Checkpoint Test Setup Worldbench Test Run Checkpoint seconds ESXi Checkpoint Save 827.2

63 Worldbench Checkpoint Test Setup Worldbench Test Run Checkpoint seconds ESXi Checkpoint Save Checkpoint Halite Checkpoint Save 821.4

64 Baseline Worldbench Checkpoint Test Setup Worldbench Test Run Checkpoint seconds ESXi Checkpoint Save Checkpoint Halite Checkpoint Save 821.4

65 For Worldbench, Halite improves checkpoint save performance. ESXi 11.0 Halite Save Slowdown (seconds)

66 Halite Fast checkpoint restore is important for a new class of applications. Predicting access locality is better than predicting accesses for complex VM workloads. Halite can restore a checkpointed Windows VM in 1.6 seconds.