Prometheus at Scale. Bartek Płotka. github.com/improbable-eng/thanos. Edinburgh, 22th October

Size: px

Start display at page:

Download "Prometheus at Scale. Bartek Płotka. github.com/improbable-eng/thanos. Edinburgh, 22th October"

Clare Fox
5 years ago
Views:

1 at Scale Bartek Płotka github.com/improbable-eng/thanos Edinburgh, 22th October 2018

2 Bartek Płotka Software Engineer

Founded: 2012 "Improbable s platform, SpatialOS, is designed to

imagine Minecraft with thousands of players in the same space or

3 Founded: 2012 "Improbable s platform, SpatialOS, is designed to let anyone build massive simulations, running in the cloud: imagine Minecraft with thousands of players in the same space or researchers creating simulated cities to model the behaviour of millions. Its ultimate goal: to create totally immersive, persistent virtual worlds." Employees: WIRED, May 2017 Games in Development: +19

4 Agenda

5 Agenda

6 Agenda Thanos

7 Agenda + Thanos

8 Agenda + Thanos Thanos + Open Source

10 Alertmanager Grafana HTTP Query API Rule & Alert Engine Query Engine every 15s Compactor Scrape Engine Local storage /metrics Service X Service X Services

11 Single Cluster Grafana Workloads you want Alertmanager to monitor

12 Single Cluster Grafana Workloads you want Alertmanager to monitor samples / sec = 1 CPU, 13 GB RAM = 3 mln series with 15s scrape interval.

13 High Availability Cluster Grafana Workloads you want Alertmanager to monitor

14 Globally distributed clusters Cluster 1 Cluster n Cluster 2 Cluster n+1...

15 Problem: Global View sum(rate(go_memstats_alloc_bytes_total[1m]))? source = a source = b

16 Problem: Global View + HA go_memstats_alloc_bytes_total{...} [deduplicated] go_memstats_alloc_bytes_total{..., replica=a) go_memstats_alloc_bytes_total{..., replica=b)

17 Problem: Metric retention

18 Problem: Metric retention SSD Remote write

19 Thanos Goals Have a global view Have a HA in place Increase retention

20 Global View See everything from a single place!

21 Targets SSD

22 Sidecar grpc (Store API) Targets SSD Sidecar

23 Store API Store API message SeriesRequest { Sidecar int64 min_time = 1; int64 max_time = 2; repeated LabelMatcher matchers = 3; remote read } service Store { rpc Series(SeriesRequest) returns (stream SeriesResponse); rpc LabelNames(LabelNamesRequest) returns (LabelNamesResponse); rpc LabelValues(LabelValuesRequest) returns (LabelValuesResponse); }

24 Querier HTTP Query API Querier Store API Targets SSD Sidecar

25 Global View Querier Merge Store API Targets SSD Sidecar Sidecar SSD Targets

26 Global View + Availability Querier Deduplicate Merge Store API Targets SSD Sidecar replica : A Sidecar Sidecar SSD SSD replica : B Targets

27 Thanos Goals Have a HA in place Have a global view Querier Sidecar Sidecar Sidecar SSD

28 Global View + Availability

29 Global View + Availability

30 Historical Metrics What exactly happened X months ago?

31 Rule & Alert Engine Query Engine Compactor Scrape Engine TSDB

32 TSDB Layout T-10h T-16h Block 1 T-4h chunks chunks chunks chunks index T T-2h Block 3 Block 4

33 Data saving --gcs.bucket=... Targets SSD Object Storage Sidecar Block Blocks Blocks

34 Data saving --storage.tsdb.max-block-duration=2h --storage.tsdb.retention=12h Targets SSD Object Storage Sidecar Block Blocks Blocks

35 Store Gateway Querier Store API Object Storage Store Gateway Blocks Cache

36 Ruler Querier Query Store API Ruler SSD Blocks Object Storage Blocks

37 Compaction Density matters

38 Compaction Object Storage Compactor Block Disk Blocks Block Blocks

39 Downsampling

40 Downsampling Raw: 16 bytes/sample Compressed: 1.07 bytes/sample

41 Downsampling Decompressing one sample takes nanoseconds.

42 Downsampling Decompressing one sample takes nanoseconds. Decompressing s scrape interval for 1 year data takes seconds alone. Plus your actual computation over all those samples, e.g. rate()

43 Downsampling Block RAW 10x 5m 12x 1h

44 Thanos Goals Have a HA in place Increase retention Have a global view

45 Rule & Alert Engine Query Engine Compactor Scrape Engine

46 Thanos Thanos Querier Thanos Querier Thanos Querier Rule & Alert Engine Compactor Scrape Engine

47 Thanos Thanos Querier Thanos Querier Thanos Querier Rule & Alert Engine Compactor Sidecar Sidecar Sidecar SSD SSD SSD

48 Thanos Thanos Ruler Thanos Ruler Compactor Thanos Querier Thanos Querier Thanos Querier Sidecar Sidecar Sidecar SSD SSD SSD

49 Thanos Thanos Ruler Thanos Ruler Global Compactor Thanos Querier Thanos Querier Thanos Querier Sidecar Sidecar Sidecar SSD SSD SSD

50 Thanos Thanos Ruler Thanos Ruler Global Compactor Thanos Querier Thanos Querier Thanos Querier Sidecar Sidecar Sidecar SSD SSD SSD Store Store Gateway Gateway Object Storage

51 Example Deployment Monitoring Cluster Querier Querier Querier Grafana Statically configured / global discovery Alertmanager + Compactor Store + Ruler Cluster n Cluster 1 Cluster 2 Bucket + Cluster n

52 Open Source & Thanos

53 Open Source: It is worth it!

54 Open source from the very beginning

55 Be flexible while staying focused

56 Stay focused: Injection API Source:

57 Less magic == better

58 Avoid magic: Gossip example Easy to misconfigure Hard to debug Difficult cross cluster setup Over-complicated for Thanos needs

59 Thanos Goals Have a global view Have a HA in place Increase retention Join effort with community

60 Summary Start small with just single. Extend your setup gradually. Just global querier. Federated global querier. Object storage. Model your Thanos deployment. Generally: Share early, keep project focused and simple.

61 ...psst, join our slack workspace for more info! github.com/improbable-eng/thanos improbable.io Credits: Percona blog post about 2 perf Emojis designed by Freepik from Flaticon Bartek Płotka

Alerting with Time Series

Alerting with Time Series Fabian Reinartz, CoreOS github.com/fabxc @fabxc Time Series Stream of pairs associated with an identifier http_requests_total{job="nginx",instance="1.2.3.4:80",path="/status",status="200"}