PostgresPro s contribu on to PostgreSQL s performance

Transcription

1 PostgresPro s contribu on to PostgreSQL s performance Alexander Korotkov Postgres Professional 2017 Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 1 / 25

2 Pin/UnpinBuffer op miza on (1/2) Spinlock is implemented in assembly. When no implementa on for par cular processor, then using UNIX semaphore. Since 9.5 PostgreSQL supports atomic instruc ons. Compiler buil ns are used or assembly implementa ons. If none of them are present, then spinlocks are used. Since 9.5 LWLocks are implemented using atomic state variable. Before 9.5 spinlock was used. Since 9.6 Pin/UnpinBuffer are implemented using atomic state variable. Before 9.6 spinlock was used. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 2 / 25

3 Pin/UnpinBuffer op miza on (2/2) Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 3 / 25

4 From GIN to RUM GIN structure RUM introduces addi onal informa on to pos ng lists. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 4 / 25

5 How did things begin in 2012? Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 5 / 25

6 RUM use-cases Fulltext search with ranking in index (offsets of lexemes inside document as addinfo) Fulltext search with alterna ve ordering (custom column as addinfo) Jsonb indexing with posi onal informa on (offsets of elements in array as addinfo) Posi onal n-grams (vgrams) for be er fuzzy string matching Inversed fulltext search (find queries matching given document) Inversed regex search (find regexes matching given string) Array similarity search using array lengths from the index Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 6 / 25

7 GIN: ranking form heap Search for top-10 (from ) pos ngs with «Tom Lane». subject, ts_rank(fts,plainto_tsquery( english, tom lane )) rank pglist fts plainto_tsquery( english, tom lane ) rank 10; QUERY PLAN (actual time= =10 loops=1) -> Sort (actual time= =10 loops=1) Sort : (ts_rank(fts, tom & lane ::tsquery)) Sort Method: top-n heapsort Memory: 25kB -> Bitmap Heap Scan pglist (actual time= =22 Cond: tom & lane ::tsquery) Heap Blocks: exact= > Bitmap Scan pglist_gin_idx (actual time= = loops=1) Cond: tom & lane ::tsquery) Planning time: ms Execution time: (11 ) Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 7 / 25

8 RUM: ranking form index Search for top-10 (from ) pos ngs with «Tom Lane». pglist_rum_fts_idx pglist rum(fts rum_tsvector_ops); subject pglist fts plainto_tsquery( tom lane ) fts <=> plainto_tsquery( tom lane ) 10; QUERY PLAN (actual time= =10 loops=1) -> Scan pglist_rum_fts_idx pglist (actual time= =10 loops=1) Cond: plainto_tsquery( tom lane ::text)) : (fts <=> plainto_tsquery( tom lane ::text)) Planning time: ms Execution time: (6 ) RUM accelera on. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 8 / 25

9 Inverse FTS (FQS) Find queries, which match given document Automa c text classifica on * queries; q tag supernova & star sn black color big & bang & black & hole bang spiral & galaxi shape black & hole color (5 ) * queries to_tsvector( black holes never exists before we think about them q; q tag black color black & hole color (2 ) Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 9 / 25

10 RUM index for inverse FTS term1: (AND 1, OR 2) term2: (AND 1, OR 2) term3: (AND 1) Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 10 / 25

11 RUM index for inverse FTS RUM index supported store branches of query tree in addinfo. Find queries for the first message in postgres mailing lists. pg_query_rum_idx pg_query rum(q); q pg_query pgq, pglist pglist.fts pglist.id=1; QUERY PLAN Nested Loop (actual time= =2 loops=1) -> Scan pglist_id_idx pglist (actual time= =1 loops=1) Cond: (id = 1) -> Bitmap Heap Scan pg_query pgq (actual time= =2 loops=1) Cond: pglist.fts) Heap Blocks: exact=2 -> Bitmap Scan pg_query_rum_idx (actual time= =2 loops=1) Cond: pglist.fts) Planning time: ms Execution time: ms (10 ) Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 11 / 25

12 RUM status Extension for PostgreSQL 9.6+ (thanks to CREATE ACCESS METHOD & generic WAL) Available at Available at PGDG yum & apt repositories. Quite stable, passed test of me. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 12 / 25

13 Bitmap index-only count # (, buffers) count(*) dblp_titles to_tsvector( english, to_tsquery( english, system ); ( = =1 width=8) (actual time= =1 loops=1) Buffers: -> Bitmap Heap Scan dblp_titles ( = =12634 width=0) (actual time= Cond: (to_tsvector( english ::regconfig, system ::tsquery) Heap Blocks: exact=20139 Buffers: shared hit= > Bitmap Scan dblp_titles_fts_idx ( = =12634 width=0) (actual time= Cond: (to_tsvector( english ::regconfig, system ::tsquery) Buffers: shared hit=2 =28 Planning time: ms Execution time: # (, buffers) count(*) dblp_titles to_tsvector( english, to_tsquery( english, system ); ( = =1 width=8) (actual time= =1 loops=1) Buffers: -> Bitmap Heap Scan dblp_titles ( = = width=0) (actual time= Cond: (to_tsvector( english ::regconfig, system ::tsquery) Heap Blocks: exact=20139 Buffers: shared hit=31 -> Bitmap Scan dblp_titles_fts_idx ( = = width=0) (actual time= Cond: (to_tsvector( english ::regconfig, system ::tsquery) Buffers: shared hit=30 Planning time: ms Execution time: Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 13 / 25

14 vops parquet layout Query Seq. exec., msec Parallel exec. (msec) Original Q1 for lineitem Original Q1 for lineitem_projec on Vectorized Q1 for vops_lineitem Mixed Q1 for vops_lineitem_projec on Original Q6 for lineitem Original Q6 for lineitem_projec on Vectorized Q6 for vops_lineitem Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 14 / 25

15 Pluggable storage engines + in-memory engine pgbench -s j $n -c $n -M prepared on 4 x 18 cores Intel Xeon E processors mean of 3 3-minute runs with shared_buffers = 32GB, max_connections = builtin builtin-func in_memory in_memory-func TPS # Clients Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 15 / 25

16 SP-GiST for boxes mapping to 4D space Parameter heap GiST SP-GiST Build me, sec Size, MB &&, ms ms ms Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 16 / 25

17 Incremental backup Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 17 / 25

18 pg_probackup Available at Supports incremental backup with Postgres Pro Enterprise or patched PostgreSQL. Various useful features. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 18 / 25

19 Index-only scans for GiST $ id, some_point, some_point <-> point(500,500) test some_point <-> point(500,500) 10; ( = =10 width=20) (actual time= =10 -> Scan tst_idx test ( = = widt : (some_point <-> (500,500) ::point) Planning time: ms Execution time: $ id, some_point, some_point <-> point(500,500) test some_point <-> point(500,500) 10; ( = =10 width=20) (actual time= =10 -> Scan tst_idx test ( = = : (some_point <-> (500,500) ::point) Heap Fetches: 0 Planning time: ms Execution time: Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 19 / 25

20 KNN-GiST with recheck PostGIS users can have results sorted by true distance using <-> operator. No need for hackery approximate queries anymore. closest_candidates ( streets.gid, streets., streets.geom nyc_streets streets streets.geom <-> SRID=26918;POINT( ) ::geometry 100) gid, closest_candidates ST_Distance(geom, SRID=26918;POINT( ) ::geometry ) 1; streets.gid, streets., streets.geom nyc_streets streets streets.geom <-> SRID=26918;POINT( ) ::geometry 1; Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 20 / 25

21 Adap ve query op miza on Let database learn on its own mistakes! Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 21 / 25

22 Page-level table compression pgbench -i -s 1000 Configura on Size (Gb) Time (sec) no compression snappy lz postgres internal lz lzfse zlib (best speed) zlib (default level) zstd Available in Postgres Pro Enterprise. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 22 / 25

23 Covering + unique indexes olduniqueidx oldt btree (c1, c2); oldcoveringidx oldt btree (c1, c2, c3, c4); newidx newt btree (c1, c2) (c3, c4); One index instead or two! Available in Postgres Pro, patch is submi ed. Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 23 / 25

24 Other improvements 2PC speedup precalculate stable func ons contrib/bloom Use pg_rewind when old master meline was switched Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 24 / 25

25 Thank you for a en on! Alexander Korotkov PostgresPro s contribu on to PostgreSQL s performance 25 / 25