glideinwms Training HTCondor Overview by Igor Sfiligoi, UC San Diego Aug 2014 HTCondor Overview 1

Size: px

Start display at page:

Download "glideinwms Training HTCondor Overview by Igor Sfiligoi, UC San Diego Aug 2014 HTCondor Overview 1"

Elwin Owen
5 years ago
Views:

1 glideinwms Training HTCondor Overview by Igor Sfiligoi, UC San Diego Aug 2014 HTCondor Overview 1

2 Overview These slides present a HTCondor overview, with high level views of Deamons involved Communication paths Scalability considerations Aug 2014 HTCondor Overview 2

3 HTCondor Daemons The basics Negotiator Submit side Glue Execute side Aug 2014 HTCondor Overview 3

4 HTCondor Daemons The basics + the master Master Negotiator Master Master Submit side Glue Execute side Aug 2014 HTCondor Overview 4

5 HTCondor Daemons One startd per (logical) compute resource Can handle multiple CPUs One schedd per submit node Can handle multiple users has the list of all other daemons Negotiator matches user jobs to machine slots Master starts all other processes Will ignore it in the rest of the talk Aug 2014 HTCondor Overview 5

6 Communication flow Push: I am (still) here Negotiator Pull: Send me the list of idle jobs Push: I am here and these are my properties One ClassAd x slot Aug 2014 HTCondor Overview 6

7 Claiming protocol s keep their state current in the By periodically pushing updates (every 5 mins by default) On a matchmaking cycle, the Negotiator will Pull the startd slot list from the collector In Unclaimed state only (unless preemption enabled) Pull the job list from the schedds Create a priority list of matches Send the matches to relevant schedds Aug 2014 HTCondor Overview 7

8 Claiming protocol The schedd will contact the startd Once the connection is accepted, the schedd owns that slots The schedd will spawn a shadow Which takes over the connection The schedd moves on to other business Similarly, the startd spawns a starter And advertise a Claimed state Aug 2014 HTCondor Overview 8

9 Communication flow Negotiator Claimed/Idle Aug 2014 HTCondor Overview 9

10 HTCondor Daemons Stage 2 Negotiator Claimed/Busy Shadow Starter A shadow and a starter are created for every running job Aug 2014 HTCondor Overview 10

11 HTCondor Daemons Stage 2 Negotiator Claimed/Busy Shadow Starter If the network connection is lost, either side can re-establish it. Aug 2014 HTCondor Overview 11

12 HTCondor Daemons The shadow takes ownership of a running job One per job The starter takes ownership of a claimed slot One per slot Together they babysit the two sides until the jobs is done and the slot can be un-claimed Corollary: Each schedd node will have O(10k) shadows Aug 2014 HTCondor Overview 12

13 Claiming protocol Once the job terminates The starter goes away The schedd will send another job to the startd, unless The lease has expired There are no more suitable jobs The existing shadow can be reused But does not need to If the schedd does not send a new job goes into UnClaimed state Aug 2014 HTCondor Overview 13

14 Matchmaking and latency The Negotiator pulls the startd slot list from the In a single transaction The Negotiator pulls the job list from the schedds Basically, one at a time! But it does cluster similar jobs together at level The idea being that it will not ask for more, if either the user runs out of priority or no more slots are available Negotiator thus sensitive to Network latencies Matching s far away may be limited by network latency not Negotiator CPU use Aug 2014 HTCondor Overview 14

15 Security considerations The glidein use case center of all trust Mutual authentication between and /Negotiator using x509 whitelisting Negotiator and co-located, FS auth Negotiator Mutual authentication between and using x509 whitelisting Aug 2014 HTCondor Overview 15

16 Security considerations The glidein use case center of all trust After initial handshake, use shared secret Negotiator and co-located, FS auth Negotiator After initial handshake, use shared secret Full x509 expensive, used only on daemon restart and periodically once every few days Aug 2014 HTCondor Overview 16

17 Security considerations The glidein use case center of all trust also sends shared secret for matchmaking Aug 2014 HTCondor Overview 17

18 Security considerations The glidein use case center of all trust shared secret sent on job match Negotiator Negotiator only authorized user of 's shared secrets may get many secrets, one per matched job Aug 2014 HTCondor Overview 18

19 Security considerations The glidein use case center of all trust Use given shared secret for auth No other credentials in play Aug 2014 HTCondor Overview 19

20 Security considerations The glidein use case center of all trust Shadow Shadow and starter inherit the socket Also inherit shared secret, for reconnect Starter Aug 2014 HTCondor Overview 20

21 Security considerations The glidein use case center of all trust If goes in Unclaimed state, a new secret is created and sent Aug 2014 HTCondor Overview 21

22 Security cost and scalability The glidein use case x509 too expensive for a single central service (both due to CPU use, and network latency issues) Glideins can start at 10+Hz rate Mutual authentication between and /Negotiator using x509 whitelisting Mutual authentication between and using x509 whitelisting Aug 2014 HTCondor Overview 22

23 Security cost and scalability The glidein use case Spread the load over multiple child s Child collectors forward all ads Randomly pick one of many child s and then stick with it New s join rarely... Co-located, thus cheap x N Mutual authentication between and child using x509 whitelisting Aug 2014 HTCondor Overview 23

24 Network/Firewall considerations HTCondor is conceptually a Peer-to-Peer system Everyone talks to everyone Negotiator Shadow Starter Aug 2014 HTCondor Overview 24

25 Network/Firewall considerations Execute nodes often behind firewalls and/or NATs Shadow Starter Aug 2014 HTCondor Overview 25

26 Network/Firewall considerations CCB protocol creates a tunnel implements the CCB A separate channel over long lived TCP socket -> communication still direct Aug 2014 HTCondor Overview 26

27 Network/Firewall considerations CCB protocol creates a tunnel implements the CCB CCB delivers messages to the startd Aug 2014 HTCondor Overview 27

28 Network/Firewall considerations CCB protocol creates a tunnel implements the CCB Only callback request goes through CCB opens a long lived TCP connection to the All further communication on that channel from there on Aug 2014 HTCondor Overview 28

29 Network/Firewall considerations CCB protocol creates a tunnel implements the CCB Shadow Shadow and Starter inherit this socket Starter Aug 2014 HTCondor Overview 29

30 CCB and scalability A single central service cannot really handle all the load Processes usually limited to O(1k) sockets We can have O(10k+) glideins Aug 2014 HTCondor Overview 30

31 CCB and scalability No real need to use a single CCB, could use any number of dedicated CCB s... x N Randomly pick one of many CCBs and then stick with it The standard strategy is to just piggy-back on the Child paradigm Aug 2014 HTCondor Overview 31

32 CCB and scalability The now needs to accept incoming connections Default HTCondor mechanism of one port x connection does not scale (only ~30k usable ports in IP) Only callback request goes through CCB opens a long lived TCP connection to the Aug 2014 HTCondor Overview 32

33 CCB and scalability HTCondor added shared_port_daemon to multiplex requests on a single port Only callback request goes through CCB Shared_Port_Daemon opens a long lived TCP connection to the shared_port_daemon Specifying that it is for the schedd Aug 2014 HTCondor Overview 33

34 CCB and scalability HTCondor added shared_port_daemon to multiplex requests on a single port Shared_Port_Daemon Same node, local UNIX command Socket is moved to the schedd Aug 2014 HTCondor Overview 34

35 CCB and scalability Can be used by starter to contact the Shadow, too Shared_Port_Daemon Same node, local UNIX command Shadow Socket is moved to the schedd Starter Aug 2014 HTCondor Overview 35

36 CCB and scalability Starter also accepts incoming connections Thus needs a CCB connection Plus there is normally one for the Master, too Starter Starter And there is one Starter per slot Aug 2014 HTCondor Overview 36 x N

37 CCB and scalability Adding a shared_port_daemon will cut number of CCB connections to exactly one Shared_Port_Daemon Route incoming requests to appropriate daemon Starter Starter Aug 2014 HTCondor Overview 37 x N

38 High Availability setup Using a single CM node risky; if it dies, the pool dies with it. Central Manager Node Negotiator... x N Having multiple child collectors does not help Aug 2014 HTCondor Overview 38

39 High Availability setup HTCondor allows for 2 or more CM nodes Central Manager Node Negotiator Central Manager Node Negotiator... x N... x N s and s talk to all of them Including one CCB per CM node Aug 2014 HTCondor Overview 39

40 High Availability setup There can be only one active Negotiator, to make user priority decision Central Manager Node Negotiator Central Manager Node Negotiator HAD HAD... x N... x N HAD daemons maintain only one alive with others in standby mode Aug 2014 HTCondor Overview 40

41 High Availability setup HA typically just partition the jobs between many schedds Temporary downtimes result in other schedds taking over the slots Central Manager Node Negotiator Central Manager Node Negotiator HAD HAD True HA possible, but requires shared FS... x N... x N No real need for HA... x M HAD daemons maintain only one alive with others in standby mode Aug 2014 HTCondor Overview 41

42 The end Aug 2014 HTCondor Overview 42

AGENTLESS ARCHITECTURE

ansible.com +1 919.667.9958 WHITEPAPER THE BENEFITS OF AGENTLESS ARCHITECTURE A management tool should not impose additional demands on one s environment in fact, one should have to think about it as little