PodCTL #4 - All the Tools in the Kubernetes Toolbox

Transcription

1 PodCTL #4 - All the Tools in the Kubernetes Toolbox Tyler: [00:00:13] Hello Kubernetes community and welcome back to the PodCTL podcast. It's another week down in the Kubernetes community. Not a lot of news this week but we have a couple of interesting things that came up as well as we're going to talk about some of the things that we see really commonly in open source projects and specifically in Kubernetes which is installers. Why is there not just one? Why is they always end up being a lot of them? Brian: [00:00:38] Right exactly. So we're going to talk about the big Kubernetes toolbox, at least part of the toolbox, like three or four shelves worth of it and get into that. The other thing, thanks to everybody who's been listening to the show has been several times in the Top 100 of the tech podcast on itunes so folks who are listening, telling a friend, if you can give us some ratings we'd love some feedback and all the details of how to reach on Twitter and the and the blog and all that stuff are in the show notes. So thanks for listening. Everybody calling you on what's been in the news like you said it was a little bit of a slow news week in terms of Q&A specifically but there was a little bit of news. Tyler: [00:01:10] Yeah. This week VMworld, their twice annual conference and you usually get some sort of mix of their usual virtualization news and announcements and what we consider modern app dev or whatever you want to call it announcements. And this year it wasn't any different it was announcement by VMware with Google and Pivotal. They did take a package version of Pivotal's Kubo project which is a Kubernetes deployer and packed up as a commercial offering called the Pivotal Container Service (PKS). Brian: [00:01:39] So just the basics of this and we'll put the link to it in the show notes for folks

2 that want to dig in. It was a project that Google and Pivotal had kicked off called "Kubo", which like you mentioned takes the BOSH installer from Cloud Foundry foundation and installs a Kubernetes instance. So we're actually going to get into lots of ways to install Kubernetes. But yes so it was the project called Kubo and then it became commercialized between Google Cloud, Pivotal and VMware. They created a packaged offering called the Pivotal Container Service, with the acronym being PKS. Brian: [00:02:14] Sort of like Google container services G.K.E. where the container is like Kubernetes. In the initial announcement they kind of laid out what they were thinking with it so it will be interesting to see as we go forward, when it actually gets close to shipping (targeting CYQ42017) and kind of what it looks like as it gets closer to that point in time. Brian: [00:02:31] Right. Because I think they said this wasn't going to ship until at least the last quarter of the this calendar year which could be anywhere from October to December 31st or something like that. We'll have to wait and see and I don't know if there's any bits available since it's now a commercial offering and VMware doesn't really do open source per se or at least that's not their business model so have to see how it will be available. Tyler: [00:02:52] Yeah I had noticed that the Kubo project had used to rely on Cloud Foundry and Pivotal Cloud Foundry integration and then they had an updated version. so I wonder why they're doing that for them. And then you saw this announcement of Kubo pieces that you can play around with but who knows about the rest of the things like you said VMware they don't always make it out on GitHub. Brian: [00:03:14] Right. The other thing that came out during the show, they announced on stage during VMworld during the keynote that both VMWare and Pivotal were joining the CNCF at like a gold or platinum member level. I think from an industry perspective it is good it says that the folks from the CNCF are basically saying hey enterprise computing is now going to flow through through the CNCF And you know I think when you when you put Kubernetes is front and center of that it's a fair statement now you've got every major cloud provider is supporting Kubernetes: Microsoft, Google, AWS, Oracle, Red Hat, VMware where you know all the major players in terms of enterprise technology are now supporting Kubernetes. So we never like to say that any one technology has "won" but it's a pretty pretty strong statement from the CNCF this week. Tyler: [00:04:03] You never know. Things change but at this point in time Kubernetes is has seemed to a pretty strong consensus has developed around it. Brian: [00:04:12] So speaking of that consensus once you start getting a lot of people using any given technology spawn's itself into lots of ways to use it. Lots of ways to install it, operate it, maintain it, upgrade it. So today we thought we would talk about all the tools that are out there which it's not unusual especially for open source projects to have lots of tools. But let's start with the first basic question like: Why are there so many tools to deal with Kubernetes? Tyler: [00:04:38] Well I mean I think it's a cultural thing that's just comes with open source software is the the idea of there's something this tool doesn't do what I want so I'm going to make something that does what I want. And because I like open source and all those things, I'm going to share it with other people in case they want it to use it. That's the upside of open source. The downside is you can usually get a pretty big growth in those areas of people taking tools and splitting them a lot different ways and it makes sense from the standpoint of people have different requirements what they're looking to do and this tool focuses more. here I need an installer that can do this, that and the other thing but I don't care about this piece. There's not an official Kubernetes installer, to submit a patch. Yeah know a pull request to with your changes to it change out of work.

3 So it's just all just either you do it myself or start and create one from scratch. So we saw that with OpenStack we've seen with plenty of other open source technologies too, where there just installers everywhere. Brian: [00:05:37] Well I think it spawns out of this open source concept. I don't want to get too philosophical and religious but I mean you know when everything is proprietary you bought software from somebody and that vendor typically gave you the installer and the package managers and all that stuff and with open source you know you have plenty of companies that are spawned up as consulting companies, or training companies, as well as commercial vendors of open source, or public cloud providers of open source, so I think a lot of times you will find that popular tool gets spawned up by a consulting group or some other group and they spin it up and then a community builds around it, then somebody else will build it like you said because they start from some other belief of a problem set and it kind of spawns from there. So let's talk a little bit about some of the common types of tools that we see. So if somebody said hey I want to run Kubernetes or I want to interact with Kubernetes, what are some of the ways that you can get started on day 1 or day zero. Tyler: [00:06:33] The two most common ways when when someone wants to get started is they want to install it on their laptop or they're looking for a service that they can like just log in and start kicking the tires. Tyler: [00:06:44] Usually time to Success is the kind of metric for developers that just want to see what this thing is. like cool I'll rip into the guts later. But right now I just want to see what interacting with this thing is like. Brian: [00:06:54] So from the laptop perspective there are a couple of tools that you might want to be aware of. So if you are working just with Kubernetes, you just want to work in the Kubernetes domain. Probably the most commonly used tool for installing on your laptop is something called MiniKube. from an OpenShift perspective from a Red Hat perspective, We've taken a similar approach and built something called MiniShift. So if you want to run OpenShift on your laptop you can do that as well. And both of those tools will give you a sort of single node, single master environment. So it's not highly available. It's going to be very respectful of the amount of RAM you have on your laptop. Typically it's going to install it in some sort of virtual machine on your laptop although there's variations on those installers and they're going to give you the same look and feel you would have is if you had a production level high availability Kubernetes environment or OpenShift environment or any other environment that you're working on like as a developer for example that run and then having that interface we can start configuring it. Tyler: [00:07:53] And what's nice is both those tools follow that pattern to also configure your local CLI tools. So once it's done you want to run your KubeCTL commands or OC commands with OpenShift. You can it's already configured to point at that local instance. Brian: [00:08:10] The other most common one you mentioned this is I don't want to deal with any of that stuff or I only have 4 gigs of RAM on my laptop or whatever. Let me just get this running in the public cloud and there's two categories of things that fall into one of them is I want to use the native Kubernetes services from somebody. So whether that's a Google container engines or G.K.E, Maybe it's Microsoft's Azure container service. And then there's some others that are out there. You may want to just log in have a Kubernetes instance start throwing some containers at it and get going. The other way to do it is to use QuickStarts. So you want to use the public cloud. You want to use some of their compute resources but you want to be a little more hands on what the Kubernetes is. And there's various kind of QuickStart that are out there the folks from Heptio, Joe Beta's company has a quick start for getting things up and running on AWS. Digital Ocean has a really QuickStart way of getting a Kubernetes environment up on their on their droplets so you can

4 kind of decide how much hands on Kubernetes is you really want but it takes you away from having to have to manage any hardware underneath it. Tyler: [00:09:19] Yeah I think that's another common way. Like you said besides just want to push containers a lot of have two groups right so you have developer who say hey I want to build containers and run them and then you have the more upside where you're saying, hey will I care about how these containers run, how they get restarted how they're scheduled, how to get access to resources. So for them \ logging into something like an OpenShift Online, or G.K.E. needs to say Well here you go here's your way to go where I can bootstrap a whole production-ish cluster on you on a cloud and start to kick the tires there so that's a popular approach too. Brian: [00:09:58] And I think what we sort of laid out even just in those those two sets of examples is basically there isn't one set of users. There may be developers that literally just want to write a java application, node application and push it, in which case they want as little friction as possible of those little kind of knowledge about Kubernetes. It's possible in some cases you have these DevOps teams which could be multiple people, it could be people with multiple skills who have to know a little bit of both you know what language and runtimes are supported. But I also need to know how did those masters interact with the Kubelet or how do I do updates. And then you're going to obviously have operations teams who need to know everything about how Kubernetes is going to run from an infrastructure perspective. So it lends itself to answering a little bit of that basic question we had at the beginning which was, why are there so many variations and options and the reality is you know there's a lot of different starting points that people have from this from. From dealing with Kubernetes or just dealing with new applications. Tyler: [00:10:51] Yeah I think what's what's even more interesting is we get to the next group which is the Kubernetes specific installers. so installers built from the ground up. All they do is install and upgrade Kubernetes. the first on Kubeadm is sort of the official Kubernetes installer.. Well wait, if there's an official one, why do we have all these other ones? One of the reasons is right now the tool doesn't deploy multiple Kubernetes masters. So if you wanted to have say like a fully H.A. production type cluster, Kubeadm doesn't yet do that. So I think that's also where the feature set to the various tools have allowed this to scale out even further. Brian: [00:11:31] Two points on that one. Before you go into any more of these one is you mentioned that the official tool doesn't support H.A. and I think there's some misconceptions out there in the market. We talked early on, Kubernetes has this concept of masters that provide the control plane and then there's the workers where the Kubelets run. The reality is you absolutely can build H.A. highly available Kubernetes environments. it's like you said, the default one that comes from the CNCF doesn't necessarily set that up for you but there are plenty of other tools and things that come from vendors and things in the cloud that absolutely are HA. The other thing real quick for anybody who's listening to this. Don't take these as completely extensive lists. So if you're working on a project and you don't hear a name called, don't feel bad. We're trying to give some some examples that we know off the top of our head, we'll try and provide some more complete list in the shownotes. But don't feel bad if we don't list the other 23 install options or something like that. [00:12:27] Yeah and that's a really great point. On the on the highly available piece, I mean Kubernetes is designed from the ground up to be highly available. one specific one I worked with is Kargo, which builds really highly available. One one of the best ways to understand kind of how that works and you know not to go too into the details but think about it this ways. Even the masters are just Kubernetes worker nodes that are only running Kubernetes core services and those Kubernetes core services are basically are the first pods that are deployed by Kubernetes.

5 Brian: [00:12:58] So Kubernetes treats its own system master resources as workload that it runs just on the specific set of nodes and the other thing about about H.A. environments is they're not simply going to be your normal like install command line and then yes, yes, yes. you have to take some pretty interesting variables into consideration. I know you and I both worked from a systems engineering perspective directly on some of these systems and you start getting into how big and how much memory and how much CPU your masters have. How physically far away are they going to be located? How much delay is in between them? Are these in the same location? Could could have fiber cut or a telco cut separate them and how many masters? one, two versus three, or more than that? So I think that's why there has to be a little more thought put into how you deal with HA and that advancement has to be built into these installers and upgrade systems. Tyler: [00:13:48] What are the things which varies on all of these, whether it's these specific ones or even some of these QuickStart ones. There's basically two levels of installer's. one is, hey give me the Linux host you want to install Docker and Kubernetes on and I'll go do that. And the other one is, point me at the IaaS provider, whether it's a public cloud or private, and I will ask for the resources I need and stand them up to and then install. So like for example Kargo, it's using uses Terraform. You could pointed at say OpenStack and say, I'm going to make a six node, three masters and three workers and we'll also ask for those six nodes. So there's kind of like two different ways and there's pluses and minuses to both, but that's usually something you see, it's kind of a breakdown between the tools also. Brian: [00:14:32] You know you mentioned Terraform. Terraform sometimes gets lumped into that category with Chef, Ansible, Puppet, Salt. Are those tools that people think of as config management tools but you are nowadays packaging all sorts of different software. Those fit into a category of interact with Kubernetes as well, true? Tyler: [00:14:47] I mean just like any other piece of software though, those tools do besides config management. they also in different levels do some amount of orchestration and use for installation so you can say I have Ansible playbooks that will install Kubernetes and actually for example Kargo is one that after it uses Terraform to build the nodes then it uses Ansible to do the actual installation. the OpenShift installer's uses Ansible. There's tons of Chef recipes and things like that so that may to make more sense if you're already a big Puppet and or Salt shop. and you might be at that as a way to deploy Kubernetes because it fits more into your kind of normal workflow. Brian: [00:15:25] So we spent a lot of time talking about day zero or day one tools. The things that are going to help you set up an environment or install an environment or get you on board with, whether it's a single node or highly available. What about the stuff from day 2 to day 2000? what are some of the categories that people should kind of be aware of in terms of tools. Tyler: [00:15:45] I just have to first start off by saying I'm I feel like at this time in its life cycle Kubernetes, It's a real good indication that there's a lot of uptake of Kubernetes nowadays is what's going on a couple of years we're talking pretty heavily about data tools how do you upgrade it. How do we monitor? how we log it? where some of these other open source projects worked on the past, they're five years in and no one's even asking about how we were logging this. How many people could possibly be running this if they're not logging it. Right? So I think that's a pretty good sign for Kubernetes as a whole. I think it goes back to installations, but I think upgrades are big too. That's when people say ok cool. Now is out. How do I get there? So there's a number of different tools to do that. Brian: [00:16:25] I think one thing for folks to put on your radar screen is I use this example all the time. So if you've been on the operations side of the infrastructure side for a long time you know keep these timelines in mind if you are a VMware show, so you did virtualization, you were getting

6 a major release of VMware about once every year and then a minor release about once every six months and people were trying to figure out how to work those into their schedule. if you were looking over the last few years at something like OpenStack, you're getting a new release every six months and with Kubernetes you are getting a new release about every quarter and then some maintenance releases sometimes in between that quarter. So with that thinking, your infrastructure isn't static. you know your infrastructure is software and you've got to be thinking okay how do I work this in from an automation perspective and repeatability perspective? how do I expect to want to update that fairly frequently because you want to stay up to date with a fairly recent set of features. Tyler: [00:17:21] I think that's huge and I feel like there's a bit of a misnomer there because people say hey I have a thing that works. Now we are into Oracle databases. I'm loathe to upgrade because the upgrade process could be potentially risky. And then the other new version may introduce new problems. What's actually faster moving software, these younger pieces of software and newer distributed systems kind of actually works backwards. once you start falling too far behind it's way hard to catch up because no one's tested skipping five releases or you know there may be a bug in this version but then the next version next quarter fixes that bug. it's once you get in that regular upgrade cadence, that actually makes things more stable and less risky than worse. Brian: [00:18:05] So I think in terms of categories of day 2 tools, it's always good to ask, ok I have an up and running Kubernetes, what's a common way to update this? we're seeing a couple of different things happen in the marketplace. obviously if you're using one of the native public cloud services, so native Kubernetes from one of the public cloud services, you shouldn't have to do anything. Think of it as like sort of a SaaS model, but you do have to be aware of when they are doing updates so you know from a compatibility perspective how are they running a 1.7, 1.6, 1.8. That's something to be mindful of how are they going to communicate that to you. But in terms of if you are running the software, whether you run your own data center or not there's a few different options. Tyler: [00:18:44] I think what you're typically looking for is are there default tools that a lot of people are using. Are there things that come with my distribution or the commercial offering that I had in general. The community is working towards much more. One click. Rolling updates and things that you can do without disruption whatsoever. So that's always been very top of mind for the community, is that this can't be something that can be an impactful event. Tyler: [00:19:07] The one thing you can do, kind of right off the bat from a thought process perspective, whether you're a developer or your more on the Ops side is as you deploy your apps and containers you know using the various capabilities within Kubernetes to make the loss of any individual container or POD non-disruptive using things like replica sets and published services and things like that. So if you have at least two or more of instances of your app running then rolling upgrades work way easier. It's that usually what happens is there are some apps that only have one of these so yes it can. It's going to go down you know won't be down long during an upgrade but it will go down.. Now you're starting to get pushback. when can we do this upgrade and can we push that back until after this busy season or and that's when you start going. So whether you're doing rolling upgrades now or not. The key thing is as you deploy and design your apps you want to make them you know very scalable and resilient. Brian: [00:20:04] And as you're talking, I'm thinking about there was a really good example video, and I'll put it in the shownotes of a company called MacQuarie Bank who is down in Australia and New Zealand covers the Asian Pacific area and they did a really nice talk at Red Hat Summit it was part of the OpenShift gathering. They were basically talking about how to they manage their upgrades from one release to another and really talked about how they will have multiple systems

7 in-house to validate new projects and new testing that they're able to not only have blue green and rolling upgrades for their API, and for their external mobile services and for their own internal testing so it's a real nice to talk about in real life production financial services with high risk visibility. Brian: [00:20:44] So you've got something up and running. How do I monitor this thing and how do I manage this thing. What are what are some of the tools that might be available to people. Tyler: [00:20:54] So I think monitoring is a big one and this is again something I really like what we're seeing with the Kubernetes community as a whole. look you can use whatever tools you like; you may even use a SaaS service like DataDog or something else. But the fact that the community quickly coalesces around a default, which right now we're seeing with the Prometheus project, open source monitoring and alerting tool, that seems to be a lot of people are moving in that direction. So it's good to say, I'll start with the default and if it doesn't do what I need I can move to something else so it's nice that we're starting to see all these different areas and defaults emerge so Prometheus is one. Obviously the type of stuff you use to like New Relic, Zabbix, SysDig, CoScale, all the usuals as being a kind of standard when you build application you know all the usual modern things apply. Brian: [00:21:52] I think the other thing that's important to remember with this is so you look at something like the see CNCF, which is where the Prometheus project is being governed these days. You know there's a whole community around Prometheus. And it's within the CNCF So there's a certain amount of integration but certain amount of awareness that the Kubernetes community should pay attention to what the Prometheus community is doing. They haven't embedded Prometheus as a working group within communities and I think that's really important to highlight for people because you know one of the things that happens when you get a lot of companies and a lot of engineers and stuff working in a popular project getting too much scope creep. And I think the see CNCF has done a nice job of saying we're going to encourage some compatibility and best practices but we're not going to force this into the Kubernetes project. So if if you don't like Prometheus but you love Kubernetes, there are going to be lots of other options and you don't have to feel like I'm going to wait for that next release of something in order for my Kubernetes needs to move forward, so they're allowed to move at different paces, but have awareness of each other, so that if they want to do some integration they they know how to do it. Tyler: [00:22:51] I think it's also important from the standpoint of even if you do want to use Prometheus is when you kind of tie that rope between the two by putting them in the project you know they start gating one another. Well we want to adjust it this way we're reporting this information at API, well it's going to break Prometheus, So you have to wait for it and you end up with way more coordination versus sort of just more traditional best practices of API versioning. things when you're in the same project you can take some shortcuts which are fast in the short term but in the longer term actually cause more problems. Brian: [00:23:22] Right. And I think the other thing is we're really learning in the community some best practices so you take for example what people are learning about monitoring. And one of the things that they're learning is how to do this concept like sidecar containers where you put the application in a container, run the monitoring agent in a container and you put those both in the same Kubernetes pod so they're logically linked together and they're learning from something like that is also being applied to proxy services for frameworks like Istio that are starting to emerge. nice thing is some of the learnings from things like monitoring or logging which are day to day things that learning and those sort of concepts and patterns are going to eventually flow into being useful in more advanced application patterns which we'll get into maybe in a later show.

8 Tyler: [00:24:04] I think that's pretty big. You know seeing those learnings come back in and kind of adjust it's really impressive to see how that helps. Brian: [00:24:11] I think the other two things that we're going to wrap this up fairly quickly because we are hitting our time limit and we've talked about a lot of tools, logging goes along with monitoring and management the same way. There are some stacks like the ELK stack and the EFK stack in terms of logging and monitoring and management. There is third party SaaS tools like Loggly and lots of others that are out there. I think that same discussion we just had about monitoring kind of applies for logging as well but like you said the fact that we're talking about logging in production just a couple of years into this project is really important. Tyler: [00:24:41] Yeah it's like you said it's just like the monitoring stuff all the same and same kind of thing. The case that seems to be kind of establishes the default but again not part of the project. And then it's what fits into your flow so why standup a whole separate logging or monitoring just for Kubernetes. And if you can fold it into what you're doing exit with your existing tools. Brian: [00:24:59] So we had a thing on our list we said maybe we should dive into some of the application frameworks that are becoming popular and you know what, that is its own show or a series of shows that we will get into very soon we're not going to cover today. And I think we're hitting up against our time frame. But Tyler I think we covered a lot in terms of why are there so many tools, there are some good reasons for that and some community reasons for that. I think we touched on getting people started getting them going after day one. Any last thoughts about how you wrap your head around what's the right tool box to have recovered time. Tyler: [00:25:27] I think the first thing is how I would pick any sort of tooling when it comes to this item in general is 1. What's the default? 2.What's the most popular thing? Those type of books because it has more community following behind it. If you don't have a strong opinion at least start there where it's the thing that everyone else is using because I don't need to have my custom homegrown solution, let me look there. And then if it doesn't do what you need, see where people have peeled off to other projects they also reuse. And also obviously reuse something you have if you like it or stick with something that's really popular. Brian: [00:26:01] Yeah I think that's great advice you know use the defaults reuse with minor modifications if needed. And then if you really have to build something on your own you know consider that sort of a last resort because Yet at some point you're going to have to you're going have to own that tool forever. So I mean to wrap it up with that folks as always thanks for listening. You know we would love reviews and feedback all the ways to contact us or in the show notes. So everybody thanks again for listening for Tyler and for myself. We will talk to you next week.