WEBVTT 00:00.000 --> 00:19.000 Hello everybody, appreciate that you all managed to attend our talk which is the latest 00:19.000 --> 00:24.000 one on the second day on that awesome constant conference. 00:24.000 --> 00:31.500 We are talking today about Kubernetes on-premise which is basically quite difficult 00:31.500 --> 00:39.500 and challenging talk and we at metal segment all created a software which helps you to create 00:39.500 --> 00:46.700 Kubernetes at scale in your own data center on-premise and should be or is in our opinion 00:46.700 --> 00:53.100 available alternative to proprietary solutions like Broadcom or VMware from Broadcom 00:53.100 --> 01:00.100 and other solutions to create Kubernetes in your own data center. 01:00.100 --> 01:06.700 My name is Stefan Mayer, I'm a CTO at excellent with a small consulting company from 01:06.700 --> 01:12.100 Munich, my colleague Gerrit will take over in the middle of the talk. 01:12.100 --> 01:19.100 We are located in Munich and about 30 people founded in 1999. 01:19.100 --> 01:30.100 We mainly focused on the financial market so our customers are banks and the insurance is mostly. 01:30.100 --> 01:41.100 And we also became two years ago also hosting provider for Kubernetes as a service which is based on the technology we implemented at metal stack. 01:41.100 --> 01:45.100 We also see in CF silver member. 01:45.100 --> 01:57.100 Yeah, and I try to, or we try to show you how we in Europe can get more confident in doing our own cloud business 01:57.100 --> 02:02.100 and not only rely on American cloud providers. 02:03.100 --> 02:14.100 As you all for sure recognize that the political landscape changed a bit in the last weeks and that will continue. 02:14.100 --> 02:25.100 I'm pretty sure and this enforces us in Europe to act. 02:25.100 --> 02:30.100 It's not only that the cloud providers are dominating the market. 02:30.100 --> 02:38.100 It's also that the on-premise market is dominated by American software companies. 02:38.100 --> 02:43.100 VMware at the most prominent one. 02:43.100 --> 02:53.100 And we at metal stack think or for pretty sure that we should do something against this dominance. 02:54.100 --> 02:59.100 Grown as an open source alternative in the European area. 02:59.100 --> 03:03.100 And we also believe that Kubernetes is here to stay. 03:03.100 --> 03:07.100 It's already 10 years old and proven to be very stable. 03:07.100 --> 03:22.100 And should be the foundation for clouds created and born in Europe for all kinds of applications for all kinds of applications. 03:22.100 --> 03:28.100 And also for as a base for virtualization workloads. 03:28.100 --> 03:34.100 This will help the European industries to gain freedom. 03:34.100 --> 03:43.100 And also leave the proper territory boundaries they actually have. 03:43.100 --> 04:00.100 In Europe the GDPR also requires that we own our data which is not the case if all our data is located or controlled by American or Chinese companies. 04:00.100 --> 04:15.100 On the other hand we are still to shy in Europe to create something new which is big enough to create an alternative. 04:15.100 --> 04:24.100 But we are not to dump we just have to start and that's what we did. 04:25.100 --> 04:43.100 The time to leave the public cloud is now as in the keynote yesterday was said it's the it's the global switch day so in our opinion it's also the global switch day to leave the public cloud now so yesterday. 04:43.100 --> 04:56.100 So what we have done so to bring the cloud into your own data center we created a metal as a service API which. 04:56.100 --> 05:08.100 Basically enables you to turn your own hardware in your own data center into an elastic cloud infrastructure and the shown here in the picture imagine you have. 05:08.100 --> 05:16.100 Here shown to different data centers full fully equipped with service and switches. 05:16.100 --> 05:25.100 They are connected to our API and as a as a user as a consumer as an API consumer you can tell. 05:25.100 --> 05:37.100 This API please give me a machine of specific size and t-shirt size for example with specific amount of CPUs and RAM and hard disks. 05:37.100 --> 06:00.100 Tell this this API and one and a half minutes later you will have a fully installed ready to use bear metal machine with this operating system and specification you gave the API and we use that's that to create later Kubernetes clusters based on that API specification. 06:00.100 --> 06:10.100 And you will ask why bear metal we have virtualization technologies as guys already talked about today. 06:10.100 --> 06:24.100 There are pros and cons to use bear metal you obviously get the best performance if you skip the virtualization layer and the lowest latency and in. 06:24.100 --> 06:39.100 Since we have 100 gene networks or more since we have flash disks you would not give 20% or more away for the virtualization layer. 06:39.100 --> 06:53.100 Instead you will gain the whole performance of the metal you also get the best possible tenants separation because if your workload runs on this server and nothing else then no one else can. 06:53.100 --> 07:00.100 For example interrupt your workloads with the performance he creates. 07:00.100 --> 07:14.100 The stack is also simpler so between your application and CPU is nothing else then the container virtualization which is. 07:15.100 --> 07:29.100 And it's also optimized for GPU workloads or even virtualization workloads which was mentioning beforehand there are of course downsides so if server dies your workload dies of course. 07:29.100 --> 07:51.100 And you can kind of do over provisioning or life migration but these are topics which are covered by the Kubernetes mechanisms so if a pot is not there anymore deployment will guarantee the Kubernetes schedule will guarantee that there will that the pod will get started restarted on a remaining server. 07:51.100 --> 08:20.100 So what does metal stack cover so it manages the whole data center from the from the foundation so it manages racks machines networks firewalls operating systems it's fast them opinionated for this special use case you will have no window lock in it's for the on premise use case you will not have any dependencies to outside cloud providers you gain data sovereignty. 08:20.100 --> 08:38.100 You can use the software in your own data center it's completely open source you will get the best price performance ratio you could possibly get because there's nothing else between and it's completely API driven and implemented in goal. 08:38.100 --> 08:52.100 And we use this in production since about five years in critical industries in Germany to our since 2018 for us the current known production deployments. 08:52.100 --> 09:08.100 Our consistent of about 400,000 containers 250 Kubernetes cluster more than 60 switches which are managed more than 8,000 and a 1,800 servers and five database of assistance. 09:08.100 --> 09:16.100 To get more in detail Gary is taking over and explains how this all works. 09:16.100 --> 09:32.100 So thank you. I think I will do it like this. Okay, so I brought a really complex slide here and I think I don't want to go into all the details it's from our documentation at docs metal stack.io. 09:32.100 --> 09:58.100 And the reason why I show this slide is just to show you that this is not just a single piece of software that runs somewhere it's also part of your data center it runs inside your data center. 09:58.100 --> 10:23.100 And it's also like an opinionated on how the network in your data center should work. So we decided for a BGP in the data center and there's also slide data would want to go into with more details there, but yeah it's just for you to know that we have this bigger concept where you have like control plane with the API and then there are we call it partitions. 10:23.100 --> 10:31.100 So I'll provide us call them zones, I guess. And for us the partition is set or group of servers which participate in the same cluster. 10:31.100 --> 10:46.100 So our solution also includes then how you cable up the wreck and we intended when you go out of machines as a provider that you can easily scale it up. 10:46.100 --> 11:02.100 The installation by adding more x and yeah the clusterology that we propose is failure redundant so all the switches in the infrastructure can be exchanged without interrupting the traffic for the users. 11:03.100 --> 11:14.100 Yeah all our machines need to be dual attached and we even have a constraint that servers cannot be allocated until they are dual attached to the switches. 11:15.100 --> 11:33.100 Yeah, an overview of a supported hardware on docs.metallistic I also we don't claim that we can support all the hardware because we also expose some management and points to managing a machine like power on power off get the power consumption of the server. 11:33.100 --> 11:38.100 The current stage and also obtain a serial console to the machine and stuff like that. 11:38.100 --> 11:47.100 So we implemented this through IPMI and red version all of these friends that you're probably aware of. 11:47.100 --> 11:57.100 So this is quite a tough part and yeah this is quite a tough part and yeah we hopefully defined a good interface to extend this. 11:57.100 --> 12:09.100 So if this is an issue for you that we only support super micro or data servers then you can also just contribute vendor implementation when you implement the interface and it should work. 12:09.100 --> 12:20.100 Yeah on the switches we use Sonic OS so yeah we thought this is a good open source operating system to run on the switches. 12:20.100 --> 12:42.100 And yeah so yeah I think that's it for this slide next one will be a bit about network so we simplified here as well but just to get the idea across so yeah we have like tenants in our API and tenants can create projects and when you. 12:42.100 --> 13:05.100 Yeah create a project you can also allocate networks within this project and yeah in this project you can place machines and you can also put them into your own project networks like node networks and yeah you can allocate machines in this network and then you can the machines can reach each other but. 13:05.100 --> 13:12.100 If you want to establish a connection to other networks then you also need to place a firewall component inside your. 13:12.100 --> 13:22.100 Yeah inside your node network so this is just another bear metal machine which can be a smaller size because it does not have so much to do it just routing traffic. 13:22.100 --> 13:40.100 And this component will do a VRF termination so as I said we have a BGP in the data center this also book that guided us which I can really recommend it's called BGP in the data center so this was our Bible back then for reference how to implement this and. 13:40.100 --> 13:54.100 Yeah with this firewall you can then make connections to other networks like the internet or other internal company networks and yeah also solution consists of an IP address management system so. 13:55.100 --> 14:10.100 Yeah people can acquire IP addresses and bind them to their nodes and yeah the appeared addresses that are bound to the nodes will automatically be routed through the entire switch layer by using FR and yeah the nodes will then. 14:10.100 --> 14:26.100 Okay so the way that we use it is like this. 14:27.100 --> 14:36.100 We actually have like a border for metal stack which is really clear it's just the bear metal machine provisioning and we end there so. 14:36.100 --> 14:51.100 I just think by itself does not know what Kubernetes is but we have an integration with another open source project which is called gardener which I want to show you after this slide actually. 14:51.100 --> 15:09.100 This will then utilize the metal API for creating machines for Kubernetes clusters but yeah as you can see as a whole you can build a complete data center stack with open source projects and it's I think a fast and clean architecture and it works well with Kubernetes and we. 15:09.100 --> 15:17.100 Yeah developed this in Europe in Germany in Munich and yeah so. 15:17.100 --> 15:21.100 Next slide how can we deploy Kubernetes. 15:21.100 --> 15:36.100 Yeah so what is gardener so I already mentioned that it's another open source project that we did not come up with but it was written in the same time when we started all journey with metal stack and we found each other quite quickly and. 15:36.100 --> 15:55.100 I thought yeah will it works well together so it's called the gardener and yeah they started in 2018 and they are actually a Kubernetes installer and it's a multi cloud platform more or less where multiple cloud providers can implement like also interfaces and. 15:55.100 --> 16:01.100 Yeah with these interfaces implemented it can utilize work or not creation. 16:01.100 --> 16:12.100 Yeah automatically and you have like these auto scaling capabilities you have like rolling updates for Kubernetes clusters and yeah stuff like this all the data operations I think they. 16:12.100 --> 16:19.100 Yeah to a really great job in this and also we know these people and they are quite friendly and have an open community so. 16:19.100 --> 16:31.100 If you're interested in like providing Kubernetes in your own company then you should definitely have a look at the gardener as well because I think it's a cool product. 16:31.100 --> 16:37.100 Yeah and it's also good for actually bare metal use cases like in our case because. 16:37.100 --> 16:47.100 Yeah the idea is that gardener spins up more Kubernetes control planes inside a Kubernetes cluster so it's like if they call a cubeception model. 16:47.100 --> 16:59.100 And yeah it's not like when you provision Kubernetes that in your cluster you can have your own API server as an end user you do not see the API server for Kubernetes inside your cluster. 16:59.100 --> 17:16.100 It's running inside another bare metal machine which is operated by by the operator actually and yeah this way you can fully leverage because you have then many tenants in one operator cluster and you can fully leverage power of. 17:16.100 --> 17:22.100 Yeah physical machine there because you can place many tenants inside this machine there. 17:22.100 --> 17:30.100 Yeah so that's it about gardener then there's also something that we want. 17:30.100 --> 17:44.100 To just start with yeah we want actually to make proposals for record configurations so everything we just told you is completely installed in your on premise. 17:44.100 --> 17:52.100 But our idea is to make this a bit easier because the whole deployment is quite complex and you also need to understand the BTP parts and stuff. 17:52.100 --> 18:01.100 As otherwise you will not be able to set this up and also not to you kind of debug this we wanted to propose like reconfigurations. 18:01.100 --> 18:12.100 Which you can use so we have one yeah smaller reconfiguration that we want to propose with just eight servers for starting and then a larger ones. 18:12.100 --> 18:18.100 And the only thing that you would need is an internet connectivity and space and the power. 18:18.100 --> 18:30.100 So yeah as Stefan already said in the beginning it's really hard to establish trust in this domain because for European. 18:31.100 --> 18:59.100 Yeah solutions I think most of the people think it's not ready yet so in 2022 we thought yeah we should even make a hosted version of it so just installed in a data center and we did this in Munich in a data center and offered this for public use to so this we call matters they're cloud and yeah they can try it out and if you don't believe it works and there's now the possibility to do it so you can spin up coordinate is clusters there. 19:00.100 --> 19:17.100 Then this small development update because I'm also the maintainer for many repositories that we have on GitHub and I just want to say thank you at this place for all the contributions that we will see from the outside it's always great. 19:17.100 --> 19:24.100 So what we're currently working at is the basic IPv6 for support so everything right now is working with IPv4. 19:24.100 --> 19:40.100 Yeah and we want to enable also like a service exposure and Kubernetes for IPv6 addresses so there's one pull request for Stefan which is going to be merged I think pretty soon so yeah I hope that. 19:40.100 --> 19:45.100 We will be able to just a provision Kubernetes cluster fuel union IPv6 soon. 19:45.100 --> 19:59.100 Yeah then one thing that we did lately and one of the latest releases is also to enable offline environment installations so it's also now possible to set up metal stack in environment where you don't have like internet. 19:59.100 --> 20:17.100 Connection but it requires you to set up like NTP servers and DNS servers on your own but at least all our components can now be configured in a way that all these external dependencies can be configured in point and three years on data center so yeah you don't need internet connection there. 20:17.100 --> 20:39.100 Then there's also one bigger MEP and MEP for our project means metal enhancement proposal and there we describe how we want to have a B2 API and yeah we in the past treated the metal API just as a low level API so there's no real concept of resource hoping. 20:39.100 --> 20:48.100 And yeah we wanted to change this now so grounded it's not required to have this but actually we want now to treat the API or users. 20:48.100 --> 21:01.100 Yeah with scoping because I think it would make a lot of things much more secure in terms of misusing a token for something different. 21:02.100 --> 21:06.100 Yeah this is one thing that we're going to work on this year and I hope we can finish this. 21:06.100 --> 21:13.100 This will be then GRPC based and yeah we also already have some ideas how to implement this. 21:13.100 --> 21:30.100 And then one other topic that we want to tackle this year is to broaden the hardware support because as I already said we only support limited set of hardware and yeah then it makes sense that we go through this again and maybe propose like a standard API. 21:30.600 --> 21:46.100 Interface or something like this which should work out of the box but if a vendor deviates from the norm then it does really not work but at least you have a like default implementation for how an IPM I mentioned server it should work. 21:46.100 --> 21:51.100 So these are stuff that we want to work on here this year. 21:52.100 --> 22:03.100 And there's also possibility to try out metal stack if you want so we have one project which is called the mini lab and there the control plane will be. 22:03.100 --> 22:21.100 Yeah started in the Kubernetes and Docker cluster and then there's another open source project which is really cool which is called container lab which helps you to emulate switches on your local machine with the help of VMs so it will also spin up to. 22:21.100 --> 22:44.100 And then there will also be like a quamo virtual machines which will be started with specific bios which does a pixie boot with the EFI and then you can provision these machines locally without having to install anything in your data center just on your local machine. 22:44.100 --> 22:54.100 So yeah this is a way how you can try it out easily without investing tons of money into a new hardware or something like this. 22:54.100 --> 23:12.100 So yeah thanks for having us I mean great great pleasure to be here at the first them and I hope that with our project we can give something to the community here and we're keen to hear about the use case that you have and the journey. 23:12.100 --> 23:37.100 And yeah for us here we have the documentation there available for you if you're interested in it the development takes place on GitHub and yeah we have a community selection where you can also join us questions where I think our open minded people so just reach out to us if you have any questions and yeah I guess that's it or do we want to add something. 23:37.100 --> 23:40.100 Okay then that's it and thanks. 23:42.100 --> 23:48.100 Questions. 24:13.100 --> 24:24.100 Okay a question was if the traffic that's going through our switch player is encrypted for every tenant right. 24:25.100 --> 24:38.100 Noter the data plane traffic is not encrypted by default inside one data center there's if you want to encrypted the traffic between data centers you can do this with. 24:39.100 --> 24:52.100 Maxack GPX for example, which are available for available for these type of citrus we use any further questions. 24:53.100 --> 25:03.100 Yeah you said it it manages the racks machines network files so does it replace like net box and I found as well. 25:03.100 --> 25:13.100 Yeah it does it reflect does it replace net box most of the question yeah yeah we started using net box at the beginning. 25:14.100 --> 25:31.100 We thought we can use net box for the ipom at least but it was to slow for our use case and we implemented that natively and go again it's called go ipom quite popular repository at. 25:32.100 --> 25:42.100 For now and yeah we have basically a whole CMDB built in yeah which is not net box. 25:42.100 --> 26:00.100 Yeah so what type of hardware is supported most of the questions so we on the server side we actually support super micro from X 11 and you are. 26:00.100 --> 26:12.100 Dell with iDrag 9 and it's carried said sonic switches so every switch which is capable of running the sonic operating system. 26:12.100 --> 26:22.100 Should be supported and GPUs are actually only the Nvidia RTX 66090 and the H100. 26:22.100 --> 26:31.100 Any more questions. 26:36.100 --> 26:38.100 Thank you.