WEBVTT 00:00.000 --> 00:12.640 Hello, I'm Yuan, and I work on swift services and infrastructure 00:12.640 --> 00:14.440 at Apple. 00:14.440 --> 00:18.440 And what that means is I often find myself building swift programs and putting them 00:18.440 --> 00:20.680 in containers. 00:20.680 --> 00:24.800 And so now I maintain a little package called swift container plug-in. 00:24.800 --> 00:28.320 It's funny how that kind of thing happens. 00:28.320 --> 00:33.440 We're living in the cloud native era, and that's changed not only how we run our services, 00:33.440 --> 00:37.560 but also how we build and deploy them. 00:37.560 --> 00:40.240 So let's see, I'm ready to deploy my service in the cloud. 00:40.240 --> 00:47.240 It used to be that I would compile my binary and then FTP it up onto a server, and that was it. 00:47.240 --> 00:51.920 But that's no good anymore, because my cloud provider runs containers. 00:51.920 --> 00:56.080 The binary alone isn't enough. 00:56.080 --> 01:00.800 And what that means in practice is that I have to wrap it in a container image. 01:00.800 --> 01:05.600 That's a standardized packaging format that holds my application, and then any supporting 01:05.600 --> 01:09.160 files that's going to need at runtime. 01:09.160 --> 01:14.920 Once I have my image, I can publish it to an image store in the cloud called a registry. 01:14.920 --> 01:19.240 You've heard of Docker Hub, but your cloud provider probably runs its own registry, and 01:19.240 --> 01:22.840 if not, you can host self-host. 01:22.840 --> 01:26.720 But the beauty of publishing a container image in a registry is that once it's up there, 01:26.720 --> 01:32.520 I can deploy it to any container-based cloud provider I like, whether it's in the public cloud 01:32.520 --> 01:37.120 or in a private data center. 01:37.120 --> 01:42.520 But the part that particularly interests me is how these container images get built in the first place, 01:42.520 --> 01:45.880 because that's the part that I deal with on a daily basis. 01:45.880 --> 01:52.800 Containers are now part of my development cycle, or let's face it's really more of a debug cycle. 01:52.800 --> 01:57.160 Because whenever I want to build or test my service, I have to build a container image, 01:57.160 --> 02:02.080 and I do it over, and over, and over, and over again. 02:02.080 --> 02:07.160 And it kind of turns my workflow inside out, because conventional container tools want 02:07.160 --> 02:10.320 to manage the whole process for me. 02:10.320 --> 02:15.200 The way I see it in this cloud native era, my final product is the container image, 02:15.200 --> 02:19.160 and I'd really like my familiar development tools to be able to produce it for me. 02:19.160 --> 02:23.240 And so the question is, can sort of package manager do it? 02:23.240 --> 02:29.000 Well, it turns out that it can, but we'll need to pull off a couple of tricks to make it happen. 02:29.000 --> 02:33.120 But don't worry, I'm not a member of the Magic Circle, I'm not sworn to secrecy, 02:33.120 --> 02:37.480 and I'll let you know all details. 02:37.480 --> 02:43.120 So, OK, so here we have just a little hello world. 02:43.120 --> 02:46.320 It's very simple, but it has everything that we need. 02:46.320 --> 02:51.200 One thing it doesn't have that you might be expecting to see is a container file. 02:51.200 --> 02:55.240 You might call it a Docker file or a podman file or something like that. 02:55.240 --> 02:59.280 But whatever you do call it, it's a recipe that tells a container runtime 02:59.280 --> 03:04.880 how to build a container image, where we are going, we won't need one. 03:04.880 --> 03:08.800 Because we're not using a container runtime to build this image. 03:08.800 --> 03:14.280 Swift package manager to do it all by itself, using the Swift container plugin. 03:14.280 --> 03:18.360 So to get started, I need to install the plugin, like I said. 03:18.360 --> 03:22.720 And now I'm ready to build my image. 03:22.720 --> 03:26.280 The plugin is asking for network permission, because on MacOS, 03:26.280 --> 03:29.880 plug-ins run in a really tightly constrained sandbox environment, 03:29.880 --> 03:34.280 and it needs network permission so that it can push the image up to the registry. 03:34.280 --> 03:39.520 I'm running a local host registry, but it's still a network call. 03:39.520 --> 03:43.680 So what it's done is it's built my service, and it's packaged it up into a container image, 03:43.680 --> 03:48.960 and printed this identifier, and now we can run it. 03:48.960 --> 03:52.040 Service running, I can color it. 03:52.040 --> 03:54.280 Great. 03:54.280 --> 03:57.280 But thank you. 03:57.280 --> 03:59.280 It could be more topical, don't you think? 03:59.280 --> 04:02.200 So let's just make a quick change. 04:02.200 --> 04:04.320 Oops. 04:04.320 --> 04:08.760 World, right. 04:08.760 --> 04:13.000 And we'll go around the dev cycle one more time. 04:13.000 --> 04:17.640 And the great thing about letting Swift package manager drive the build process 04:17.640 --> 04:22.240 for the container is that we should get incremental builds for free. 04:22.240 --> 04:27.760 If you've ever struggled with a container build that just insisted on rebuilding the world 04:27.760 --> 04:32.560 for no reason you could, you could possibly figure out, particularly in the middle of a demo, 04:32.560 --> 04:35.760 then you know what I'm talking about. 04:35.760 --> 04:37.360 So what we've done is we've rebuilt. 04:37.360 --> 04:41.480 We've got another container image reference. 04:41.480 --> 04:48.480 I run it, hit the endpoint again, and that's much better. 04:53.280 --> 04:54.280 OK. 04:54.280 --> 04:57.720 So did you spot the tricks? 04:57.720 --> 05:01.280 They were all behind this single command, and I promise you it is a single command. 05:01.280 --> 05:03.400 It's just been line wrapped. 05:03.400 --> 05:05.800 Check number one is to build the server binary. 05:05.800 --> 05:08.440 And that doesn't really sound so difficult. 05:08.440 --> 05:13.520 Except, remember that I'm running on macOS, but we built a container image, which deployed 05:13.520 --> 05:16.280 on Linux, except it was coming from Ubuntu. 05:16.280 --> 05:20.280 And that must mean that we've built a Linux binary at some point in that container image. 05:20.280 --> 05:22.880 And that's a bit more magical. 05:22.880 --> 05:26.000 And the second trick is building the container image. 05:26.000 --> 05:28.520 And we did it from first principles. 05:28.520 --> 05:35.200 We're building completely organic free range container images, and so let's see how it was done. 05:38.480 --> 05:40.720 So we'll start with the server binary. 05:40.720 --> 05:45.160 And the conventional way to build one to build a container image is to use a container run 05:45.160 --> 05:48.320 time on your laptop or desktop. 05:48.320 --> 05:51.600 And it's not always obvious, but when you do that, when you build a container that way, 05:51.600 --> 05:58.960 you're actually doing your build on Linux itself, even if like me, you're running macOS locally. 05:58.960 --> 06:04.480 And that's because behind the scenes, your container run time is running a Linux virtual machine. 06:04.480 --> 06:06.080 It reads the container build recipe. 06:06.080 --> 06:10.520 And that's that file that we didn't need a moment ago with the container plugin. 06:10.520 --> 06:13.440 It starts up a container build inside the VM. 06:13.440 --> 06:15.040 It copies in your code. 06:15.040 --> 06:19.160 And then in there, it runs the Linux version of the Swift compiler. 06:19.160 --> 06:26.400 In a full Linux environment, and that's how it's able to build a Linux binary to put in the container. 06:26.400 --> 06:30.320 But there's another way that we could build that Linux binary, using the native Swift compiler 06:30.320 --> 06:32.640 that we already have. 06:32.680 --> 06:36.560 And that's because Swift has always been great for cross compilation. 06:36.560 --> 06:41.800 The Swift compiler is based on LLVM, which gives it the ability to compile for lots of other 06:41.800 --> 06:43.200 platforms. 06:43.200 --> 06:48.240 But a cross compiler isn't quite enough on its own, because to produce viable binaries, 06:48.240 --> 06:54.960 we need to be able to link against the runtime libraries and other files for the target system. 06:54.960 --> 06:57.360 And that's where Swift SDKs come in. 06:57.360 --> 07:00.960 They provide supporting files that we need to cross compile to Linux. 07:00.960 --> 07:06.400 And even to different processor architectures using the open source Swift tool chain. 07:06.400 --> 07:10.240 And cross compilation is just as useful if you're running on Linux. 07:10.240 --> 07:14.240 Maybe you have an X86 laptop and you want to deploy an ARM. 07:14.240 --> 07:18.400 Well, a Swift SDK will help you to do that. 07:18.400 --> 07:22.560 Earlier on, we built our binary using the static Linux SDK from Swift.org. 07:22.560 --> 07:26.400 So let's take a closer look. 07:26.880 --> 07:32.080 Okay, so I've already installed the static Linux SDK. 07:32.080 --> 07:37.200 And because it depends on the open source Swift tool chain, I've set that up as well. 07:37.200 --> 07:40.800 But that's everything I need to cross compile. 07:40.800 --> 07:44.960 I'll just create a little test project and build it. 07:44.960 --> 07:52.000 So I'm passing the Swift SDK argument to Swift build to ask it to cross compile using my SDK. 07:52.000 --> 07:54.000 The output is kind of underwhelming. 07:54.000 --> 07:56.880 It looks just like what you would normally expect. 07:56.880 --> 08:02.560 But what's a bit more impressive is we ended up with Linux binary. 08:02.560 --> 08:05.680 And it's a statically linked one as well. 08:06.720 --> 08:11.440 Now, statically linked binary is a great because they have very few runtime dependencies. 08:11.440 --> 08:13.280 They're self-contained. 08:13.280 --> 08:19.760 But they may not always be what you want because they can be a little bit bigger than dynamic linked 08:20.160 --> 08:22.560 binary, precisely because they're being all that dependencies with them. 08:23.920 --> 08:28.560 And you also might want to use some libraries that are available on your Linux system that aren't in the 08:28.560 --> 08:30.400 SDK, you can get it from Swift.org. 08:31.120 --> 08:34.080 And if that's the case, don't worry, we have you covered. 08:34.080 --> 08:42.960 You can build yourself custom SDK tailored to your own requirements using project called the Swift SDK Generator. 08:42.960 --> 08:45.600 It's also available on GitHub and I have it here. 08:46.240 --> 08:54.240 So let's get it to build us an SDK based on the official Swift 603 image. 08:55.840 --> 08:59.040 The first time you run this, it'll have to download quite a lot of data, 08:59.040 --> 09:00.800 but it'll cache it so it only happens once. 09:01.760 --> 09:06.960 And what it's doing behind the scenes is it's extracting out of that container image just enough 09:06.960 --> 09:11.600 of the Linux Swift environment to support the micro, the cross compiler running on my QS. 09:12.560 --> 09:19.600 So it's run and it's produced us a bundle, but before we install a bundle, let's just take a 09:19.600 --> 09:25.040 look and see what we're dealing with here. It's quite a long path, I think this is what they 09:25.040 --> 09:31.840 mean when they talk about a deep dive. But if you keep digging down, you'll find some directory 09:31.840 --> 09:36.080 start the existing names at loop familiar if you've done any development on Linux. And if we go 09:36.080 --> 09:44.560 a bit further, here are the dynamically linkable or dynamically libraries that support various 09:44.560 --> 09:51.200 familiar looking Swift libraries. Okay, so let's install this thing and try and build. 09:52.320 --> 09:59.040 So I install my SDK, come back up here and I should have a new SDK in my list, which I do. 09:59.760 --> 10:06.160 And this time, I will pass a slightly different identifier to that Swift SDK parameter. 10:06.160 --> 10:11.520 I get another really, they could put more fanfare into this because it's doing something 10:11.520 --> 10:20.960 quite interesting. Another fairly, fairly vanilla output, but this time, we have a dynamically 10:21.040 --> 10:33.680 linked Linux binary using the SDK that we just built. Okay. So that's how to build a Linux binary, 10:33.680 --> 10:38.320 using the Swift compiler that we already have on macOS. The second trick is to build our container 10:38.320 --> 10:45.440 image. But what is a container image really? There are a bit of a mystery because they're usually 10:45.440 --> 10:52.320 managed for you behind the scenes. They're a bit like the Yeti. You see their footprints as 10:52.320 --> 10:56.960 they're pushed and pulled around. You type some container command and a progress bar 10:56.960 --> 11:02.400 creeps across the screen, but you rarely see one close up. You don't have a container image 11:02.400 --> 11:07.280 sitting in your project build directory, usually. Well, actually it turns out that image are 11:07.280 --> 11:13.280 pretty straightforward inside, possibly also like Yeti's. And because they're an open standard, 11:13.360 --> 11:17.680 which was originally defined by Docker and then donated to the Linux Foundation, 11:17.680 --> 11:21.920 there are now lots of tools which work with them and we can equally build our own tools in Swift. 11:24.160 --> 11:30.960 So inside the image, there's a tree of objects and to keep it simple, we'll look at a single 11:30.960 --> 11:36.800 architecture image here and that's actually the thing called a manifest. That points to some 11:36.800 --> 11:42.000 configuration which tells the container runtime how to run the image and it also points to a stack of 11:42.000 --> 11:48.160 layers which combine together to make up the file system. The metadata objects are Jason 11:48.160 --> 11:53.120 and the layers are turbals and so this is something a structure that we can easily build in Swift. 11:55.760 --> 12:00.080 And since one of the, I think one of the best ways to understand things is to take it apart, 12:00.720 --> 12:04.880 even better if I can boot it back together again, let's see if we can find one of these creatures 12:04.960 --> 12:14.320 and take a closer look. So, over here, right, so the registry server is just an HTTP server, 12:14.320 --> 12:20.240 so we can fetch a container image by hand using curl. We need to start with the manifest, 12:20.240 --> 12:26.080 at the top of the tree and work out from there. So here's our configuration blob, 12:26.240 --> 12:34.640 we'll fetch that and it's Jason. So as you can see, this is the only information the runtime will 12:34.640 --> 12:39.360 use to run this image and the most important thing is probably this entry point. That's the binary 12:39.360 --> 12:45.920 that you should run inside the container when it starts up. So remember that name. The rest of it is the 12:46.000 --> 12:57.600 file system and this is our ball and so this is our, this first layer is our basic underlying 12:57.600 --> 13:02.880 Linux image and as you can see it's based on Ubuntu. There's lots of Linux stuff going on here, 13:02.880 --> 13:07.520 lots of files etc. We page down, we'll find eventually binaries that you'd expect to see on 13:07.520 --> 13:12.160 the Linux machine and further down there are libraries that we don't need to go that far. 13:12.880 --> 13:19.040 The next layer isn't terribly interesting, it just adds on the CA certificates that this container 13:19.040 --> 13:25.840 image would trust at start up, but this layer here is really important for us because this is where 13:28.080 --> 13:33.040 we put Swift into the mix. So you can see here's the Swift system, you've got Swift module, 13:33.040 --> 13:41.360 you're seeing already and if we come down further, here are those Linux.so files that we saw 13:41.440 --> 13:47.280 moment ago in the SDK. So this is the layer that takes our generic Linux image and specializes 13:47.280 --> 13:53.680 it into a Swift image. The only thing we require now is our binary and that's in the last layer. 13:57.680 --> 14:04.480 So there's our hello world and that's how you take a container image apart. What about 14:04.480 --> 14:10.080 putting it back together? Well, like so many things in life, we just need to retrace our steps 14:10.160 --> 14:15.440 in the opposite order and we could do it all using curl, but there's a program in the container 14:15.440 --> 14:21.680 plugin repository that does it for us in one hit. So we'll do that. What this is doing is it's 14:21.680 --> 14:27.680 taking our hello world binary, it's constructing a layer around it, stacking it on top of the Swift 14:27.680 --> 14:33.200 slim image by default but you can choose your own base image and then uploading it to the registry 14:33.200 --> 14:44.720 in this new address. We have yet again an image reference, we can run it, it starts up and there's 14:44.720 --> 14:58.080 our server. Okay, so you've seen all the tricks. We built a Linux binary by cross compilation and then we 14:58.160 --> 15:04.880 packaged it in a container image. We've just got time, I hope we've just got time, five minutes. 15:06.160 --> 15:11.840 To mention the unsung hero of our little show, the magicians are system stitching everything behind 15:11.840 --> 15:15.920 this together behind the scenes. That is Swift package manager plugins. 15:17.920 --> 15:23.040 Plugins license extends with package manager to suit our own requirements by adding our own commands 15:23.040 --> 15:29.440 and build steps which look just like built-in features. When we ran this Swift command earlier, 15:29.440 --> 15:34.560 what I actually happened behind the scenes was that Swift package manager started up and it spawned 15:34.560 --> 15:40.640 our plugin, passing it its own set of arguments. And the plugin then called back in to Swift 15:40.640 --> 15:46.880 package manager to say, please would you build the hello world target using our SDK. Swift package 15:46.880 --> 15:54.480 manager obliged and it gave us a Linux binary back, which the plugin then wrapped in a container 15:54.480 --> 16:01.440 image using container tool just like we did a moment ago by hand. That uploaded it to the 16:01.440 --> 16:07.280 image to the registry and then returned an image reference and Swift package manager printed it on 16:07.280 --> 16:13.360 the terminal ready to pipe into the next command. So Swift package manager plugins are the last 16:13.360 --> 16:19.680 little trick that lets the Swift tool chain build a brilliant explainery on macOS package in a container 16:19.680 --> 16:26.960 image and upload it to a registry all in one command. Hopefully we've also de-missed to find a 16:26.960 --> 16:32.640 couple of things along the way. We looked inside an SDK and found that it's really a bundle of 16:32.640 --> 16:39.440 Linux resources for the Swift cross compiler to link against. And we've also tracked down a yeti, 16:39.440 --> 16:44.080 I mean it's container image and we discover that it's not really all that mysterious. It's just 16:44.080 --> 16:52.560 some turbos and fancy dress. You can find all of this in the Swift container plugin repository on 16:52.560 --> 16:57.440 GitHub. But what I hope you've also seen is that this is just one of the tricks that you can 16:57.440 --> 17:03.680 conjure up with the tools and the Swift tool kit. You can use it as is to build container images 17:03.680 --> 17:09.120 for your projects. It's particularly well suited for iterative local development like we saw at 17:09.840 --> 17:13.440 the beginning. Or for places where you can't easily run a full container on-time like in some 17:13.440 --> 17:19.520 CI systems. It's been tested with several popular registries. Please try it with your own favorite 17:19.520 --> 17:26.800 and let us know how it goes. But remember it's a tool kit and you can use it to build your own 17:26.800 --> 17:32.480 custom container tools designed to suit the way you work with containers. Take it apart but please do 17:32.480 --> 17:38.240 put it back together again and see what tricks you can come up with. I'll be available in the whole 17:38.240 --> 17:46.800 way after this if you'd like to talk about some of your ideas. Thank you very much.