WEBVTT 00:00.000 --> 00:13.600 Around of a glass, please, for Paul Meier, who's going to talk about, go in the next ecosystem 00:13.600 --> 00:21.000 and some interesting work he did in the area. 00:22.000 --> 00:27.000 Welcome to my talk. I'm Paul. My guitar panel is cut X or HIN. 00:27.000 --> 00:31.000 I'm also a master-done, I'm a security software engineer. 00:31.000 --> 00:36.000 The executors maintainer and I'm also part of the next package of the Scouting. 00:36.000 --> 00:40.000 So if you're wondering why you didn't receive the latest GoVinability update, 00:40.000 --> 00:46.000 I would tell you that it was currently in staging next and you have to wait or apply an overlay. 00:47.000 --> 00:53.000 So I want to quickly give another view of the GoVinability Penance Management in general, 00:53.000 --> 01:00.000 outside of NICS, go and manage dependencies as modules, which is the collection of packages, 01:00.000 --> 01:06.000 that is released versions and distributed together. So this is usually a repository scope. 01:06.000 --> 01:13.000 We have to go that not file that declares minimum required versions of module dependencies. 01:13.000 --> 01:19.000 Let's also go that some file that contains cryptographic hashes for these dependencies, 01:19.000 --> 01:25.000 but to go that some file is not an exact log file, and also the hashes in there are incompatible with NICS, 01:25.000 --> 01:29.000 so we cannot use import from the abbreviation here. 01:29.000 --> 01:40.000 And then you go use this minimum version selection to, which is a deterministic algorithm to select the actual versions that I used in the build list. 01:41.000 --> 01:46.000 And only at the one version of a dependency is selected in the end. 01:46.000 --> 01:51.000 And use a quick look at the GoVinability Mod file. 01:51.000 --> 01:57.000 Here just to get an impression, the required statements declare or direct, 01:57.000 --> 02:02.000 and at the bottom indirect, so transitive dependencies. 02:03.000 --> 02:10.000 So now come to the NICS part, how do we build our package, go, stuff in NICS. 02:10.000 --> 02:15.000 In NICS packages, we have a single build up of this called Big Go module. 02:15.000 --> 02:22.000 And yeah, for most of this might be straightforward, we have a source that we fetch from GitHub, 02:22.000 --> 02:28.000 and we have a package name and version, and what I really want to point out here is the vendor hash at the bottom, 02:28.000 --> 02:32.000 which pins our dependencies. 02:32.000 --> 02:38.000 So we will take a closer look at this, and also at the delay, the relations structure here. 02:38.000 --> 02:46.000 So we have one derivation of which is Big Go module, the thing we want to build, then we have the source, 02:46.000 --> 02:51.000 which is a fixed operating relation that pulls our source from GitHub. 02:51.000 --> 02:57.000 And then we have a third derivation here called GoModules. 02:57.000 --> 03:09.000 And this depends on the source, and use the GoMod file and the Go command to download the dependencies of the model we want to build, 03:09.000 --> 03:18.000 interface output derivation, and then this is pinned with a by the vendor hash. 03:18.000 --> 03:28.000 So if you now look at two different Go packages, pet and pin act, both will have GoModules, derivation they depend on, 03:28.000 --> 03:33.000 and yeah both will have some Go dependencies in there. 03:33.000 --> 03:39.000 And as we see here, now these two packages depend on the same GoModules. 03:39.000 --> 03:47.000 We have the same dependencies, but in next, we do not have any introspection in each of this. 03:47.000 --> 04:00.000 So we will download the same dependency again, and yeah, we do not know that these two fixed operating relations actually have the same content. 04:01.000 --> 04:11.000 So what does this mean for the current status and next message from GoModule, we have to download things multiple times. 04:11.000 --> 04:15.000 We are storing these dependencies over and over again. 04:15.000 --> 04:19.000 Very important, we don't have any form of caching. 04:19.000 --> 04:28.000 So we have to rebuild the dependencies over and over again, both between two different packages, 04:28.000 --> 04:34.000 and also in case we change something about our actual package, we want to build. 04:34.000 --> 04:43.000 There's no caching for the build of the dependencies, so we also have to rebuild our dependencies in this case. 04:43.000 --> 04:49.000 Then we do not have any introspection into the dependencies in nix. 04:49.000 --> 05:00.000 It is really hard to update dependencies of the package, for this we would actually need to create the patch for the GoMod file, 05:00.000 --> 05:05.000 and we would have to do this for each GoPacket individually. 05:05.000 --> 05:12.000 And following, we do not have an easy way to patch vulnerabilities. 05:12.000 --> 05:17.000 So first I want to look at the vulnerability aspect, how bad is this. 05:17.000 --> 05:21.000 I've created a small project, GoModule check nix packages. 05:21.000 --> 05:27.000 GoModule check is the official vulnerability scanning tool of GoModule. 05:27.000 --> 05:37.000 And it has, go has a separate vulnerability database containing CVEs, but also GoModule. 05:37.000 --> 05:47.000 So the vulnerabilities that are not assigned to CVE, GoModule check is quite cool because it checks if the vulnerable code is actually reachable. 05:47.000 --> 06:00.000 So you get less false positive in case you would just import a vulnerable package, but the isn't a core path that will make your code reach the vulnerability. 06:00.000 --> 06:13.000 And then GoModule check nix packages runs GoModule check on the source of all Go packages and nix packages that are built with GoModule. 06:13.000 --> 06:21.000 So what I have the results here, and black we see the packages that are discovered by GoModule check nix packages. 06:21.000 --> 06:26.000 The discovery is currently based on simple attributes of the output. 06:26.000 --> 06:32.000 Then in blue we have the packages that are actually scan tool successfully. 06:32.000 --> 06:41.000 This is just like a really simple project, and for example for packages that you see go, we do not provide the CD dependencies. 06:41.000 --> 06:50.000 So the vulnerability check I will just fade because the dependencies aren't there, so we are missing out on some of those. 06:50.000 --> 07:01.000 And then in red we have the number of vulnerable packages which is quite stable around 30% of the scan packages. 07:01.000 --> 07:06.000 And then in yellow we have the number of vulnerabilities in total. 07:06.000 --> 07:13.000 We have like around three vulnerabilities per vulnerable package. 07:13.000 --> 07:25.000 And yeah just remember we have we don't have like an easy way to fix vulnerabilities, especially not at this scale. 07:25.000 --> 07:33.000 So now I want to to make a jump to the catching problem again and just yeah. 07:33.000 --> 07:53.000 How the cache usually works with go is we have like an environment variable GoCash which points to a five system path and there's our cache located and go will read and write to this path and manage the cache itself. 07:53.000 --> 08:05.000 Really cool feature in go124 and also in earlier versions as experimental feature that lets us provide binary called goCash Proc. 08:05.000 --> 08:10.000 And yeah this gives us a lot more flexibility. 08:10.000 --> 08:16.000 Go will just invoke the binary during the bit presses and talk a JSON based protocol. 08:16.000 --> 08:31.000 He has done that in standard out with two basic operations get to request a cache entry via an external idea and then in case we got the cache miss go where built the package and then called put. 08:31.000 --> 08:38.000 So the go cache port can can write the cache result somewhere. 08:38.000 --> 08:48.000 Now at this point I want to introduce goboot.next which is next generation build up for go. 08:48.000 --> 09:00.000 I use this goCash Proc and just so you get an idea the goboot.next tries to provide the package sets of go dependencies. 09:00.000 --> 09:12.000 Where each go module has its own derivation so we will try to module the go dependency graph index. 09:12.000 --> 09:18.000 And first I want to show how goboot.next uses goCash Proc. 09:18.000 --> 09:26.000 So we have written a goboot.next cache binary that implements goCash Proc. 09:26.000 --> 09:35.000 In the way this works is we have an environment variable where we read a number of paths from. 09:36.000 --> 09:49.000 And then go the cache over will serve the cache entry is found at these paths and the pass will point to the next door. 09:49.000 --> 10:00.000 And in case we have a cache miss the cache address writes the built result into a defined output. 10:00.000 --> 10:12.000 And yeah we can add the stop paths we set up hook so we put the go module dependencies into our built input. 10:12.000 --> 10:22.000 The set up hook will add the stop path to the environment and then the go cache the cache can serve these. 10:22.000 --> 10:31.000 Now looking at this from a derivation level so if we want to build getup.com full slash a. 10:31.000 --> 10:39.000 We have a source in green then we have the source of our dependencies which is required for the build process. 10:39.000 --> 11:01.000 And we also provide the cache so the build output of when we build the dependencies to the cache. 11:01.000 --> 11:08.000 And now the idea is that we will not have to rebuild that the dependencies because they are already cached. 11:08.000 --> 11:18.000 The cache will serve the build cache for for the dependencies and we will only need to rebuild the packages that belong to full slash a. 11:18.000 --> 11:24.000 And for these packages the output of the build process is written to out. 11:24.000 --> 11:33.000 And then the final bit result will contain the cache entries for full slash a. 11:33.000 --> 11:38.000 And looking at this at the package set perspective. 11:38.000 --> 11:46.000 These packages will construct the package set and mainly consisting of the build results. 11:46.000 --> 11:52.000 The cache entries that the cache are stored in the output. 11:52.000 --> 12:01.000 And also quite important is that we have to have the source of our transitive dependencies available for build. 12:01.000 --> 12:11.000 And if we now want to build a go binary we would do that outside of the package set and just add all the built inputs. 12:11.000 --> 12:24.000 As usual so the go modules as built and put and we would have all the dependencies already available and also built. 12:24.000 --> 12:30.000 And yeah every dependencies only available in one single version in this package set. 12:30.000 --> 12:40.000 So we are doing like some overwriting of the of the version where where package depends on. 12:41.000 --> 12:50.000 So what we get here is we get increments as far as fetching we can fetch every dependency on it's own we get rid of this vendor hash. 12:50.000 --> 12:58.000 Also we get rid of the of the bad development workflow that comes with that. 12:58.000 --> 13:01.000 We get increments to build. 13:02.000 --> 13:07.000 For a build process because the dependencies are already cursed. 13:07.000 --> 13:13.000 We get great composability because we're using set up hooks. 13:13.000 --> 13:21.000 We get the possibility to patch vulnerabilities because we only have dependency once in our set and we can then bump that. 13:21.000 --> 13:26.000 And it will propagate to to all the dependence. 13:26.000 --> 13:28.000 That's really cool. 13:28.000 --> 13:32.000 This project targets next packages. 13:32.000 --> 13:38.000 The status is highly experimental it's not usable yet. 13:38.000 --> 13:46.000 I did some research and I figured we will have to have like 12,000 go dependencies or so next packages. 13:46.000 --> 13:54.000 We've got packages to build all the other packages that are currently in next packages. 13:54.000 --> 14:00.000 But now we are like about 30 packages that we are building. 14:00.000 --> 14:03.000 So there's let's do a lot of work. 14:03.000 --> 14:10.000 And we still are figuring out like how can this this work at all. 14:10.000 --> 14:12.000 Yeah and it's a lot of work. 14:12.000 --> 14:15.000 I'm looking for people to collaborate on this. 14:15.000 --> 14:22.000 Both for the the builder the new builder but also for the vulnerability research thingy. 14:22.000 --> 14:24.000 Please talk to me. 14:24.000 --> 14:30.000 You can reach me on networks or here and person. 14:30.000 --> 14:35.000 And then I want to give shout out to all go related next projects. 14:35.000 --> 14:41.000 And I want to point out especially the one at the bottom which is quite new. 14:41.000 --> 14:50.000 And which uses the same go cash product mechanism to serve the go cash from outside the sandbox. 14:50.000 --> 14:56.000 So this introduces some impurity but it relies on go to ensure the correctness of the build. 14:56.000 --> 15:01.000 So yeah that's something that might be usable today. 15:01.000 --> 15:03.000 Yeah thanks a lot. 15:09.000 --> 15:11.000 Thank you very much for the great talk. 15:11.000 --> 15:14.000 We have a lot of time available for questions. 15:14.000 --> 15:17.000 Are there any questions? 15:17.000 --> 15:20.000 I see your hand back there. 15:28.000 --> 15:33.000 So the question is if this still uses the go.mod file. 15:33.000 --> 15:35.000 And the answer is now it does not. 15:35.000 --> 15:43.000 You have to add the build dependencies as built inputs in nx in your next derivation. 15:43.000 --> 15:48.000 And we will also rewrite the go.mod file. 15:48.000 --> 15:54.000 So because if you as I said there's only like one version available. 15:54.000 --> 16:02.000 And this means that when there's like a diff in the version you have in your go.mod file and the next packages. 16:02.000 --> 16:06.000 There could be like 250 dependencies that have changed. 16:06.000 --> 16:11.000 So yeah we rewriting the go.mod file completely at this point. 16:12.000 --> 16:14.000 There is another question here in the file. 16:14.000 --> 16:16.000 I think a format of the question. 16:16.000 --> 16:22.000 Is there a kind of equivalent log file that you already generated? 16:22.000 --> 16:23.000 And next. 16:23.000 --> 16:31.000 Because you have to manually add all the dependencies to your build file or do your query. 16:31.000 --> 16:37.000 Generate a log file like a log file with the idea is that you do not. 16:37.000 --> 16:41.000 And the question is if you create some kind of log file for this. 16:41.000 --> 16:48.000 And then so it's now that here is that first you have your dependencies available in nx packages. 16:48.000 --> 16:52.000 So you just have to add them at this build input. 16:52.000 --> 16:56.000 And then they are pinned as we fetch the source. 16:56.000 --> 16:59.000 And at this point they are pinned. 16:59.000 --> 17:01.000 So quite a lot of work initially. 17:01.000 --> 17:03.000 Yes this card is. 17:03.000 --> 17:05.000 This is a lot of work. 17:05.000 --> 17:06.000 No. 17:06.000 --> 17:07.000 No. 17:07.000 --> 17:09.000 I don't have better value. 17:09.000 --> 17:12.000 But you may automatically respond to a go.mod format. 17:12.000 --> 17:15.000 You can kind of have a tool that takes the load file. 17:15.000 --> 17:20.000 That gives you the next array of the dependencies that you have done. 17:20.000 --> 17:22.000 You can make it. 17:22.000 --> 17:23.000 Yes. 17:23.000 --> 17:25.000 So the ideas that this could be. 17:25.000 --> 17:27.000 There's like a lot of room for automation. 17:27.000 --> 17:29.000 And sure there is. 17:29.000 --> 17:32.000 I don't think we will have to. 17:32.000 --> 17:36.000 Yeah package all these things by hand. 17:36.000 --> 17:39.000 Especially as there are a lot of libraries that are. 17:39.000 --> 17:41.000 We do not have like any. 17:41.000 --> 17:44.000 Any build input that all or we can use the inputs. 17:44.000 --> 17:45.000 The cloud in the go. 17:45.000 --> 17:46.000 Not file. 17:46.000 --> 17:47.000 Yeah. 17:47.000 --> 17:49.000 But we're not there right now. 17:49.000 --> 17:51.000 Quick in between question is Josh. 17:51.000 --> 17:55.000 Can you hear who's going to be the speaker for the next talk? 17:55.000 --> 17:56.000 Okay. 17:56.000 --> 17:57.000 Okay. 17:57.000 --> 17:58.000 No. 17:58.000 --> 17:59.000 Okay. 18:00.000 --> 18:02.000 Questions about the action. 18:06.000 --> 18:07.000 No. 18:07.000 --> 18:08.000 No questions. 18:08.000 --> 18:11.000 Well, then thank you very much. 18:11.000 --> 18:13.000 Thank you for finishing on time. 18:13.000 --> 18:14.000 Please. 18:14.000 --> 18:15.000 Thank you.