WEBVTT 00:00.000 --> 00:11.000 The last presentation that's not listed for the day, generating for build stream projects 00:11.000 --> 00:15.000 yet another ecosystem by Abdrahim and the guest. 00:15.000 --> 00:17.000 Don't move your ears. 00:17.000 --> 00:18.000 Okay. 00:18.000 --> 00:22.000 There you are. 00:22.000 --> 00:25.000 So, hello everyone. 00:25.000 --> 00:26.000 I'm Abdrahim. 00:26.000 --> 00:28.000 I work at Co-Think. 00:28.000 --> 00:31.000 And this is Dom Michaelig. 00:31.000 --> 00:35.000 So, he worked with me on this project. 00:35.000 --> 00:40.000 So, who am I? 00:40.000 --> 00:43.000 I'm making of the Apache build stream project. 00:43.000 --> 00:56.000 The project management committee member and I've been a PMC the chair of the PMC since December. 00:56.000 --> 00:59.000 And I work at Co-Think. 00:59.000 --> 01:01.000 So, what is build stream? 01:01.000 --> 01:06.000 Some of you might not be aware of build stream. 01:06.000 --> 01:18.000 So, it's a powerful software integration tool that allows you to automate the integration of software components, including anything you want, 01:18.000 --> 01:25.000 from operating systems to a small application. 01:25.000 --> 01:28.000 I test the following advantages. 01:28.000 --> 01:31.000 Again, that's from the website. 01:31.000 --> 01:36.000 So, it has a declarative build instruction definition. 01:36.000 --> 01:42.000 In Neamo, it supports both the developer and integrative workflow. 01:42.000 --> 01:44.000 So, we will do the same. 01:44.000 --> 01:46.000 We will use the same tool for developers. 01:46.000 --> 01:48.000 You can open our workspace. 01:48.000 --> 01:51.000 That's what build stream does to work on the project. 01:51.000 --> 01:54.000 And then you can build the whole operating system. 01:54.000 --> 01:56.000 Imagine with it. 01:56.000 --> 01:59.000 It's fast and predictable. 02:00.000 --> 02:07.000 Especially because it isn't affected by what you have installed on your system. 02:07.000 --> 02:10.000 Everything is in a sandbox. 02:10.000 --> 02:11.000 It's extensible. 02:11.000 --> 02:15.000 You can write plugins for anything you want. 02:15.000 --> 02:18.000 We discuss the two kinds of plugins. 02:18.000 --> 02:23.000 And we can use it to bootstrap tool chains and operating systems. 02:23.000 --> 02:29.000 There's this project called free desktop SDK binary seed, which uses the live bootstrap. 02:29.000 --> 02:36.000 To bootstrap from a very small binary all the way to a whole tool chain. 02:36.000 --> 02:39.000 So, what does a minimal? 02:39.000 --> 02:43.000 I'll go ahead and explain a bit about the build stream concepts. 02:43.000 --> 02:51.000 Because I will need them because like when the previous presenter was talking about Swift product. 02:51.000 --> 02:54.000 And the package is. 02:54.000 --> 02:59.000 So, if you don't know the concept, it will be hard to follow. 02:59.000 --> 03:07.000 So, as a minimal build stream project, we'll have one or more import elements to import the base tools. 03:07.000 --> 03:10.000 So, that you have at least something in your sandbox. 03:10.000 --> 03:13.000 Because by default there's nothing. 03:13.000 --> 03:16.000 One or more build elements. 03:16.000 --> 03:20.000 So, using whatever build tool you want. 03:20.000 --> 03:24.000 So, there are plugins for auto tools, meson, cmake. 03:24.000 --> 03:32.000 And you could write things for plugins for any other build system. 03:32.000 --> 03:40.000 Then you have the compose element which takes multiple elements and puts them together. 03:40.000 --> 03:43.000 And excludes allows you to exclude files you don't need. 03:43.000 --> 03:48.000 For example, debug information or development files and headers. 03:48.000 --> 03:57.000 And then we have an element to package them all, depending on what you want to do. 03:57.000 --> 04:08.000 So, what you want to produce and always image the Docker image for whatever you want. 04:08.000 --> 04:13.000 I flatback for a flatback application if you're on Linux. 04:13.000 --> 04:21.000 And then we have a project you need a project that can file at the root of the directory. 04:21.000 --> 04:29.000 So, we'll try to, well, not a whole project, but at least an idea of how that would work. 04:29.000 --> 04:34.000 So, here we have the base image. 04:34.000 --> 04:42.000 I am importing a Docker image from the free desktop SDK project, which is a project build with build stream. 04:42.000 --> 04:53.000 And if you are building your thing with build stream, you generally use free desktop SDK. 04:53.000 --> 04:56.000 You could use it at the build stream level. 04:56.000 --> 04:58.000 This is what we call a junction. 04:58.000 --> 05:05.000 But here for simplicity, I'm just using the Docker image that's produced by free desktop SDK. 05:06.000 --> 05:11.000 So, the first line, the kind of the element. 05:11.000 --> 05:20.000 So, this is an import element, which just takes its sources and puts them as is into the artifact. 05:20.000 --> 05:29.000 Then we have the list of sources, and we have one single source of kind Docker, which downloads the Docker image. 05:29.000 --> 05:32.000 In this case, free desktop SDK slash SDK. 05:32.000 --> 05:36.000 We haven't specified the URL and so it's from Docker Hub. 05:36.000 --> 05:42.000 And then the graph here is the manifest digest of the image. 05:42.000 --> 05:49.000 So, it is exactly the image that you, you have to specify the image exactly. 05:49.000 --> 05:57.000 You can just give it a tag and ask it to pull whatever is the latest image in that tag. 05:57.000 --> 06:05.000 So, you have to use the exact shaft to 56 manifest digest. 06:05.000 --> 06:12.000 So, here is an example where I use the Docker image, but it could be a tarboard, a root FS tarboard, 06:12.000 --> 06:16.000 generated by the bootstrap or anything. 06:16.000 --> 06:19.000 It doesn't really depend on Docker on it. 06:19.000 --> 06:26.000 So, we can do the same thing with platform.bst. 06:26.000 --> 06:34.000 So, for free desktop SDK platform, which is, these are used at the flatback manifest. 06:34.000 --> 06:44.000 So, the SDK contains the compilers and things like that, and the platform only contains the runtime. 06:44.000 --> 06:47.000 Then we have built elements. 06:47.000 --> 06:53.000 For each element, we have, again, the kind, auto tools in this case. 06:53.000 --> 06:56.000 So, that's the plugin, in question. 06:56.000 --> 06:58.000 So, here we have the auto tools plugin. 06:58.000 --> 07:04.000 If we'd have Mason, C, Mac, water. 07:04.000 --> 07:09.000 The sources here I'm using GNU hello. 07:09.000 --> 07:11.000 So, it's a tarboard. 07:11.000 --> 07:18.000 So, kind of tar, the URL and the share to 56 of the tarboard. 07:18.000 --> 07:23.000 And I have built dependencies and runtime dependencies. 07:23.000 --> 07:34.000 Here, I'm using the platform and SDK that we imported before, but you could depend on other build elements that you have built it. 07:34.000 --> 07:36.000 Then the compose element. 07:36.000 --> 07:39.000 In this case, it's not really useful. 07:39.000 --> 07:45.000 GNU hello doesn't really have much tons of development files. 07:45.000 --> 07:52.000 And we didn't add instructions on how to split the debugging formation. 07:52.000 --> 07:56.000 So, it won't be excluded because it's integrated in the binary. 07:56.000 --> 08:02.000 But this is how you would do it in general. 08:02.000 --> 08:08.000 And then we can have another element to package the result in whatever you want. 08:08.000 --> 08:15.000 And OCI image, this image, flat back, whatever. 08:15.000 --> 08:20.000 Yeah, this is something that I said. 08:20.000 --> 08:24.000 But there is an OCI plugin that is unmaintained. 08:24.000 --> 08:31.000 But there is script element, which is a core plugin in build stream, which allows you to run any command you want. 08:31.000 --> 08:40.000 So as long as you have a tool that can generate OCI images that doesn't require privileges, 08:40.000 --> 08:47.000 so it can run in the sandbox, you can use it. 08:47.000 --> 08:52.000 And then the project.com, not much to see here. 08:52.000 --> 08:59.000 The name of the project, the minimum build stream version needed and the plugins. 09:00.000 --> 09:10.000 In this case, we've used O2 tools and Docker, which are part of the build stream plugins collection, which is a collection of plugins, 09:10.000 --> 09:15.000 made by the build stream project. 09:15.000 --> 09:28.000 We haven't, you may notice that we haven't declared all other plugins, like the tar, the import, the script we haven't released it. 09:28.000 --> 09:39.000 And these are core plugins. They are part of build stream, then we have build stream plugins, which is a collection of plugins, 09:39.000 --> 09:43.000 made by the build stream team. 09:43.000 --> 09:52.000 Then we have another project build stream plugins community, which contains community plugins. 09:52.000 --> 10:03.000 So what we can see here is that build stream should know everything about all the sources that get into the construction of whatever artifact we have. 10:03.000 --> 10:13.000 So there is no, nothing on the system that build stream doesn't know it, that affects the artifact. 10:13.000 --> 10:20.000 So it should be able to generate a complete as one, right? 10:20.000 --> 10:30.000 So there are a few challenges here. 10:30.000 --> 10:37.000 So the first one, just knowing all the tar balls, the Docker images and the Git repository, 10:37.000 --> 10:46.000 commit that get into the build isn't enough, because people expect more information of the BS1. 10:46.000 --> 10:58.000 For example, version numbers, dependencies between various components, licenses, and many other things. 10:58.000 --> 11:04.000 Some of them, most of them are considered optional, but... 11:04.000 --> 11:10.000 But later, someone may consider something to be very important. 11:10.000 --> 11:14.000 For example, the license is sometimes very important. 11:14.000 --> 11:19.000 So we can't really say that optional. 11:19.000 --> 11:28.000 So the solution you have implemented with build stream 2.5 is to allow the source plugins, 11:28.000 --> 11:40.000 so like the Docker and tar in these examples, there are other source plugins for Git for things like that. 11:40.000 --> 11:55.000 We asked them to provide this information about sources, rather than having a single tool that tries to know everything every possible plugin. 11:55.000 --> 12:07.000 So build stream can also ask the plugins for this information, and we can display it as Yamu, which can be consumed by other tools. 12:07.000 --> 12:17.000 So here's an example. So this is coming from the Git repo plugin, so that's the kind. 12:17.000 --> 12:29.000 It gives us the URL, the medium, it says that it's a Git repository, and this version is a commit, so the commit hash. 12:29.000 --> 12:40.000 Then it has a version guess, 2.42, and it's allowed to give more information. 12:40.000 --> 12:49.000 And in this case, it gave us that it guessed the version from the tag that's called GLipC2.42. 12:49.000 --> 12:58.000 But there are 48 commits ahead of that tag. 12:59.000 --> 13:09.000 That's some information that the plugin can give us, we can try to make it into an aspa. 13:09.000 --> 13:26.000 So this is using this, where we developed a tool called build stream aspa, that you're on inside your build stream project, and that can generate an SPDX format aspa using that information. 13:27.000 --> 13:44.000 Something to add here is that the dependency information is was already available from build stream, so this is more information that we can get, and that we can use to generate an aspa. 13:44.000 --> 13:56.000 Another thing that's, I'd say in progress, we allow the author of the build stream project to add more information to the sources. 13:56.000 --> 14:05.000 So whenever we have with the clear source, like Git repository or a tarball, we can add some more information here. 14:05.000 --> 14:27.000 For now there is this homepage and issue tracker, but because that's something that came up, but we are working on making this more generic and allowing a lot more information to be added by the project author. 14:27.000 --> 14:31.000 We can come back to it. 14:31.000 --> 14:38.000 So this comes into the, this gets added to the source information. 14:38.000 --> 14:43.000 That's another example, but yeah. 14:43.000 --> 14:54.000 Another challenge we faced is marking the build stream concepts of elements and sources to SPDX. 14:54.000 --> 15:04.000 So SPDX has packages files and snippets, I'm not sure exactly what snippets are, but at least we have packages and files. 15:04.000 --> 15:14.000 It looks, yeah, and build stream is not a package manager, so we don't really have packages. 15:14.000 --> 15:24.000 So what we could do, the closest I could come up with is mapping elements and sources to packages. 15:24.000 --> 15:43.000 So each source is a package, each element is a package, and then we add the relationships generated from to link elements to their sources and the depends on to link the dependencies between the elements. 15:43.000 --> 15:51.000 Yeah, but not all sources are packages. 15:51.000 --> 16:02.000 We have sources, anything that gets into the build has to come in via a source and even things like patches. 16:02.000 --> 16:10.000 I take this star ball and I patch it, that both of these are a tar source and then a patch source. 16:10.000 --> 16:20.000 And this is something we hit recently is that these patches, they're not really packages. 16:20.000 --> 16:28.000 Yeah, I don't know, they should be. 16:28.000 --> 16:34.000 Yeah, it's a single file, this case. 16:34.000 --> 16:44.000 Yeah. 16:44.000 --> 16:50.000 Yeah, that's all, thank you for. 16:50.000 --> 17:00.000 And yeah, I would like to open the floor for comments, suggestions, questions, whatever you may. 17:01.000 --> 17:07.000 It's a good lab, yeah. 17:07.000 --> 17:18.000 It's a good lab, it's a good lab, so good lab dot com, build slash build stream, slash build stream as well. 17:18.000 --> 17:27.000 Thank you, thank you, great. 17:27.000 --> 17:39.000 So basically what you're doing, there is similar, just take little line in the clear and get it like fish. 17:39.000 --> 17:50.000 So the data that you're collecting there is essentially what goes into us as a 17:50.000 --> 17:54.000 statistician to generate real provenance. 17:54.000 --> 17:58.000 Do you support that? Have you planned on doing that? 17:58.000 --> 18:02.000 We don't have a plan to do that, but... 18:02.000 --> 18:04.000 Yeah, exactly. 18:04.000 --> 18:09.000 Yeah, so because the data is the real data station, it's exactly what it does. 18:09.000 --> 18:14.000 All of the inputs, and then the formula that you applied to them, which you've already 18:14.000 --> 18:16.000 had them as well. 18:24.000 --> 18:28.000 Can you go back a couple slides to when I was sort of about to say something? 18:28.000 --> 18:33.000 So if you go back to that one. 18:33.000 --> 18:35.000 Yeah. 18:35.000 --> 18:41.000 Are there things that you think we should make sure we can capture for the provenance? 18:41.000 --> 18:47.000 Like, I'm sort of wondering, like, vulnerability, notifications, things that so they can get populated and 18:47.000 --> 18:53.000 instantiated into these S-bombs for contacts and things like that as part of the agent stuff. 18:53.000 --> 18:57.000 Yeah, the security, like, the security relevant information. 18:57.000 --> 18:58.000 Yeah. 18:58.000 --> 18:59.000 Yeah. 18:59.000 --> 19:06.000 So one thing that we definitely want to have regarding security is the ability to map this source 19:06.000 --> 19:10.000 to a P-O-R-L or a C-P-E. 19:10.000 --> 19:11.000 Yeah. 19:11.000 --> 19:17.000 So that we can use the output S-bombs to pass it to another tool to extract vulnerabilities, 19:17.000 --> 19:23.000 which currently working on using, on making sure that our S-bombs can be consumed by C-B binary tool. 19:24.000 --> 19:25.000 Okay. 19:25.000 --> 19:28.000 And that's, that is our plan regarding that. 19:28.000 --> 19:29.000 Okay. 19:29.000 --> 19:30.000 Yeah. 19:30.000 --> 19:32.000 I'm just sort of thinking that since you're actually taking in might, pulling in some of the data, 19:32.000 --> 19:38.000 there may be some things to enrich the quality of the final S-bombs that you've 19:38.000 --> 19:43.000 been captured in a standard way and then set up a precedent for other projects to do. 19:43.000 --> 19:44.000 Yeah. 19:44.000 --> 19:46.000 We are both C-B's. 19:46.000 --> 19:47.000 Yeah. 19:47.000 --> 19:49.000 Yeah. 19:49.000 --> 19:50.000 Yeah. 19:50.000 --> 19:52.000 What's the motivation for doing this? 19:53.000 --> 19:56.000 Never, I'm not know what build stream is. 19:56.000 --> 19:57.000 Yeah. 19:57.000 --> 19:59.000 But what's the motivation? 20:05.000 --> 20:09.000 What's the motivation of having S-bombs anyway? 20:09.000 --> 20:10.000 Yeah. 20:10.000 --> 20:11.000 Yeah. 20:11.000 --> 20:12.000 Yeah. 20:12.000 --> 20:13.000 I'm fine. 20:13.000 --> 20:14.000 I'm fine. 20:14.000 --> 20:15.000 I've got some pain. 20:15.000 --> 20:17.000 That's absolutely amazing. 20:17.000 --> 20:18.000 Okay. 20:18.000 --> 20:19.000 Okay. 20:20.000 --> 20:22.000 That's as important as it is. 20:22.000 --> 20:23.000 Yeah. 20:23.000 --> 20:24.000 Thank you. 20:24.000 --> 20:25.000 I think. 20:25.000 --> 20:29.000 I was just wanting to reply to the answer to that in terms of both spaces. 20:29.000 --> 20:34.000 So actually, I mean, I'm in the middle, but I also think that using those three, 20:34.000 --> 20:37.000 I mean, I do some of these and all of these for our customers. 20:37.000 --> 20:43.000 So what we're doing is, is we're then embedded devices where we might be using 20:43.000 --> 20:48.000 courses of images for these and we need and not customers. 20:48.000 --> 20:49.000 Yeah. 20:49.000 --> 20:56.000 We need to have all these information, but doing this is such a way that they can do this 20:56.000 --> 20:58.000 of a single Python. 20:58.000 --> 21:04.000 So in build stream, essentially, as I'm there again, as a speaker, laid out this, 21:04.000 --> 21:07.000 you can put all your sources in that. 21:07.000 --> 21:12.000 And you can use all your videos using a single system image of single source of truth. 21:12.000 --> 21:17.000 So that for this tool, it can be very important because ultimately, you can also 21:17.000 --> 21:23.000 just take that files and to use high quality assformers. 21:23.000 --> 21:26.000 You know, I was, I had a player being in the track earlier. 21:26.000 --> 21:29.000 Today, here I'll be added to that. 21:29.000 --> 21:31.000 This is one. 21:31.000 --> 21:34.000 What do you summarize that for the mic? 21:34.000 --> 21:35.000 Yeah. 21:35.000 --> 21:38.000 That's a good thing long. 21:38.000 --> 21:39.000 Okay. 21:39.000 --> 21:40.000 Okay. 21:40.000 --> 21:46.000 So the comment was that, yeah, people actually use that to actually produce images. 21:47.000 --> 21:54.000 And having a single whole build set up that can also generate complete 21:54.000 --> 21:59.000 as an accurate response, it's definitely a plus, right? 21:59.000 --> 22:02.000 All right. 22:02.000 --> 22:03.000 Let's thank again. 22:03.000 --> 22:04.000 The speakers. 22:04.000 --> 22:05.000 Thank you.