WEBVTT 00:00.000 --> 00:13.840 Okay, so no introductions, okay, this is distributing Rustian RPMs relative for fun, relatively 00:13.840 --> 00:19.440 speaking and profit, and Fabio, my nickname basically everywhere is Dekla Torps, so if you 00:19.440 --> 00:23.960 want to find me online that's what you need to search for. 00:24.760 --> 00:30.360 Short introduction for myself, I've been for the raw packages since 2016, submitted my first 00:30.360 --> 00:35.800 Rust package to Fedora in 2020, and became the primary maintain of the tools we use for Rust 00:35.800 --> 00:44.360 packaging in about 2022. I've been a packaging committee member since 2018, and engineering 00:44.360 --> 00:52.360 steering committee member since 2019. I think I'm going to look at my screen because that 00:52.360 --> 01:02.840 one isn't flickering. Okay, so what's this talking about? Basically kind of answering the 01:02.840 --> 01:09.400 question that's been floating around in this space for a while now is basically our traditional 01:09.400 --> 01:16.120 package managers becoming obsolete, and I think I'm going to give a short answer to that at the end of 01:16.120 --> 01:25.240 the talk. So for a quick background for people who don't know Rust, which shouldn't be in that 01:25.240 --> 01:32.680 many panel, the compiled language based on LLVM, typically everything is statically linked 01:32.680 --> 01:39.400 because there's no stable ABI, which may be coming in the future, but not yet. There's heavy 01:39.400 --> 01:45.560 usage of features like compile time code generation, meta programming with macros or build scripts 01:45.640 --> 01:52.280 and stuff like conditional compilation. You also have an official package manager and build 01:52.280 --> 01:59.720 system and official package registry, which is cargo and crates.io, which means one of the benefits 01:59.720 --> 02:05.160 of using Rust is that package management and dependency management is really easy, but that also 02:05.160 --> 02:13.560 makes it really easy to add lots of dependencies to your projects. And on the codora side, 02:14.520 --> 02:19.640 we have basically a hardware requirement to build everything from source. We have a hardware 02:19.640 --> 02:25.800 requirement for compliance with the license terms, of course. There is a strong preference not to 02:25.800 --> 02:33.080 bundle or vendor dependencies. We have a preference for running test speeds if that's possible. 02:33.720 --> 02:39.800 And of course there are like incentives to provide good system integration, which for example for 02:39.880 --> 02:44.360 command line applications that's like providing shell completions and manual pages and stuff like that. 02:47.240 --> 02:52.440 The approach we use for libraries is that we create packages for Rust crates, 02:52.440 --> 02:59.160 that only create contained a crate sources and no compiled artifacts. Again, because no stable ABI. 02:59.960 --> 03:06.760 And even if it had a stable ABI, shipping the compiled artifacts wouldn't really be feasible 03:06.760 --> 03:11.560 because you have a lot of combinations for all the feature flags you can enable for those crates. 03:12.360 --> 03:17.960 And you basically get a combinatorial explosion for like a package that has 10 feature flags 03:17.960 --> 03:26.440 you need to ship to the 10, which is a 124 different compiled artifacts, which just blows up 03:26.440 --> 03:33.640 and not really possible. And basically the philosophy we have for packaging Rust applications 03:34.280 --> 03:40.360 is to make installing it from the package manager provide something that's as close to installing it 03:40.360 --> 03:49.320 from the cargo registry, basically as close to possible as to that or if it can provide something 03:49.320 --> 03:59.480 better. So what are the things that we can do better than that? Let's take it. Oh my god, that's really 03:59.640 --> 04:11.160 distracting. I hope it's not the cable. Okay. Okay, how does installation application installation 04:11.160 --> 04:17.560 work when you use cargo? I only support installing applications by combining them from source. 04:17.560 --> 04:23.000 So it requires users to have a local Rust tool chain installed, which is like cargo, 04:23.000 --> 04:29.320 Rusty and LLVM, which is quite a lot of stuff. If your application has any external dependencies 04:29.320 --> 04:37.800 from like CLI per race, you also need to install those. And at the end of that 10 cargo install, 04:37.800 --> 04:45.000 also only supports installing executables into your own users path. So it only installs 04:45.000 --> 04:50.040 executables, it only installs them for the local user. It cannot handle any other files except 04:50.040 --> 04:58.360 those executables. So basically in the Rust ecosystem, some kind of workaround for those limitations 04:58.360 --> 05:07.160 have been developed and have developed over time. Some of them are usually considered bad 05:07.160 --> 05:15.960 practice like limitations of cargo build that incentivize bundling CLI per race or 05:17.240 --> 05:22.920 statically linking those CLI per race and some creates even ship prebuild object files, 05:23.000 --> 05:32.280 which is like, don't please don't. Because doing that means you don't have your users installed 05:32.280 --> 05:36.920 those libraries on a system wide level. Like if you can bundle like a statically, 05:38.040 --> 05:43.720 you can bundle SQLite sources, then it don't need this SQLite development 05:43.720 --> 05:52.280 had as available on the host system. And like because cargo install only installs executables, 05:52.360 --> 05:59.400 that kind of made projects adopt the self-contained executable approach. Basically bundling 05:59.400 --> 06:05.560 all the data files and stuff you they need into the executable itself because there's no 06:05.560 --> 06:12.360 not really a good support for loading them from disk because you can't install them to disk 06:12.360 --> 06:20.520 with the default installation mechanism. But the other hand installing applications with RPM looks 06:20.600 --> 06:25.000 quite different. Everything is integrated with the system package manager. The packages are 06:25.640 --> 06:31.240 when you get them from the system like Fedora. They're built on a trusted environment. They're 06:31.240 --> 06:37.960 cryptographically signed. When you install them, you don't need a rust tool chain. Any 06:37.960 --> 06:43.560 development had as available on your local system and the packages can actually ship support files 06:43.640 --> 06:50.760 for good system integration. As an example for command line application, it's usually 06:52.280 --> 06:57.400 preferable for users to have shell completions manual pages, config files, data files, 06:57.400 --> 07:05.160 whatever you need for the application to work correctly. Actually you also get like a smaller 07:05.160 --> 07:12.520 footprint for the built application both at in-memory and on disk because they're like dynamically 07:13.160 --> 07:20.360 C and C++ libraries. For example, SQLite is de-duplicated both on disk and in-memory 07:20.360 --> 07:26.040 and it also disentangles the update cycles between like those libraries and the application. 07:26.040 --> 07:31.640 So for example, if there's a security vulnerability in SQLite, it won't need to update 07:31.640 --> 07:37.240 a bundle SQLite in every rust application that includes it. But you can just wait for the system 07:37.240 --> 07:40.680 copy to be updated and you don't need to do anything for the rust application. 07:43.400 --> 07:48.840 Okay, so you installed the application. Now you want to update it. How does that work with cargo? 07:49.560 --> 07:56.840 Short answer is, it doesn't. It always requires user interaction. So basically running cargo 07:56.840 --> 08:03.320 install again for all applications you have on your system. There's no mechanism to notify users of 08:03.400 --> 08:09.480 available updates, no mechanism to put security updates and sometimes rerunning cargo install 08:09.480 --> 08:16.120 isn't even enough. Because by default, it does nothing if the application version hasn't changed. 08:16.680 --> 08:23.800 So if you want to pull in like a security fix from one of your applications, transitive library dependencies, 08:24.200 --> 08:31.160 you need to force it to read download all the library dependencies and compile the application 08:31.480 --> 08:37.880 even though the application version itself hasn't changed. And that's assuming that the security fix 08:37.880 --> 08:43.480 is published in the library version that's compatible with the dependency specifications in that dependency 08:43.480 --> 08:51.400 tree. So if that's a separate breaking version, then even passing the force flag to cargo install 08:51.400 --> 09:00.520 won't help you. On the other hand, package updates in RPM. Again, integrated with system package 09:00.520 --> 09:07.160 manager support notifying users of available updates and security updates. And in some cases, 09:07.160 --> 09:12.200 there's even automatic installation of critical updates. For example, on servers, which just 09:13.160 --> 09:20.440 systems like the NF automatic or on image based systems, which also have started to apply 09:20.520 --> 09:26.440 system updates automatically. And that also decouples application and library depends 09:27.160 --> 09:37.960 dependency update cycles mostly. Bonus points for using installing rust applications from 09:37.960 --> 09:44.520 RPM packages, we actually provide much better test coverage for those applications and for 09:44.840 --> 09:53.160 libraries. And the package builds run that test treats in an environment that actually matches the 09:53.160 --> 09:59.400 environment that the users are going to install those applications in. So test treats is run on Linux. 09:59.400 --> 10:04.920 It's run on the same fit or a version that the package is going to be installed on. It's going to run 10:04.920 --> 10:10.600 the test against the same library versions and against on the same CPU architecture that you're 10:10.680 --> 10:16.760 going to install the package on. And by doing that, we can also test coverage for environments 10:16.760 --> 10:25.720 that are usually not available like in CI environments like PowerPC or system C. And in 10:25.720 --> 10:33.320 overall, we also have a kind of package CI, which continuously rebuild packages against new versions 10:33.320 --> 10:39.880 of their dependencies. And that way, we can actually catch regressions quite quickly, usually. 10:40.840 --> 10:49.080 Okay, so you might ask yourself how does this work with packages that use mental dependencies? 10:51.400 --> 10:55.720 Because the default approach is to not do that, but how does it work if you do? 10:58.520 --> 11:04.120 I mean, the pro basically for rendering dependencies is that you can quickly get to a building 11:04.200 --> 11:11.400 and working package. And because you can mostly avoid dealing with dependencies. And on the 11:11.400 --> 11:18.280 negative side, it's often quite difficult to actually get from a working package to a package that 11:18.280 --> 11:24.120 would be acceptable for inclusion in the package repositories. Because you cannot actually avoid 11:24.120 --> 11:29.560 dealing with those dependencies. You can just put them in a box and say, I don't want to deal with this. 11:30.040 --> 11:37.720 It would be nice if you could do that, but yeah. So some of the problems that that happen 11:37.720 --> 11:45.400 in this case is that, yeah, you can't take it as a short cut for avoiding your responsibilities. 11:47.160 --> 11:52.040 Like, you need to check or all the project license is declared correctly and completely, 11:52.040 --> 11:57.640 you need to check through all the projects, contain the mandatory license text for licenses 11:57.720 --> 12:04.280 that require license text to be included. Are all the licenses acceptable? Not all licenses are 12:04.280 --> 12:11.240 acceptable for inclusion in the fedora. Are all projects legally acceptable? Like, is it covered by 12:11.240 --> 12:16.200 patents? Is it some kind of media codec that can be added to the fedora repositories? That's all 12:16.200 --> 12:23.240 things that you still need to check. And, like, due to limitations of how cargo windows dependencies, 12:23.320 --> 12:29.480 it's also not actually possible to run tests for your library dependencies. So you can only run 12:29.480 --> 12:36.120 the tests on of your actual top level application, but that doesn't cover tests in library dependencies. 12:38.280 --> 12:45.800 So, how do we do things the hard way? As far as I know, mostly fedora and Debian, and I think 12:45.800 --> 12:53.240 Maria also does it this way to some degree, which is basically create one RPM package per Rust 12:53.240 --> 12:58.600 crate, and they, as mentioned previously, only contain the source code and some metadata. 13:00.120 --> 13:05.800 But, by doing that, you actually share responsibilities between application maintenance. 13:05.800 --> 13:11.320 Because by now, there's quite a large overlap between the dependency trees of most applications. 13:11.720 --> 13:19.720 I mean, the Rust ecosystem has mostly stabilized on some popular libraries and most projects use those. 13:19.720 --> 13:25.720 So, for that stuff, you can just rely on sharing those responsibilities with other people. 13:26.280 --> 13:32.760 You avoid duplicated or it effort both for legal issues and technical issues, and you can actually 13:32.760 --> 13:39.640 quite easily run the test suite for those libraries, and often find bugs, especially related to 13:40.520 --> 13:45.880 less popular CPU architectures, like big-endian stuff, which is always funny. 13:47.880 --> 13:56.760 We also tend to find a little VM bugs quite frequently. So, yeah, those are always fun. 14:00.280 --> 14:06.040 We also don't get into dependency hell by doing this, because it's actually quite easy to 14:06.440 --> 14:15.320 provide parallel installable library versions, cargo namespaces, dependencies both by name and version. 14:15.320 --> 14:18.840 So, there's no conflicts involved. 14:22.040 --> 14:27.720 One of the benefits of doing it this way is that actually looking at the libraries, fixing issues we find, 14:29.400 --> 14:31.560 submitting those changes to upstream projects. 14:32.440 --> 14:37.880 Over time, we hope that this involves improved the ecosystem for all users of those crates, 14:37.880 --> 14:43.800 not only for other upstream packages, so eventually those fixes will roll out to everyone, basically. 14:46.680 --> 14:51.960 Okay, how do we make the doing the quote unquote right stuff as easy as possible? 14:53.640 --> 14:59.800 Subtemist little work has been put into improving the Rust packaging tooling that we have in 14:59.800 --> 15:02.920 Fedora, like Rust to RPM, and all the tools around that. 15:04.040 --> 15:09.240 By now, I think we've mostly eliminated all the busy work from the packaging workflows, 15:11.240 --> 15:16.520 and I don't think there's any language ecosystem that has tooling that is polished in Fedora, 15:18.120 --> 15:27.080 which is kind of the basting the myth that Rust packaging is hard. I hope, but if you've not looked 15:27.160 --> 15:33.000 at Rust packaging recently, then I recommend you to take a look again because things are really nice now. 15:33.880 --> 15:40.600 There are more improvements coming in the future. Some features that we have already developed 15:40.600 --> 15:46.920 can't really be enabled by default yet because some of the targets we support ship RPM versions that are 15:47.000 --> 15:57.640 too old, like RL 9 ships RPM 4.14 and RL 10 ships RPM 4.19, which are a bit old at this point, 15:59.000 --> 16:07.480 but we're still supporting those by building for Apple, so extra packages for Enterprise Linux. 16:09.640 --> 16:14.760 Two examples of recent improvements that we've made are like dynamically generated sub packages 16:15.400 --> 16:20.120 that dramatically reduces the boilerplate that you have in RPM spec files for Rust packages. 16:20.760 --> 16:27.240 It mostly makes it unnecessary to rerun Rust to RPM for crate updates or metadata changes, 16:27.240 --> 16:33.000 but it requires RPM 4.19, so that's not available on RL 10. 16:33.960 --> 16:41.880 Then there's support for fully declarative build system in for Rust crates for common cases 16:41.880 --> 16:47.800 that feature was added to RPM in RPM 4.20, so that's only available on Fedora, 16:47.800 --> 16:50.760 not even in Apple 10. 16:56.120 --> 17:01.240 Can we answer this question now? I mean, at least in my opinion, 17:02.440 --> 17:08.920 based on the missing features that are in new kinds of package managers, I don't think they can replace 17:09.000 --> 17:14.440 traditional package managers quite yet, especially because of missing support for like 17:15.400 --> 17:20.040 pushing security updates, which is kind of a critical feature in my mind. 17:22.040 --> 17:25.080 Okay, so I think we actually have quite a bit of time for questions. 17:27.000 --> 17:29.720 Yeah? 17:29.800 --> 17:38.040 Yeah, I was wondering, how many of us are built in this like five of us where you don't even have crates, 17:38.040 --> 17:44.600 really, the same sense, could I speak a bit? There are packages that are like a bit special, 17:44.600 --> 17:50.760 the Firefox is one of them. I think the publish sources, the sources, they publish for Firefox, 17:50.760 --> 17:55.000 actually contain all the things that you need to build Firefox, so they use a different process. 17:55.960 --> 17:59.160 Okay, so what else did you just skip it up for in here? 17:59.160 --> 18:04.680 Those packages are maintained in a different way, so they don't use the same tools. 18:08.920 --> 18:10.200 Yeah. 18:10.200 --> 18:16.920 So you mentioned that the Matean's workload of the dependencies shared by these users, 18:16.920 --> 18:22.760 I mean, but you think that is actually happened, or the Matean has these actually, 18:25.080 --> 18:29.720 taken more by the first person that meets the MSE, and they will need it anymore, 18:29.720 --> 18:31.160 and then you need to find another one. 18:32.760 --> 18:38.360 So the question was that that's sharing the responsibility for package maintenance, 18:38.360 --> 18:47.480 actually work, or is it just sort of onto one person? Usually, that works, yeah. 18:47.480 --> 18:53.960 So especially if you have like, hey, my application requires a library update for this library, 18:53.960 --> 18:59.880 and for that, we need to like, push changes to five other packages that depend on this, 18:59.880 --> 19:06.680 and usually that's the way things are shared between those applications maintainers by. 19:09.320 --> 19:13.080 Needs coming from different sites, if that makes sense. 19:13.880 --> 19:14.360 Yeah. 19:14.360 --> 19:19.560 Can you provide a more details on the state of the center's line and center's network 19:19.640 --> 19:23.640 as well as what is provided by the two last packages and whatnot? 19:27.640 --> 19:34.840 The question was what the tooling support is missing for Apple 10 and Apple 9? 19:36.520 --> 19:41.400 The only features that are not supported on those targets are the dynamically generated 19:41.400 --> 19:45.800 top packages and the declarative build system. The packages themselves are identical between 19:45.880 --> 19:46.920 federal and Apple. 20:07.880 --> 20:13.560 The question was how do we determine if there are security issues, or how do we determine what 20:13.560 --> 20:22.600 packages are affected? Yeah. There's a database of security issues in Rust Crates. 20:22.600 --> 20:29.720 There's a website called RustSec and that's basically the standard way to track vulnerabilities across 20:29.720 --> 20:36.680 the Rust ecosystem, and we are keeping track of those. There's also a CVE box filed against 20:36.680 --> 20:40.520 Rust libraries sometimes, though, happily not that frequently. 20:44.200 --> 20:51.880 Yeah. We have the mapping from the crate name to package name is easy because that's just 20:51.880 --> 20:58.520 the one-to-one mapping. And even for packages that used to the end of dependencies, they have 20:58.520 --> 21:07.320 mandatory metadata so you can map. You can query the package repositories for which package 21:07.320 --> 21:15.800 it bundles this crate. And so you can compile a list of affected packages and then apply 21:15.800 --> 21:23.000 the fixes, rebuild them with the fixed versions. How do you handle dependencies that are very 21:23.000 --> 21:31.480 happy to play with your semantic question? Okay, the question is how we handle dependencies that 21:31.560 --> 21:45.880 publish frequent major versions? Yeah. Yeah. Usually don't like them. 21:51.240 --> 21:56.360 Yeah, it causes more work than if they don't publish breaking versions, of course. 21:57.320 --> 22:02.680 I mean, as I said, it's possible to just ship multiple versions of the same crate. I mean, 22:02.680 --> 22:08.520 I think the one where we ship the most different versions is the nix crate. If you know that one, 22:08.520 --> 22:16.840 which has like 0.23, 0.24, or 0.25, up to 0.31 by now. We're trying to reduce the number of 22:16.840 --> 22:22.040 different versions that we ship just because all versions kind of tend to go stale. And a 22:22.040 --> 22:29.320 cumulative box, especially like with, I mean, for example, all versions of the lix crate are 22:29.320 --> 22:37.880 now kind of incompatible with glipc 2.43. So I've been trying to get rid of some of those old versions. 22:38.440 --> 22:45.480 But yeah, we can. The two approaches you can take is just try to patch everything to the new 22:45.560 --> 22:50.840 version and submit those changes to the upstream projects. So those eventually add up end up in 22:50.840 --> 22:57.320 new releases. Or you can package both versions in parallel, at least for a limited amount of time. 23:00.280 --> 23:07.080 Yeah. Have you ever encountered support for your ideas for a resistance from the community? 23:07.160 --> 23:20.040 Yeah. The question was, if there's been resistance to our efforts within the rust community, 23:21.320 --> 23:33.240 or support, both. I think. I mean, some of the, some of the frequent responses we get is 23:33.240 --> 23:42.840 why are you doing it this complicated way? And I mean, I think I provided some amount of 23:42.840 --> 23:50.680 reasoning why we do it the way we do it, just because it. Not doing it this way is kind of irresponsible. 23:51.640 --> 23:55.800 Um, on the other, Meteorical. 24:04.760 --> 24:07.280 I think it's. It's like transmit the road to some? 24:11.560 --> 24:17.220 Anyway, what about Aren't you thinking the number of cricket because of the 24:17.220 --> 24:23.020 Sometimes, there are features that just add staff them, but they don't like it. 24:23.020 --> 24:27.620 I get enough in ways if you want to feature them, but sometimes you can't really be sure 24:27.620 --> 24:32.220 that you have just adding stuff, we'll use that to reduce the packages and add more features 24:32.220 --> 24:43.220 than in what was actually used upstream, or just answer the last question, because 24:43.220 --> 24:48.220 this is up time is up now. 24:48.220 --> 24:53.220 I think the question was, how do we reduce the number of packages that we ship, 24:53.220 --> 24:59.220 because some combinations of feature flags are not compatible with each other? 24:59.220 --> 25:04.220 Yes, I mean feature flags are supposed to be additive, but that's not always the case. 25:04.220 --> 25:08.220 Like, for example, you have a package that supports two different backends, 25:08.220 --> 25:15.220 and you choose one of them, so enabling both is really kind of defeats the point. 25:15.220 --> 25:22.220 Mapping both features from cargo metadata to RPM package metadata is still necessary, 25:22.220 --> 25:27.220 because different applications might use different feature flags. 25:27.220 --> 25:33.220 But again, enabling both of them doesn't really make sense, but they're both need to still be present. 25:34.220 --> 25:40.220 I think that the time is over. 25:40.220 --> 25:43.220 You can ask me afterwards.