WEBVTT 00:00.000 --> 00:09.360 But yes, let us introduce ourselves. I'm Daniel Medado. I'm a software engineer. I 00:09.360 --> 00:14.600 re-hat and I'm also training training. I'm trying to run up the Fedora's EBPF 00:14.600 --> 00:20.440 secret. We will discuss another later. Basically, what we want to just speak about here 00:20.440 --> 00:26.080 is about the sciences that we have been dealing with when we were trying to package EBPF 00:26.080 --> 00:31.320 programs and Fedora. While this may see no, I thought when I first started to 00:31.320 --> 00:34.560 this, it has another program. Let's go package it. It's going to be a spec 00:34.560 --> 00:40.960 file and that's it. It was necessary forward. So, our plan is to go through a little 00:40.960 --> 00:45.840 bit about how this works. It's time to have fun while doing so as some solutions. 00:45.840 --> 00:49.120 And afterwards, we'll be happy to have some discussions probably offline because of the 00:49.120 --> 00:53.400 time constraints. This is my colleague, Mikael. He's also involved into 00:53.400 --> 00:57.400 the packaging and also working for Rehat and I'll let him start. Thank you. 00:57.400 --> 01:02.200 Thank you, Danny. So, good morning, everyone. So, our plan is to talk a little bit about 01:02.200 --> 01:07.880 why we think that packaging EBPF can be hard. How we plan to solve it and we will 01:07.880 --> 01:15.640 be giving you a list of checklists that we created for when we work packaging EBPF 01:15.640 --> 01:22.360 applications. So, EBPF is here. We think it can be a packaging name for some people. 01:22.360 --> 01:27.160 So, the thing is that EBPF is no longer optional. EBPF is everywhere. 01:27.160 --> 01:32.120 EBPF is already running in the most of our systems through system D or using 01:32.120 --> 01:37.160 different CNIs or whatever that we are using. But the thing is that the 01:37.160 --> 01:41.800 use cases are growing a lot. And we want our 01:41.800 --> 01:47.320 distributions to become part of the EBPF platforms. I would say like, 01:47.320 --> 01:55.080 how we as a distribution packages can take EBPF applications to the users. So, 01:55.080 --> 02:01.720 usually the, okay, the packaging patterning for traditional software was to take the source 02:01.720 --> 02:07.800 to compile it, create a binary and ship it. We were relying on non-library, like 02:07.800 --> 02:13.160 lip-c, lib-c, lib-accessary, cool or whatever, right. But in the case of EBPF 02:13.160 --> 02:18.040 software is a little bit different, because the kernel itself is the library. It's a dependency. 02:18.040 --> 02:27.800 And the kernel has not a stable ABI so it can be a problem, okay. Also, while we are trying 02:27.800 --> 02:34.200 with EBPF is to try to inject kernel models that behave as a user space application. So, 02:34.200 --> 02:39.000 that's something new for many people. But at the end, we think that many users, 02:39.000 --> 02:43.800 expect that they can just run DNF or whatever tool they use to install packages in the 02:43.800 --> 02:49.400 distributions and get the same feeling or experience with installing it. 02:50.280 --> 02:55.480 Their reality behind that is that, as I said, kernel structure changes, but if 02:55.480 --> 03:01.320 structure changes and so on. And they also, we may have some capability issues that we need to 03:01.320 --> 03:06.600 define that this can make a C, or a Selineux policy, I think, like that, that they use as 03:06.600 --> 03:12.680 might find complicated to do it by themselves. So, the stacks are high. If we do it wrong, 03:13.320 --> 03:18.680 since then others will break whatever software we are running, monitoring, or whatever the EBPF 03:18.680 --> 03:24.520 application was doing, we will fail and users will lost the trust in the distribution. 03:24.520 --> 03:29.160 But if we do it correctly, we think that EBPF might be kind of another boring infrastructure 03:29.160 --> 03:35.160 technology, like, and like we are doing with other languages like could be go or asked in 03:35.160 --> 03:41.800 Fedora, for example, we think that it should be installed once and run forever, and our goal at 03:41.800 --> 03:46.920 the end, we will show with that. What we are trying is that this will become EBPF friendly, like 03:46.920 --> 03:54.840 another C descent. So, the main highlights is that the old way, the EBPF compiler collection that 03:54.840 --> 04:00.600 was supposed, you were meant to use that at runtime to compile the applications that were 04:00.600 --> 04:04.440 that you were planning to use, the EBPF application you were planning to use at runtime, 04:04.440 --> 04:08.520 and the problem with that is that that requires a lot of software to be installed in the system. 04:08.520 --> 04:13.080 Like, we are talking about C language, LLVM, and so on. And usually you don't want to have that 04:13.080 --> 04:18.760 kind of applications in running server. It takes up to, in the case of Fedora, it was more than two 04:18.760 --> 04:25.160 gigabytes, and it's a lot. So, as a distro, what we want is to have just a small pre-compile binary 04:25.720 --> 04:30.680 that has minimal runtime, that will check everything for you, that will make you update and all the 04:30.680 --> 04:39.080 stuff easy. So, we need to go to the second part. The second headache is that, as I was mentioned, 04:39.080 --> 04:47.400 the cannot doesn't have an ABA stability. So, extract that, we might depend on my change between 04:47.400 --> 04:52.920 between releases. This is one example of how the process is. The process extract that is defined 04:52.920 --> 05:00.760 by the cannot changed between release 5.15 and 6.8 with PID, feel that change the position. 05:00.760 --> 05:06.920 So, if your application were running, what's really in that position 16, it will 05:06.920 --> 05:11.880 re-guarbet with newer kernel. So, we need to take care of that. And then we need to make sure 05:11.880 --> 05:15.800 that the matrix of toolings that we are using is the correct one, that's something that 05:15.800 --> 05:23.560 distributions helps a lot to do, because it's not easy to maintain the compatibility between 05:23.560 --> 05:28.360 the application, the BPF, BPF tool, C-Lang, and the kernel. So, that's where we think that 05:28.360 --> 05:36.280 the distribution matrix can help. So, for that, we will rely on core and Daniel will talk more 05:36.280 --> 05:44.280 about that. 05:44.280 --> 05:50.280 Okay, so, as my colleague was saying, like, this is not really a stick. So, I would like to 05:50.280 --> 05:56.280 be in this distribution, not even again, to have in it to recompile every BPF program that 05:56.280 --> 06:02.280 I'm going to be packaging, every time I just go and upload the kernel. So, BPF provides 06:02.280 --> 06:06.440 the new tooling, which suits work in every model, Linux kernel distribution, starting in 06:06.440 --> 06:13.880 67, which is called BPF Core, which stands for Compile once around everywhere. So, I want to 06:13.880 --> 06:19.560 know if it can be seen there quickly, it doesn't click it well. But basically, let me try and 06:19.560 --> 06:26.120 be brave. So, here we got the source code. This is our, yes, BPF program. We are going to be using 06:26.120 --> 06:31.400 C-Lang and BPF, which stands for BPF type format. This is just some kind of metadata, which 06:31.400 --> 06:37.960 would make use of kernel header information, and that would stop us having to rely on kernel 06:37.960 --> 06:43.400 headers on their running system. What does that mean? That means, let's say, we are going to 06:43.400 --> 06:48.840 be compiling this using, you see, see, or whatever, and the end we are going to be having 06:49.880 --> 06:56.360 this file, which would be workable in any kernel. So, not only, like, I wouldn't have to 06:56.360 --> 07:01.720 go over the compiling dober, and this is also really important when we think about the 07:01.720 --> 07:08.600 Fedor infrastructure, we got codians of course, and I would like for the codi on any build system 07:08.600 --> 07:14.680 to have it to go and download a lot of dependencies, yes, for even building, and even while 07:14.680 --> 07:19.080 building is fine, I would like for me to have it to go to a production cloud, and having to 07:19.080 --> 07:26.280 have all those dependencies in runtime. So, then goes BPF, and BPF would read this, BPF, which 07:26.280 --> 07:30.920 I said, you just submitted data, and would rewrite all these issues with the headers that we 07:30.920 --> 07:35.640 were seeing in all those lights, and so then we wouldn't have a kernel pane, basically. 07:36.040 --> 07:40.040 Was that happens? Our binary would be working into all the kernel distributions. 07:41.320 --> 07:50.840 So, one else. Yeah, so here, as I was saying before, a compiling time, if our codi is using 07:50.840 --> 07:56.680 core macros, we can just rely on here. Let's say, again, getting back to the example of the 07:56.680 --> 08:02.280 we were speaking before. If we are needed in the pit here for the data struct, that's not going 08:02.280 --> 08:07.400 to be here coded. It's not going to be a stored as the BTF header here, we made a data, and in the 08:07.400 --> 08:12.440 end, it would just work the same Fedor after nine, forty, forty one, relying or open success, 08:12.440 --> 08:18.120 what probably? Basically, any Linux with a modern kernel, I guess it's started in Fedor 08:18.120 --> 08:25.240 specifically in 39, but I need to check probably in 32. And how does it work? We'll be BTF. 08:25.240 --> 08:30.760 It would reinvent the metadata, which is stored here, but away. It would find a new struct here, 08:31.160 --> 08:37.960 and it would be fine, and wouldn't do any kind of kernel pane. So, this is a little bit 08:37.960 --> 08:44.920 the same as I was saying before, like probably better explain here. So, we have our source code, 08:45.560 --> 08:50.920 and we can just put it here. The PTF has a score read. It would be read by Siulang, 08:50.920 --> 08:55.640 it's going to be compiled into our final guide. And this byte code would be now fine, 08:55.640 --> 09:00.120 because it wouldn't be just going in trying to serve this specific pit. Then, when the 09:00.120 --> 09:06.680 hour BTF code is going to be logged in, relying on BTF, also the kernel would be fine, and our byte 09:06.680 --> 09:14.360 code would be just passed and it would work across any kernel. So, before, 09:15.320 --> 09:21.880 BTF core or BTF, we were, again, relying on all these dependencies, even in run time. 09:21.880 --> 09:27.720 Now, we all need them for in build time. And in our packages, we will need to do any kind of build 09:27.720 --> 09:35.720 requests anymore, and yes, our requires. This is the, sorry, again. Awesome. Well, I'll be going 09:35.800 --> 09:43.000 over there because I can't read. So, it's not there, it's the, it's a, it's a, it's a, 09:43.000 --> 09:51.400 a tournament. So, this is like, basically, the equivalent of dynamic linking, you'll see that 09:51.400 --> 09:56.440 on those slides, it's going to be available and post a link. But anyway, it's also helps us 09:56.440 --> 10:00.360 like if there's any CBL, say, for the BTF, we wouldn't have to write that, it's going to be dynamic 10:01.320 --> 10:05.640 how does it work for Fedora, specifically? So, all the interest ready, as I was saying, 10:05.640 --> 10:12.360 this works Fedora 39 onwards and so forth. And if you, well, let me see here, please 10:12.600 --> 10:28.440 stop flittering for a second. Anyway, anyway, anyway, so here is, well, if you can see that, 10:30.840 --> 10:37.080 BTFL system is a BTFL system, so it's exposed as it's off there, and you'll be able to 10:37.880 --> 10:41.880 get that it's here, and you can just write that, and you're going to be happy and like the 10:44.680 --> 10:56.360 BTFHeaders here on the kernel, it'll be online. And as I was saying, the dynamic, well, how does this 10:56.360 --> 11:05.800 relates to our spec file getting back to our, so we shouldn't have to rely on this static 11:05.800 --> 11:13.480 table, it's because, again, when I'm just shipping any RPM in the absence case, I wouldn't like to 11:13.480 --> 11:19.000 see this specific metadata here on as a table there with a spec file, but as we are relying on BTFCore, 11:19.000 --> 11:24.280 we can just go and use kernel call and kernel labeled here internally, and this would be just 11:24.280 --> 11:27.400 getting the one for the proper one and pass it. 11:28.120 --> 11:37.880 All right, same thing for like the environment, so in Koyumog, slash Cs would reflect the 11:37.880 --> 11:41.720 build stop journal, and as I was saying, I would like for these to be happening in Fadoris 11:41.720 --> 11:46.360 infrastructure, because I would like for these to be having to, let's say we got this build note, 11:46.360 --> 11:51.560 which is Koyi, it might be Fador 43, run like whatever, I don't care, whatever we happen there, 11:52.120 --> 11:57.160 but what happens if we've got a target of Fadoro height? I mean, this kind of wouldn't be fine, 11:57.240 --> 12:01.560 because this would be pre BTFCore, and this is direct kernel, and the code used, 12:01.560 --> 12:06.760 would use in that, which is good at it, and 60, what happens? Yes, it would fail, and boom, 12:07.560 --> 12:12.280 how do we do that? Again, using BTFCore, we would be able to get the kernel version directly from here, 12:12.840 --> 12:16.040 and you just start to compress kernel, and you might use in this active version, 12:16.760 --> 12:21.800 and we would be generating this header, using BTFCore, this BTFCore command, basically, 12:21.880 --> 12:25.160 regenerate the BNLunus.header that we were speaking about before. 12:27.560 --> 12:34.680 And then we had him back to Mikael for some examples, and we even got a demo, let's see if the 12:34.680 --> 12:45.000 Flictros allows us. Okay, so for the sake of this presentation, we created a small BTFCore 12:45.000 --> 12:50.120 tool that is called ProcessSnub, that all you will do is print in the screen, all the commands 12:50.200 --> 12:55.080 that are launched in a system, okay? So we are not doing in the scripting way, I was telling before 12:55.080 --> 13:00.120 the BCCA way, like we are not doing it on runtime, so we are doing this tool mode, we will have 13:00.120 --> 13:07.160 several dependencies, so that I'll lip see a BTFCore, I will generate a small 20k binary, and 13:07.160 --> 13:14.200 I will have everything on it, okay? It will be, it's just this simple script, it relies on the 13:14.200 --> 13:20.360 bridge, the headers that we are generating in the script, that I will, the spec that will show you. 13:20.920 --> 13:25.560 Now, okay, but that's the important part, okay? The first line, but the script is just this 13:26.760 --> 13:31.160 to create a BTFCore application, so what we call the goal and stand out, or what we do 13:31.160 --> 13:38.680 the whole somehow, expect would be this one, okay? Everything is linked in the presentation, so 13:38.680 --> 13:43.240 we will have the bill requirements, bill requirements, to build the application, that will be the 13:43.240 --> 13:50.200 compilers and the kernel core kernel level, and we will require just libbpf, as said to be able to 13:50.200 --> 13:57.400 launch the Bbpf applications, then as Daniel said, we are going to look after the kernel that we 13:57.400 --> 14:04.680 need to use and generate the headers there, and then we will run this bpf tool to create a 14:04.680 --> 14:13.400 code for the, sorry, to generate the headers, I wonder now, sorry, sorry, we will compile the 14:13.400 --> 14:19.560 bcode of the Bbpf, then we will create the headers of that Bbpf applications, that when we will 14:19.560 --> 14:27.400 use GCC to link against the libbpf and create the final binary that will be running. Then 14:27.400 --> 14:33.080 there is a full script after we have two options, I mean we can just create the install the file, 14:33.080 --> 14:38.920 the binary file has been created, we can't say the capabilities while creating the 14:38.920 --> 14:46.200 spec that my allow regular users to just run it, the option is just to launch it using system 14:46.200 --> 14:52.280 integration with a regular service, the important part is that we need to define the capabilities 14:52.280 --> 15:01.240 and the ambient capabilities, okay? The good thing about using this kind of spec is that we 15:01.240 --> 15:05.640 will allow us to run the same spec and the same application against different many different 15:05.640 --> 15:13.320 kernels, okay? So sometimes we might be tempted to to pin whatever releases we want to use to 15:13.320 --> 15:18.280 create the bpf because we might have read a C-lang by code might change between versions or whatever, 15:18.280 --> 15:24.360 but we think that is good idea to have us open spec as possible and they rely on things like 15:24.440 --> 15:29.640 Koshae or Koji or whatever to find the problems I would call my half, okay? 15:36.280 --> 15:40.680 So when that's the base, of course things doesn't work as smooth as we want to do, 15:40.680 --> 15:46.360 and we found a couple of edge cases, first of all, like some stuff it depends on the 15:46.440 --> 15:53.720 bpf problems themselves, so bpf uses the map which is a memory structure that is basically 15:53.720 --> 15:59.080 using between both user space or internal space, and what happens if I just go and update 15:59.080 --> 16:05.720 maybe bpf tool? It may be the case, that's kind of interesting here anyway, not really, but yeah. 16:08.840 --> 16:15.000 So if we stop a service, my bpf service may still be pointing to the old map, the new service 16:15.000 --> 16:22.760 starts, then boom, how do we handle that? So click there, yeah. So this is an upstream requirement, 16:22.760 --> 16:29.720 it's something that it relies on not only of the spec file, but on the bpf code or the developer 16:29.720 --> 16:35.720 application, so bpf programs must be written to handle that and reuse existing maps. 16:36.360 --> 16:48.920 It's, this is an example using yes cbpf library, bpf, but yes a requirement, not only for spec files, 16:48.920 --> 16:56.760 but just over lbpf programs. Then I think this is important, and I was having another set 16:57.320 --> 17:03.720 in another session yesterday, which is like, you cannot get the ability to use. So when we first 17:03.720 --> 17:09.320 started doing bpf programs, we were using basically, copy-satmen, this is legacy, don't use this, 17:09.320 --> 17:14.120 is basically covered, you're giving root access to your bpf program, which you may not want to. 17:14.840 --> 17:21.800 Another option would be, let's assume we are going to have a packaging and xdp program, 17:21.800 --> 17:27.320 we would need like, capperf mom, kbpf, and capenetalmen, okay pretty much like root, 17:28.200 --> 17:32.520 there's a new change that we are going to be proposing to fedora, which is like the 17:32.520 --> 17:38.280 system-wide usage of bpf tokens, which they don't only use kernel capabilities, but anything 17:38.280 --> 17:42.680 called delegations, and that means that as a root user or a set admin, you would be able to 17:44.600 --> 17:51.960 allow like non-root users to basically run a specific bpf programs, which is specific and 17:51.960 --> 17:57.880 more, much more grain like permissions. I'm happy to discuss that as we are also running the 17:57.880 --> 18:01.000 bpf system group and that's something that we are going to be proposing as a system-wide 18:01.000 --> 18:06.920 since we're fedora 44 or 45, but anyway, just FYI, this may be an issue, just go ahead. 18:10.360 --> 18:18.360 Also for, as is the disservice, we would need to take that into consideration just for the 18:18.360 --> 18:24.120 amount, this may be enough, but again if it's just an xdp program, it may not be enough, 18:25.160 --> 18:28.840 we may also need to get different capabilities for both time, 18:29.080 --> 18:31.800 positive and of course as elinux, which I guess is the next slide. 18:34.280 --> 18:39.560 Yeah, here we got as elinux. So far by default, if you're too advanced in a confidiment, 18:39.560 --> 18:44.520 as elinux would just deny all your bpf operations, and depending on what you want to do, 18:44.520 --> 18:48.760 it may be also the next. You would need to go and add a companion or another option that we were 18:48.760 --> 18:55.960 just thinking about is to add some global system-wide system policies for bpf, but that's a 18:55.960 --> 19:03.720 no topic. Yeah, this is just an example. Well, of course, we would also need to detect the 19:03.720 --> 19:09.000 kernel. I was speaking about bpf core, which I said it was totally okay for different kernels, 19:09.560 --> 19:14.280 but what happens if your kernel doesn't support bpf core? As I was saying, you would need to go 19:14.280 --> 19:20.440 and recompile your application for every different kernel pre-core, and so forth. And even if 19:20.440 --> 19:25.400 that's somehow fine, basically you need to be aware of that when you're packaging, if you want 19:25.480 --> 19:31.960 your package to support lex resistance. So, basically using require kernel higher than 6-1, should be 19:31.960 --> 19:39.080 fine. Um, this is what happened to one of the kernel updates. Again, core comes to the risk you, 19:39.080 --> 19:46.040 anyone just worked, but if we got some lexic kernels, it may or may not work depending on how 19:46.040 --> 19:50.440 they maps on headers have been changed. If you're just lucky, it may just happen to work, 19:50.440 --> 19:53.400 but it doesn't have to be. So, this is something that you should take into consideration 19:53.560 --> 19:59.960 we created your spec file for lexic systems. So, um, I'm going to be handing over down to 19:59.960 --> 20:06.520 Michael Lee and let's see if we got them of them. Okay. Yeah, that's right. Okay. So, 20:07.560 --> 20:12.280 this is the checklists we are following, uh, spark edges. So, before packaging, we need to check 20:12.280 --> 20:17.480 if the application supports core. If, uh, make sure that, uh, it doesn't require any specific 20:17.480 --> 20:24.200 LibbPF to, uh, that is ban the load that is statically, uh, require, request statically compile, 20:24.200 --> 20:31.160 uh, LibbPF. It's important also that, uh, LibbPF will check what kind of license the application 20:31.160 --> 20:36.440 have. So, it's important to check that everything is right at the level. And also that the headers 20:36.440 --> 20:42.120 are generated at bill time. So, make, we can make sure that we are using, uh, every release, 20:42.680 --> 20:48.760 every version, every kernel version generated here. This is really important. And then, 20:49.400 --> 20:54.760 after we create a package, uh, it's important that we serve the correct, uh, minimal capabilities 20:54.760 --> 21:00.360 that, uh, say Linux, uh, policies created if it's required for that, that applications that we 21:00.360 --> 21:05.240 have to comment in any change that is, or any change that might affect, any change in the kernel 21:05.240 --> 21:10.520 that might affect the package and so on. So, it makes maintenance much easier. And then, as we said, 21:10.680 --> 21:16.280 it's important that, uh, we tested it in many different, uh, releases to make sure that we are 21:16.280 --> 21:23.240 not breaking anything. Uh, quick reference card would be the spec that we already show on the 21:23.240 --> 21:30.040 system, the essential just as a one as a light of, of how we should be done. Uh, yeah. And last, 21:30.040 --> 21:35.320 and just quickly, uh, we as a LibbPF's group, we are working in both adding new macros to the spec 21:35.400 --> 21:40.840 to support directly these ebbF's new tooling and spec files and bbf token as I said. And we're 21:40.840 --> 21:47.400 going to be proposing a new changes for, uh, like for 244 onwards. So please join our group if you 21:47.400 --> 21:52.840 want to help us into that and we are totally open for contributions and, um, you know, basically 21:52.840 --> 21:58.280 feedback about that. So, with that, we are running out of time. So, thanks for attending and we are 21:58.360 --> 22:08.280 up and through any questions. Thank you. 22:08.280 --> 22:15.720 So, uh, you move to another solution is we are actually seeking the new stage that corresponds 22:15.720 --> 22:20.520 to the kernel of the machine as if a package. I think it's a lot of those the same and we're using 22:20.520 --> 22:25.480 the system to give you. So I'm wondering why you are advocating for everything that everyone 22:25.800 --> 22:39.000 can. Yeah, I was going to answer the question. So in the book, he won't do, uh, 22:39.000 --> 22:47.080 is already packaging bm Linux headers. Uh, why are we advocating to create, uh, in, in the 22:47.080 --> 22:55.400 spec itself again and again? Um, that's basically, uh, I mean, that could be also done, 22:55.400 --> 22:59.960 but in, I was speaking specifically about the use cases for ebbf program packaging when 22:59.960 --> 23:04.520 not using bbf core because basically it wouldn't just be able to reuse that directly or not 23:04.520 --> 23:09.720 without some specific things. What may happen is that if you don't use bbf core is that your 23:09.720 --> 23:15.080 change in that and you might need to renew that for your specific packets. I mean, uh, I can give, 23:15.080 --> 23:19.000 I mean, we can just discuss that offline because we're running out of time, but that's also an 23:19.000 --> 23:22.600 option that we would like to discuss in this new proposal. We are speaking. Thank you. 23:25.640 --> 23:26.440 Any other questions? 23:28.280 --> 23:32.280 Yep. 23:35.480 --> 23:43.160 So is the are we already shipping this? My question will be which part of it. So no, no, you can. 23:43.240 --> 23:49.000 So we, uh, we created already as I said, uh, let me check. So call this already available in 23:49.000 --> 23:58.840 a small repo. So this is the bare minimum. Sorry, uh, it's not okay. Yeah, so we have an example. 23:58.840 --> 24:06.440 We have also created. So it's not that one, but we also created a copper repo to build it 24:06.440 --> 24:11.720 against different versions and so on. So it's already, you're already able to use it. The thing 24:11.800 --> 24:16.200 is that the macros are not there here. For example, the mixed thing is here, but if you work, 24:16.200 --> 24:20.040 you can create a spec and the applications and so on, if you'd work. For example, this, this 24:20.040 --> 24:25.880 little application, we just compiled yesterday. We built it against different federal releases 24:25.880 --> 24:30.600 and Apple releases and we are able to run the same application with the same spec. In federal 24:30.600 --> 24:38.040 43 and in, I was in all my nine or something like that yesterday. So we have a lot of time, but we 24:38.120 --> 24:42.120 get to be on site and if anybody wants to keep it that, then we'll be right back. Thank you all. 24:42.120 --> 24:46.840 Thank you.