WEBVTT 00:00.000 --> 00:12.960 Hi everyone, that is the corporate disclaimer. My disclaimer is that I'm about to tell you 00:12.960 --> 00:19.840 a lot of lies. Yeah, you're welcome. That's because if I told you everything that was in detail 00:19.840 --> 00:23.680 that was absolutely strictly accurate, we will be here all day and we just don't have time 00:23.680 --> 00:29.520 for it. So if you know better, if you think, hey, he's oversimplifying this, yes I am. I'm 00:29.600 --> 00:33.680 going to mislead you. I'm going to give you enough information that you understand why I'm doing 00:33.680 --> 00:40.880 what I'm doing, but don't don't rely on what I'm telling you if you're working on something else. 00:40.880 --> 00:47.120 What I'm telling you is true for the purposes of this talk. So, Mem, why do you want to shrink it? 00:47.120 --> 00:53.840 Well, it takes 1.6% of your memory at boot up and some people think that maybe they have 00:53.840 --> 01:00.640 better uses for it. I think of it as being 16 megabytes per gigabyte. So if you have like an 8 gigabyte 01:00.640 --> 01:06.400 phone, then that's 128 megabytes of your RAM that's being used for all mem app. And maybe you'd 01:06.400 --> 01:11.360 like to load one extra tab in your web browser or maybe you have a 1 terabyte server and that's 01:11.360 --> 01:19.280 16 gigabytes of memory that maybe you could use to rent out one extra VM on your server. So it's 01:19.360 --> 01:27.200 not a trivial amount of memory. I think working on performance, stuff for a while and it's 01:27.200 --> 01:32.560 never been big enough to get on my radar, but there are other people with other use cases for whom. 01:32.560 --> 01:38.080 Yeah, this is enough that it's worth spending a bit of time on and by a bit of time I mean we're 01:38.080 --> 01:47.440 coming up on 8 years. So we've got some other benefits on the way. So what is Mem map? If we're 01:47.440 --> 01:51.120 going to shrink it, we have to know what it is. And it's actually just an array of struct page. 01:53.520 --> 01:57.120 And there's actually there are actually three opinions about how big struct page ought to be. 01:59.680 --> 02:02.880 Some people think it's too large because that's 1.6% of your memory that you could be using 02:02.880 --> 02:09.520 or something else. There are people who think it's too small. They want to add extra stuff to struct page. 02:09.520 --> 02:13.440 There are constantly proposals for, hey can I just add this one extra thing to struct page? 02:14.000 --> 02:17.040 The most audacious of these was something you want to double the size of struct page. 02:17.760 --> 02:24.480 So they could put different things on different cash lines. And that was like, wow, no. 02:27.520 --> 02:32.880 There is an extension struct page x, which exists. It's absolutely horrible, don't look at 02:32.880 --> 02:38.240 it and it's slow. If you don't put performance stuff on it, you put stuff on it that you've 02:38.240 --> 02:45.120 got nowhere better to put it. And I hope we can get rid of it. And there are people who say don't 02:45.120 --> 02:50.640 change a thing. Struct page is exactly the right size. It is 64 bytes. 64 bytes is a magic size on 02:50.640 --> 02:55.200 current CPUs. It's the size of a cash line. So if you do one thing to a struct page, it doesn't 02:55.200 --> 03:00.320 affect the struct page that's next to it with somebody else that you're using. So it is perfect 03:00.320 --> 03:04.000 size, don't change a thing. So we're going to bear these three perspectives in mind and get 03:04.000 --> 03:10.560 back to them at the end. So how do you get a struct page? We call the page allocator. What's 03:10.560 --> 03:15.760 the page allocator? It is the fundamental memory allocator in Linux. There were lots of other 03:15.760 --> 03:20.480 memory allocators which have other uses and you almost certainly want to be using one of these 03:20.480 --> 03:26.400 other memory allocators, but for some things you want to use the page allocator directly. 03:26.960 --> 03:32.640 And all these other memory allocators sit on top of the page allocator. They all get their memory, 03:32.640 --> 03:40.240 but they hand out from the page allocator to begin with. So when you call alloc page, which is 03:40.240 --> 03:44.720 your interface to the page allocator, that gives you not just the four kilobytes of memory, 03:45.280 --> 03:51.040 it also gives you the struct page that controls that four kilobytes of memory. And you can use it 03:52.400 --> 03:58.560 because you call alloc pages is yours. With some arcane restrictions, I'm not going to go into 03:58.640 --> 04:06.000 because we don't have time. So when I came to struct page, this is what I like in late 2017. 04:07.840 --> 04:11.920 And you're looking at that as the thinking that is a wall of text. I like to use this as a 04:11.920 --> 04:21.360 scare people. You can't tell what most of this is useful. If you want to change anything in 04:21.360 --> 04:32.880 struct page, if you want to use struct page, you look at this. I don't know. I have an unsigned 04:32.880 --> 04:39.280 long. I want to sort in struct page. Where should I put it? Well, there's all these counters. 04:39.280 --> 04:44.560 Counters is an unsigned long. I'll use counters. And then you realize that actually you just 04:44.560 --> 04:50.320 overwritten riff counts and everything breaks. So you try to do something else. And it's all for 04:50.400 --> 04:56.960 right. You can't really use this without spending a week trying to figure out where are the 04:56.960 --> 05:04.320 holes, what can I use, but what do some of the else we're using for? So this is now late 2018. 05:05.760 --> 05:09.520 These are the long time support reasons I went through found the right years. 05:11.920 --> 05:15.600 So I've rearranged it and now you can tell because there's comments. 05:16.480 --> 05:21.040 There were comments before. They weren't very useful comments. But now the comments actually tell you 05:21.760 --> 05:26.400 who uses these things. What are they useful? So if you've got something to use for slab, 05:27.120 --> 05:33.200 it's in the struct slab. These are all anonymous unions because that made everything keep working. 05:33.200 --> 05:37.280 We could have gone through a named all the unions, but that would have been a lot of shurn. We didn't 05:37.280 --> 05:41.280 really need to. This was good enough for what we were doing at the time. 05:41.760 --> 05:51.440 So yeah, this is now a little bit more understandable. And that paved the way for what came next, 05:52.720 --> 06:01.520 which was moving stuff out of struct page. You can see. We could actually go this into two columns, 06:01.520 --> 06:08.560 maybe even one column. We would have been able to get it shrunk further, but we also discovered 06:08.560 --> 06:17.200 page pool was using some of the elements that were here. It was a hidden user. The page pool 06:17.200 --> 06:23.360 user was using some of these elements and we didn't know because nobody had gone to look. 06:25.280 --> 06:29.360 So it's been made as official and we said, okay, you've got your own substructed 06:29.360 --> 06:36.080 struct page. But what we've done is we now have a separate struct slab and we have a struct 06:36.880 --> 06:45.760 and we have other structs. They overlay struct page. So you can't see them in struct page anymore, 06:45.760 --> 06:49.920 but struct page is still being used for those purposes. And this is a necessary intermediate step. 06:49.920 --> 06:53.920 It's not a great step because it's kind of nice to look at struct page and go to see where do 06:53.920 --> 06:59.680 these how do all these things line up. We have build time assertions to make sure that things 06:59.680 --> 07:06.320 that do need to line up still do line up and scary comments saying that there are dragons here, 07:06.320 --> 07:14.160 please don't touch. And you'll see that this is a kind of temporary state of affairs. 07:16.400 --> 07:22.080 Okay, so now we've organized struct page. How do we shrink it? I mean we shrunk with a 07:22.080 --> 07:26.240 finished struct page and there's fewer lines of sea code, but it's still occupying 64 bytes. 07:26.720 --> 07:33.920 So we need to identify redundancy. So what do we have in struct page that we could get rid of? 07:35.600 --> 07:41.600 And the primary thing that we're doing is dealing with multi-page allocations. So it's not just 07:41.600 --> 07:48.240 alloc page, the alloc page is actually just a front, it's just a macro. It's, of course, 07:48.240 --> 07:56.240 alloc pages which fascism order. Sure, that's exactly the kind of comment I didn't want to hear. 07:59.680 --> 08:05.520 So alloc pages gives an order. So order is the log to base two of the thing you actually want. 08:05.520 --> 08:12.320 So if you want to allocate four pages you call alloc pages two and that gives you two to the two pages, 08:12.320 --> 08:23.680 which is four. So when, when you do that, all, we, we, we talk about the head page, 08:23.680 --> 08:28.560 the first page. And then the tail pages, which are all the ones that come after it. So if, if you 08:28.560 --> 08:36.000 call alloc, alloc pages three, you'll get seven tail pages and one head page. And the only thing 08:36.000 --> 08:41.360 in the tail page is this thing called compound head. There's a little bit more to it than that, but 08:42.640 --> 08:50.400 simplify. So if all we've got in the tail pages is a pointer that says, hey, that guy over there, 08:50.400 --> 08:58.080 he's the real head page ask him not me. We can shrink that because using 64 bytes to store eight 08:58.160 --> 09:05.360 bytes is wasted. So instead of pointing to the head page, we now, we're going to shrink 09:05.360 --> 09:15.680 drop page dynamically allocate the data that was stored in that 64 byte head page and then have 09:15.680 --> 09:22.960 all eight of the pages point to that page. So eventual goal is an eight byte drop page. 09:23.120 --> 09:34.160 That's a lot of work because we have actually encouraged people to use stroke page. If you're 09:34.160 --> 09:39.920 calling alloc pages, then you get to use stroke page. That's the contract, that's what we told people. 09:39.920 --> 09:46.800 And now we're saying, yeah, could you stop doing that? So, you know, it is on us as, as Mary 09:46.800 --> 09:52.880 Management people to give them a path forward, right? We're not, we're not, it is up to us to 09:52.880 --> 10:00.240 fix their code for them because we've we've changed the contract not that. It's going to be a lot 10:00.240 --> 10:06.480 of work to get to eight byte struck pages. So the 2025 goal is a 32 byte struck page. So we're 10:06.480 --> 10:12.640 going to heart half it. We're going to give you back eight megabytes per gigabyte instead of the 10:12.720 --> 10:21.280 14 gigabytes that we're promising. So it's going to be much much smaller. It's going to have flags, 10:21.280 --> 10:28.320 Mendesk, RF count, and a private. So you can allocate extra memory and hang it off private 10:29.600 --> 10:38.000 or maybe hang it off Mendesk because that's going to be up to you. So what we need to in 2025 10:38.960 --> 10:45.360 we to finish the folio conversions and file systems. So there's still a lot of code. There's still 10:45.360 --> 10:53.440 some file systems which are using the elements of struck page. So they're doing page arrow mapping 10:53.440 --> 11:02.240 or page arrow index and they should all now have been converted to folio index. I think 11:03.200 --> 11:13.520 Seth is so outstanding. F2FS just finished. There's a few other things but most mostly we're done. 11:14.160 --> 11:19.680 I did have a goal of getting rid of page index this much window. It was always a bit of a stretch goal 11:19.680 --> 11:25.040 and we didn't quite make it but we got almost all the patches that we need. So next murder window 11:25.040 --> 11:31.840 page arrow index is going away for sure. I have a bunch of patches queued up to get rid of page mapping. 11:33.120 --> 11:39.520 I started a patch series to get rid of page Mendesk gd data and we're almost there. I need to go 11:39.520 --> 11:48.160 back and we visit page LIU. So despite the fact it's called page LIU it's just a list head. So 11:48.160 --> 11:53.600 if you want to put pages onto a doubly linked list you use page LIU. So well that's not an LIU. 11:53.600 --> 11:56.880 Well but it's the right type to be a list head so we're using it. 11:57.360 --> 12:05.200 Okay we will fix this. We will find a way that you can keep track of your pages that doesn't 12:05.200 --> 12:10.880 involve using a list head because linked list or evil and I've given rants about how evil 12:10.880 --> 12:18.320 linked list are before. So a lot of independent projects here like everywhere in the kernel that you 12:18.320 --> 12:24.960 see page LIU or page mapping or any of these other things being used that's somewhere that if you're 12:24.960 --> 12:37.520 inclined to help you could picture. If you do please we have a THP cabal call every other Wednesday 12:37.520 --> 12:41.920 and just let me know if you want an invite and we can call out we can coordinate and make sure 12:41.920 --> 12:46.720 nobody's working on the same project because there's a few projects that there's a few uses of 12:46.720 --> 12:51.840 page LIU still out there that I'm not entirely sure how to fix but we'll figure out a way. 12:52.320 --> 13:00.720 So we have some places which do implicit casting between struck page and struck folio which is 13:00.720 --> 13:06.080 all fine for now because they're the same data structure but we're going to separate them then 13:06.080 --> 13:13.120 we need to stop pretending that the same data structure. So one of those is again within file systems 13:13.120 --> 13:17.520 we have this thing called the buffer head buffer head used to have a well they still have a 13:17.520 --> 13:24.000 B page pointer and the points to the struck page that the buffer head points into. Now there's a 13:24.000 --> 13:29.760 B folio as well and we need to get rid of all the mentions of B page and make sure we're 13:29.760 --> 13:39.520 pointing to B folio. There's a couple of other examples like that but that's the kind of thing 13:39.520 --> 13:46.080 we're talking about. We need to remove the uses of folio arrow page so one of the things I did to make 13:46.080 --> 13:51.840 this all easier is that inside the definition of struck folio it says oh yeah there's a struck page in 13:51.840 --> 14:01.680 here as well so it looks like the struck page is within the struck folio. So and that was always a 14:01.680 --> 14:09.360 deliberate sign that this was code that was going to have to be properly converted at some 14:09.360 --> 14:19.680 point in the future. So I made it easy to grip for and we've been getting good progress 14:19.680 --> 14:25.440 I'm getting rid of uses of folio page. This was particularly handy for file systems because there 14:25.440 --> 14:34.080 was so many interfaces that file systems used that as we progress through the folio conversion 14:34.080 --> 14:39.440 we needed to have this kind of easy escape hatch like oh this this is all stuff that hasn't 14:39.440 --> 14:45.600 been converted yet let's just convert back to struck page and we'll deal with it later and we're 14:45.600 --> 14:51.760 getting to a point where that's just about finished almost every API as a file system wants to use 14:51.760 --> 14:58.320 has already been converted to folio there's some weirdo stuff like FScript that still hasn't been 14:58.560 --> 15:05.120 converted and that's just something that needs to happen and it's something that I haven't done yet 15:05.120 --> 15:09.360 because I'm not all that comfortable with FScript I don't know the code that well and I'm just 15:09.360 --> 15:16.720 going to have to learn it. The page pull out a key to need to be split out of it so I'm struck 15:16.720 --> 15:23.280 I just talked about that a few minutes ago and then the months we've done all of that we can 15:23.520 --> 15:30.720 we can dynamically allocate all these struck folios lab CS mallet pts page will 15:30.720 --> 15:37.120 I'm calling all these things Mendesx generally so that memory descriptors name is hard 15:37.680 --> 15:44.320 particularly since there's there's no struck Mendesx everything has its own unique type 15:44.400 --> 15:51.040 so we don't really need a good name we just need a name that we can all use 15:52.160 --> 15:58.080 when we need to use it or should we rarely so is everybody happy with this 15:59.920 --> 16:03.120 we're making struck page smaller so we're getting memory back sometimes I could book 16:03.120 --> 16:07.120 or make more money I mean so that we can charge less per virtual machines 16:09.680 --> 16:13.360 struck pages now rarely modify when we get to the 32 byte struck page 16:13.520 --> 16:22.400 very little it gets modified at allocation and free but at runtime while the struck pages in use 16:23.360 --> 16:30.480 we don't often modify anything so we don't have this problem with interfering with neighbors 16:31.680 --> 16:36.800 and particularly if you're doing large allocations which was kind of the whole point to the 16:37.120 --> 16:42.800 of the folio project was to make it easier for the page cache when in my most memory to do these 16:42.800 --> 16:52.720 large allocations the pages that you're modifying will also belong to you so you're not 16:52.720 --> 17:00.240 interfering with anyone else but it's all your stuff anyway so that's satisfied the first two crowds 17:00.880 --> 17:07.360 the third point of view that but I want to add struck stuff to struck folio so that's okay you 17:07.360 --> 17:11.360 can add you can add stuff to struck folio and it won't affect struck slab and it won't affect 17:11.360 --> 17:15.920 struck page pool well vice versa you can grow struck slab and not care about how's that going to 17:15.920 --> 17:20.720 affect struck folio because the real problem growing struck page was it affected everybody 17:20.720 --> 17:26.480 affects the entire system and now everyone's got their own allocations you only have to justify 17:26.480 --> 17:31.600 it to the other users of struck folio or to the other users of struck the SM allock which is 17:31.600 --> 17:37.600 actually an easy SM allock so the the the the the the the last radius of your change has has 17:37.600 --> 17:44.240 much decreased which makes it much much easier it's that a good thing I don't know probably 17:45.280 --> 17:53.840 I mean people want to do these things for reasons right so okay it's all the downsides here 17:53.840 --> 18:00.480 well at least for the 2025 edition if you're still doing order zero or order one allocations 18:01.360 --> 18:06.880 we're going to use more memory in total so we're going to have like a 96 byte struck folio 18:06.880 --> 18:14.880 and let's say we're doing order one allocation two 32 byte struck pages such 64 byte plus 96 18:14.880 --> 18:22.080 by such 160 bytes which is more than the 128 bytes that we use to use so that's the net loss 18:22.160 --> 18:28.720 but once you get up to doing order two allocation you're using less memory in total so that's a win 18:31.840 --> 18:36.400 the moral of the story is that if you know a file system that isn't using the 18:36.400 --> 18:40.720 isn't large folio enabled yet you you should you should talk to the fastest and developers and say 18:40.720 --> 18:46.080 why haven't you done this yet because I tell them that and they're listening to me so 18:47.040 --> 18:51.200 I shouldn't say so many many many many files systems have now been converted to use 18:52.000 --> 18:57.040 not just folios but large folios and more are in progress 18:58.240 --> 19:06.320 the fuse is particularly interesting because they did it and then they found a performance problem 19:06.320 --> 19:13.920 and so they've had to undo it at least temporarily which is unfortunate but these things happen 19:16.960 --> 19:22.800 so one thing that might make people unhappy is that we may be getting cash misses when we go 19:22.800 --> 19:29.680 from pages to folios and and back again there's not necessarily going to be a lot of 19:29.680 --> 19:37.840 call to be going from pages to folios I'll give you an example when when when you do an ODIRECT 19:38.320 --> 19:44.880 I know we have this thing inside the kernel called get user pages so you say hey I want 19:44.880 --> 19:50.400 I've got this trunk of my address space that I want this this this set of memory addresses I 19:50.400 --> 19:56.000 want to do I owe to and inside the kernel we walk the page tables and we say oh here here with 19:56.000 --> 20:04.240 the struck pages here here here are the page table entries for that range of addresses and we will now 20:04.320 --> 20:09.280 get what will now get the struck pages for each of those and we will convert those into the 20:09.280 --> 20:14.560 folios and then we'll send it off to the file system and so that's going to be slower because 20:14.560 --> 20:18.560 there's going to be this this in direction right we're going to take one extra point of 20:18.560 --> 20:23.040 dereference to do that and maybe that's going to be a problem and maybe it isn't because you're 20:23.040 --> 20:33.920 doing the IO maybe it's just not the big deal now what what may make some of these things better 20:33.920 --> 20:38.400 is that we can actually shrink struck page for it with more work we can get more benefit so we can 20:38.400 --> 20:43.920 go to 16 bytes relatively easy I think that's something we get done in 2026 I don't think we have 20:43.920 --> 20:52.720 time to finish off in 2025 but we can certainly you know try hard going down to 8 bytes is going to 20:52.720 --> 20:59.040 be really painful and may not actually be worth it because there may just be so many places that 20:59.040 --> 21:08.000 need to be changed that we don't want to do it I mean all of these things may not be worth it right 21:08.000 --> 21:14.480 we may we may get to it and say hey the performance loss that we get by doing the page to 21:14.480 --> 21:20.640 folio conversion in the back end this whole thing just isn't worth doing and that would be very 21:20.960 --> 21:27.760 sad I don't I think I don't think it's going to be a problem or you know wouldn't be trying but 21:27.760 --> 21:33.440 if we if it does turn out that we just need to give up on that side here well at least we've made 21:33.440 --> 21:39.760 the kernel a little bit smaller a little bit cleaner we we no longer have quite so much type confusion 21:39.760 --> 21:49.680 and we'll have a simpler to understand kernel and that will be worth it so I I use the kernel 21:49.680 --> 21:57.920 newbies wiki as my development notes so I just just other people can see what I'm working on 21:58.640 --> 22:04.720 that's the current page I'm using for for this chunk of it this has not been a one-man project 22:06.080 --> 22:10.640 there's about 14 aims on that list and I've probably missed a few people who have helped out 22:11.120 --> 22:21.520 throughout this project it it it has been progress for about eight years we've we've accomplished 22:21.520 --> 22:28.160 really quite a lot in that time I mean we're we're not on the real time when it's 20 years 22:28.160 --> 22:36.000 plus kind of schedule but we've got all the code merged already and you know we've really been 22:36.000 --> 22:41.200 structuring it that way that every step along the way there has been some measurable benefit 22:41.200 --> 22:49.360 to somebody to some workload I mean we we got a doubling in right bandwidth of NFS for example 22:49.360 --> 22:55.840 and you know that does that does just an amazing performance improvement but I want to thank 22:55.920 --> 22:59.520 all of these people and I want to thank you for coming to the talk 23:08.640 --> 23:16.160 all right questions hi so in the 32 by the version of the struct there's a memdesh 23:16.160 --> 23:21.600 and a private if the future intent is to shrink it even further why wouldn't you make the second one 23:21.680 --> 23:27.120 online long reserved and not permitted to be used in first place you already have 23:27.920 --> 23:36.880 things in mind that users will are going to need that for yeah there's a bunch of different ways 23:36.880 --> 23:41.760 we can get to a 16 by the struct page the important thing is it's not worth going to a 24 by the 23:41.760 --> 23:50.080 struct page we want to keep it a pair of two so we need to get rid of ref count and then we can 23:50.160 --> 23:56.320 get rid of I think flags I think we're going to get rid of ref count and flags and just have 23:56.320 --> 24:03.200 memdesh comprise that there's a bit more to it right I I've been I've been it's been simple 24:03.200 --> 24:11.200 flying things here but I think we can get to as 16 by easily if we get to 8 by it's just memdesh 24:11.200 --> 24:17.040 right that's the only thing in it and memdesh is actually a types point so the the bottom 24:17.040 --> 24:22.560 four bits are used to know what kind of memdesh we're pointing to and then the flags or the flags 24:22.560 --> 24:27.360 move it into the memdesh there's a lot to it and it's actually all up all on the the wiki page and 24:27.360 --> 24:33.840 a lot of detail so I take a look and if the wiki page doesn't explain it properly then drop me an 24:34.000 --> 24:37.360 i don't know you no doubt. 24:39.840 --> 24:44.080 Ok let's see how we do 24:50.560 --> 24:59.840 Well we're just a person below no it's alright ok yes wands simple question if we share one page 24:59.840 --> 25:05.480 which track between one holo can, I would say, 25:05.480 --> 25:09.560 CPU user pages for tracking access. 25:09.560 --> 25:15.800 And we have follow with one page, physical page, 25:15.800 --> 25:18.480 hot and other code works. 25:18.480 --> 25:21.520 It happens, in this case, all follow will be marked 25:21.520 --> 25:26.200 as hot usage or do we have a chance to split 25:26.200 --> 25:30.800 and follow to hot and cold and safe memory. 25:30.800 --> 25:34.240 OK, yeah, that's a great question. 25:34.240 --> 25:37.440 It's more part of the folio project than part of the Mendes project. 25:37.440 --> 25:42.360 But this is kind of the trade-off that we're 25:42.360 --> 25:46.080 choosing to make that we have decided 25:46.080 --> 25:50.920 that it is not worth managing memory in four kilobytes 25:50.920 --> 25:51.280 trunks. 25:51.280 --> 25:55.480 Four kilobytes is just too small to be managing memory in. 25:55.480 --> 26:01.760 And that means that we're trading off the precision 26:01.760 --> 26:05.360 of managing 4K for the bandwidth 26:05.360 --> 26:10.040 of being able to manage in 16K, 64K chunks. 26:10.040 --> 26:16.880 I think it is an excellent improvement to try 26:16.880 --> 26:20.200 and detect this kind of situation where we 26:20.200 --> 26:23.600 have made the wrong decision, where we 26:23.600 --> 26:27.960 have the situation that we should not have merged these multiple 26:27.960 --> 26:31.040 pages together into a single folio and track them individually. 26:31.040 --> 26:33.400 And we have the flexibility to do that. 26:33.400 --> 26:37.360 We just don't have a mechanism to do that today. 26:37.360 --> 26:38.880 We can split folios. 26:38.880 --> 26:41.280 That part works. 26:41.280 --> 26:44.600 But we don't have the system in place to say, 26:44.600 --> 26:49.240 to know that has happened and do something about it. 26:49.240 --> 26:50.440 And if somebody wants to do it, that will 26:50.440 --> 26:52.320 be a fantastic improvement to learn next. 26:52.320 --> 26:56.560 And I think I hope somebody takes on that as a project. 26:56.560 --> 26:57.920 And that actually works. 26:57.920 --> 27:03.120 Because it's always been in my mind that something we should do. 27:03.120 --> 27:04.760 It's just I don't have time. 27:04.760 --> 27:08.200 And I don't necessarily have all the best knowledge 27:08.200 --> 27:11.120 about how we could do it, what the right way to do it is. 27:11.120 --> 27:14.080 So yes, you've raised some excellent points. 27:14.080 --> 27:18.400 And I hope somebody does do that. 27:18.400 --> 27:19.000 All right. 27:19.000 --> 27:20.480 Well, after time, thank you very much. 27:20.480 --> 27:24.320 APPLAUSE