WEBVTT 00:00.000 --> 00:14.440 Hello everybody. Good morning. Thank you for your help. Hello. So, we'll talk about 00:14.440 --> 00:20.800 reproducible bits. We are first to brief history and then we have some status reports about 00:20.800 --> 00:29.880 various distros. But first about you, who knows about reproducible bits? Okay. 00:29.880 --> 00:38.880 Then we can go home. Great. Who contributed or contributes? Yeah. Thank you. 00:38.880 --> 00:44.080 Who knows? We produce a bits. It has been around for more than 10 years. Probably there was 00:44.080 --> 00:52.480 to no. For 30 years. But I don't explain what it's talk. And you know, an S-bomb is basically 00:52.480 --> 01:09.680 built in for files as we designed them in 2014. About us, K-P. Hello. My name is K-P-CYD. I contribute 01:09.680 --> 01:17.560 to multiple distributions, like little dust for deviant, alpine and actually looks. That's 01:17.560 --> 01:23.960 been most active. I've been doing security research for 10 years and I created like a website 01:23.960 --> 01:31.080 to index source code in response to this exact incident last year. There you can compare 01:31.080 --> 01:38.120 tab or file there on different distros, what source. Yeah. So, professionally, I work for 01:38.120 --> 01:46.760 Red Hatz. In my hobby, I do art stuff. I'm a holder. I'm old. I installed deviant for the first 01:46.760 --> 01:52.840 time 30 years ago. And I work on reproducible. It's in 10 years. And we try to make all 01:52.840 --> 02:00.200 software reproducible, all free software. And it's not only us, but many people have been 02:00.200 --> 02:09.960 involved in this. And yesterday, 11 years ago, Lunar gave a presentation here about reproducible 02:10.120 --> 02:17.960 deviant back then. Then 10 years ago, also yesterday, Lunar and me gave this talk. I had lots 02:17.960 --> 02:28.840 of fun. And then sadly, last year, November, Lunar died. Yeah. Lunar made a big difference, as you 02:28.840 --> 02:38.120 will see. And we are here for many projects. I'm not going to list them all here now. But 02:38.120 --> 02:44.760 so the introduction, probably all know, source code is free available. Most people installed 02:44.760 --> 02:49.880 here in store binaries. And no one really knows whether they really correspond to each other. 02:51.320 --> 02:58.600 And as a result, they are very supply chain detects. And if documented this by now very well, 02:58.600 --> 03:03.640 always a bit livable. It will be reproducible given the same source code bit environment and 03:03.720 --> 03:09.960 bit instructions. Any party can recreate bit by bit identical copies of the specified artifact. 03:11.880 --> 03:17.000 And this is a slide from 10 years ago. And we already had the same thing. 03:18.600 --> 03:24.600 And our mission is to enable anyone to independently verify that a given source and produce 03:24.600 --> 03:29.480 bit by bit identical results. That's what we do. And that's easy to explain, right? 03:30.040 --> 03:37.240 And they are important building block that don't make software more secure, but they make the supply chain more secure. 03:37.240 --> 03:47.560 And unsecure software still is unsecure, it just changes that. And as this is very difficult to 03:47.560 --> 03:56.120 understand, we came up, what does that even mean? Can you explain this to your co-workers who are not 03:56.200 --> 04:02.760 involved with this? So our new slogan is just enabling supply chain security. That's what we do. 04:02.760 --> 04:16.280 Everybody gets this. So we produce a bit by 24 or 25. And we have been widely understood. 04:17.000 --> 04:22.040 We have all these resources. We have the documentation. We have scientific publications. 04:23.000 --> 04:29.240 We can look up this because the stock is very brief. And even the White House has understood 04:29.240 --> 04:35.560 reproducer this. They made a memo in 2021 about supply chain security. And in the 04:35.560 --> 04:43.960 additional measurements, they also explain a bit by bit of abusive bits. So how did we get there? 04:44.920 --> 04:48.920 Money and what's no? 04:49.640 --> 04:56.040 Why money? In 2011, Bitcoin made their client reproducer work. That time Bitcoin was 04:56.040 --> 05:00.600 worth $4 billion and they were afraid. If some backdoor client would steal the money, 05:00.600 --> 05:05.320 they would be blamed. And so they made it reproducible. And that was for the first time 05:06.520 --> 05:13.400 we learned that software can be made reproducible. And Snowden was a huge motivation in 05:13.400 --> 05:20.440 many things. And set Perry, Mike Perry, and set Shuren. And others made the tour brother reproducible. 05:20.440 --> 05:24.920 And that's firefox. And firefox at that time was 50 mega by binary or something. 05:24.920 --> 05:31.480 And every bit reproducible was pretty neat. And how did we really get there? 05:32.760 --> 05:42.280 Lots of work by many, many people. Yeah, it was the reproducible. Some of the 2019 in Maracas. 05:43.800 --> 05:51.480 And so in 2013, Lunar hosted a brainstorming meeting at Debt Com 13. And another Debt Com 14. 05:52.280 --> 05:59.560 The Debt Conference. And there were 30 people there spontaneously planned. And that's how it started. 06:00.520 --> 06:09.240 Then Lunar gave this talk at first time. And then Lunar and others, the first patch is for deep 06:09.800 --> 06:14.360 decades. Sorting fixes that the contents of the Debt Com 14. The Debt Com 14. 06:14.360 --> 06:19.240 The Debt Com 14. Sorted correctly. And we designed bit in four files where the building 06:19.240 --> 06:28.040 for files store the sources, the defense and the output. And in 2014, I started to rebuild 06:28.040 --> 06:34.280 100 Debt Com 14. And compare this and that became the setup where we test all the 06:34.360 --> 06:41.480 packages later. And Mike Perry and Citroen gave this presentation at the Chaos Communication 06:41.480 --> 06:48.200 Congress in 2014, explaining the problem space very well. Back then, there were not solutions, 06:48.200 --> 06:55.240 but rather it describing the problem. And this is the presentation. And most people think 06:55.240 --> 07:02.840 they are trustworthy people. And why should somebody not trust them? And I am the developer 07:02.920 --> 07:11.000 I built it myself. I am stored it. I know what I am doing. But developers are really good targets. 07:13.240 --> 07:23.640 They explain very well that if nation states have an interest to get the benefit of attacking 07:23.640 --> 07:32.120 developers is very high. And they also showed this vector like this was a CV and 07:32.520 --> 07:39.880 SSH. And one bit was different in the code. It was greater or greater equal. And in the binary, 07:39.880 --> 07:46.440 it is a single bit difference. And the binary is 500 kilo wide. And one bit can make this difference. 07:47.560 --> 07:53.800 And they also had a nice proof of concept, the kernel module. So when a source code was 07:53.800 --> 07:58.520 viewed with an editor, it would show this source code. But if the compiler would use it, 07:58.520 --> 08:05.720 that would different source code compiled. They showed that vector also, that's by the motivation. 08:07.320 --> 08:12.920 And then at first, 10 years ago, Lunar and I gave a talk trying to invite the whole free software 08:12.920 --> 08:20.200 work to get involved. And Lunar gave a presentation at camp showing many problems like timestamps 08:20.200 --> 08:25.720 and sorting issues and whatnot. I think 50 issues and 50 solutions for this problem. 08:26.680 --> 08:33.800 And we had the first, we put into the summit in Athens. We are more than debut and people 08:33.800 --> 08:38.760 gathered and that was really the kickoff. And we wrote source state, epoch the spec. 08:40.360 --> 08:47.800 And the Lunar Road, the Bindiff. And the Bindiff is now difficult. And there are more than 80 08:47.800 --> 08:54.200 contributors. So it is really good. Explain this in a moment. And the common reasons for 08:54.200 --> 09:01.000 unreboтоcibility are timestamps. Really timestamps. And build pass and build pass. And for 09:01.000 --> 09:07.560 build pass, we've come to conclude just rebuild in the same build pass. And all the rest. And all 09:07.560 --> 09:14.360 the rest is a lot. But so, and we have some resources. We have this Lunar's talk or general 09:14.360 --> 09:20.840 the documentation and resources has the talks. And it's also easier to show how what makes a 09:20.840 --> 09:25.160 package unreboтоcibility. So we have a Git repository with the unreboтоcibility package 09:25.160 --> 09:32.600 which has many problems and the fixes. And we also counted, we have like over 400 issues, 09:33.400 --> 09:39.160 different types of issues which are all in this Git repository. And so we came up with source 09:39.160 --> 09:44.760 state epoch, build timestamps are meaningless and source that epoch describes the time of the 09:44.760 --> 09:50.280 last modification of the source code in seconds since the Unix epoch. And we did this spec 09:50.360 --> 09:56.680 in 2015. We did one small patch in 2017 and since then it's stable. And it's adopted 09:56.680 --> 10:01.960 as a border by a lot of software like compilers, but also documentations and our rate as 10:01.960 --> 10:07.960 later PDF what not they all. If source that epoch is set, then the current data is replaced with 10:08.040 --> 10:15.320 that. And difficult scope, who uses or has used difficult scope? Are you happy? 10:19.320 --> 10:23.000 So difficult scope tries to get to the bottom of what makes files for packet 10:23.000 --> 10:28.280 a direct list difference and will recursively unpack archives of many kinds and transform various 10:28.280 --> 10:34.920 binary words to what human readable form to compare them. And you can just go to try difficult scope 10:35.000 --> 10:42.680 or upload two things to whatever and it will compare that. That has test text and HTML output 10:43.480 --> 10:51.320 and it supports many five formats. Anything that's way more than by now you can 10:51.320 --> 10:57.400 two directories, two ISOs, two PDF things. I know lawyers who compare PDFs of laws 10:58.120 --> 11:02.520 to see how the law has changed with the first code. There's many use cases and 11:03.240 --> 11:10.600 if you haven't seen it, this is the diff and they're source the diff between two versions. 11:16.280 --> 11:22.520 And then we had these summits, like we had eight I think now. And this year we tried to have a 11:22.520 --> 11:30.280 summit in Vienna, like in September to November something. If you want to attend please contact us 11:30.360 --> 11:37.720 we are happy to have more people there. It's like three days event. And there are many projects 11:37.720 --> 11:43.160 not only distros but also academic researchers companies using it. There are many people 11:44.040 --> 11:53.080 and another picture from Paris summit. And we had funding, we have funded, that's five people 11:53.080 --> 11:58.920 being funded as a soccer freedom conspiracy project. That's Chris Lamp, a very grand cascadian 11:59.000 --> 12:05.400 material. It's a low myself and since November it's KP also is funded. 12:08.520 --> 12:16.120 We need funding for all these work to prepare the summit also. And we are very thankful for 12:16.120 --> 12:22.600 all the funding. And now about rebuilding why we have KP now. 12:22.920 --> 12:32.120 In December 2019, I've been working for about two years in the beginning of the project. 12:34.120 --> 12:38.760 Two of the point of us mostly comparing if we take the source packages and build it twice 12:38.760 --> 12:42.600 in different environments doesn't match, but actually care about the binaries that are in my computer 12:42.600 --> 12:51.080 that actually install through the packet manager. So especially just a scheduler, 12:51.160 --> 12:56.680 there's different projects. One is debris build and for entrepreneurs it's actually 12:56.680 --> 13:03.880 a free throw that Foxbond and Robe. It's been in two or three parts. One is keeps tracks of 13:03.880 --> 13:08.040 research. The other one actually runs the build. And the third one is like conforming it in 13:08.040 --> 13:15.320 current research. For entrepreneurs that's right now about five instances that are run 13:15.400 --> 13:23.480 better than groups. So if each of them kind of confirms the binary, the other did a conspiracy 13:23.480 --> 13:33.000 or it's actually good to use. It's certainly in risk by me and source code is available here. 13:36.120 --> 13:41.480 I've already did a package ring for Arch. You can build it from source code and we're working on a 13:41.480 --> 13:51.000 debut package. This is kind of what it's built around. There's some kind of build input over here 13:51.000 --> 13:57.080 and it's expected to have these two results and then we run this build in our own machine 13:57.080 --> 14:05.640 and we expect to get the same binary. If we don't we throw the diff. It started as a prototype 14:05.800 --> 14:13.080 has been used for five years as a prototype. It's kind of good enough to find issues and 14:13.080 --> 14:18.920 troubles with them. We found some issues that we couldn't find on like the regular CI that 14:18.920 --> 14:24.120 only showed up in the wild. One of them was the group package for example. We didn't 14:24.120 --> 14:28.520 we noticed that if we built on a machine of us booted through like the master boot record 14:29.480 --> 14:34.920 we would get a different binary than when it was booted. That's because the build tested 14:35.720 --> 14:41.960 if there was nothing set it would pick the one that you booted as a default and we found through 14:41.960 --> 14:53.880 as you like in the wild deployment. So this is sort of you of the way of the Arch 14:53.880 --> 15:03.080 in involvement. So we already 10 years ago we started recording created a build info. So 15:03.080 --> 15:08.120 if you build a package we would record the all the installed packages in the build routes 15:08.120 --> 15:13.320 so that we then can recreate this build route and build the package again on a different machine. 15:16.360 --> 15:21.880 After that we still needed to support like source it epoch so we would normalize all the time stamps 15:22.920 --> 15:29.080 and at the time we also made a reproducer tool so this is a small batch break which 15:29.560 --> 15:37.960 allows you to reproduce an existing package in the repository. But for that to be useful we 15:37.960 --> 15:44.200 needed to have an archive of all the past packages we had we have which we normally didn't 15:45.480 --> 15:51.720 yeah we didn't really have the structural archive of all the past packages. So now so that was 15:51.720 --> 15:56.920 some work done so that we were properly organized when you were at it through the repository 15:57.720 --> 16:05.000 and then they could be used to reproduce packages. Then yeah like four years ago we started 16:05.000 --> 16:11.800 this rebuild a build review and since it was a great success so we had I think two B3 machines 16:12.680 --> 16:23.320 with like 100 cores 64g ram and we managed to get at 86% of the core repository I think like 80% 16:23.320 --> 16:30.360 of the actual repository so it was a good first step and in four years later we 16:30.920 --> 16:38.680 KP actually managed to make a minimal container um usually fully reproducible so that I think 16:38.680 --> 16:46.200 that's the base Docker image but now it's no longer reproducible but that's it has progressed 16:46.840 --> 16:56.040 um yeah and now it's a current state we had like 12% so that like the 10% hardest the 10% is 16:56.040 --> 17:02.360 going to take another I know a few years probably to get resolved and one of the big packages which 17:02.360 --> 17:12.440 are like reproducible is like Haskell and the kernel is also a fun topic and yeah oh and I have 17:12.440 --> 17:18.360 some graphs so this is the or the font and I wrote for the Rebuilder D instance so this shows you 17:18.360 --> 17:22.840 all the bad packages to good packages and you can click on the link and get the this was 17:22.840 --> 17:28.120 scope where you can get the build box so it was incredibly useful to treehouse stuff and this is 17:28.120 --> 17:36.120 the recent I think yeah this is a year of the rebuild status and as you can see it has I'm not sure 17:36.120 --> 17:42.120 why recently bumped up to more bad packages but it has been quite stable for some time 17:43.000 --> 17:53.480 now David Kevin yeah we have reproducible potmen or Docker images for that period we have 17:53.480 --> 18:00.120 life images and we have individual packages that was the status like until recently we had the 18:00.120 --> 18:06.040 CI builders where we built twice this trying with maximum variation but we've done the last 10 years 18:06.040 --> 18:14.040 and this is the graph and went up to 96% now and we found lots of bugs related and unrelated 18:14.040 --> 18:23.320 to reproducible which is a nice side effects like really many bugs and we get now testing migration 18:23.320 --> 18:28.280 we fixed this snapshot that we in all which has all the packages was broken for the last 18:29.000 --> 18:35.720 more than four years and was finally fixed last summer and so last September we could 18:35.720 --> 18:42.280 or finally rebuild again David packages but we were able in 2016 already so just I eight years later 18:43.000 --> 18:48.760 and in October we set up reproduced every in that which is a rebuilder the instance 18:50.840 --> 18:57.400 and that is comparing against what David and distributes on FTP debut and all and this is the current 18:57.480 --> 19:05.000 graph now we are there at 86% for AMD 64 i386 is also getting there in the beginning we had 19:05.000 --> 19:11.000 rebuilder D running on i386 but that doesn't work so now rebuilder D is always running on AMD 64 19:11.000 --> 19:16.280 and just the workers on different arts we have arm 64 the hardware has some problems like the 19:16.280 --> 19:21.880 file system is too small the same for arm hf and risk the machines are not powerful enough 19:21.960 --> 19:29.880 and we miss four more release architectures so if you have these hardware or if you have other hardware 19:29.880 --> 19:36.120 please talk to me we are very happy to use the same tool for debut as art Linux it's really great 19:37.960 --> 19:43.000 but the debut service also way more way bigger we have eight architectures instead of one we have 19:43.000 --> 19:48.520 two suites instead of one and we have three times as many binary packages as art so scaling is a bit 19:48.600 --> 19:54.200 more an issue we need more beefy hardware and also there is a package rebuilder from Frederick 19:54.200 --> 20:00.360 P.E which is also exist is written in pison and I would love if somebody sets this up and does 20:00.360 --> 20:07.320 a rebuilder with this software just to test this alternatively if somebody is into pison you might 20:07.320 --> 20:14.440 want to try this more help much work and the difference between theory and plexus for debut 20:14.440 --> 20:23.480 is at the moment 10% we have 96.6 in cia and 86% on reproduced but we only do this in three months 20:26.520 --> 20:32.680 so on how we want to reach 100% in practice 100% is a political decision in nothing technical 20:32.680 --> 20:36.680 because there will always be packages and for debut we can just say this will not get into this 20:37.080 --> 20:43.800 release so we need to change debut in policy we can work around known offenders with a 20:43.800 --> 20:50.840 loudest in the beginning and the goal is still 100% loudest is just to work around to get there 20:53.640 --> 21:00.040 and policy debut in policy was changed in 2017 packages should be built reproducible we hope to 21:00.120 --> 21:06.200 change this year packages must not regress and also new packages must build reproducible 21:07.560 --> 21:13.160 and hopefully in two years must be reproducible then there will be still policy violations 21:13.160 --> 21:20.840 so maybe in 29 or something I don't know and this will be the release team enforcing this and so 21:20.840 --> 21:26.920 I made the past 200% the upcoming debut in releases with whitelist so maybe 40 plus two 21:27.880 --> 21:35.560 in whatever it's eight years I don't know next OS everybody knows next OS is reproducible by design 21:36.520 --> 21:41.960 but that is maybe not true or that is true by the design yes but somebody related reboots 21:43.240 --> 21:49.000 it's a slim Julian he gave a talk here yesterday and published a black block course and he found that 21:49.640 --> 21:58.280 what is it? 91% very reproducible in April 23 so we all use the same software we all get the same 21:58.280 --> 22:03.000 resides because we all have the current problems with the kernel has to be I think it's reproducible if you 22:03.000 --> 22:11.240 build it on one core that's a good work around next OS has also a graph it also goes to 90% 22:11.800 --> 22:18.040 previously there was also a talk at first in 2016 about making freebius D reproducible 22:19.640 --> 22:24.600 the base system we tested on the strength and stability in the stands but it has become 22:24.600 --> 22:32.680 test reproducible and now they tested ports which is packages under the freebius D word 22:33.720 --> 22:40.280 and it's even 80% and then it got stored and just got picked up this month there's now this 22:40.360 --> 22:47.240 zero trust initiative from the freebius D foundation I skip it a little bit where there's other 22:47.240 --> 22:51.160 more than just reproducible there's a five point initiative and they got some funding 22:53.240 --> 22:57.480 so there's looks like this year freebius D will make some huge progress as well 22:58.520 --> 23:07.000 done Benedict Law is testing net BSD and finally net BSD built on Linux is more reproducible 23:07.640 --> 23:15.320 net BSD built on net BSD but the really good part is that 44 to half of the ports a bit less 23:15.320 --> 23:20.840 a bit by bit identical whether you build them on Linux or net BSD the reside is the same which is 23:20.840 --> 23:28.280 pretty neat our BOS is from Bernard Wiederman who normally works on suze and here's been testing 23:28.280 --> 23:33.960 suze since 10 years and it's done in a lot of patches but it's done in a lot of them upstream and he made 23:34.040 --> 23:39.720 a BOS which is a proof of concept where he did lots of hacks like he built Haskell on one core and 23:39.720 --> 23:46.520 he did other stuff that you cannot do in a distro but he achieved a minimum VM images which is 23:46.520 --> 23:51.720 100% reproducible all the packages built from source are reproducible and he did a second thing 23:51.720 --> 23:59.320 which is the DVD with a small graphical UI which is 100% reproducible that is very nice 24:00.280 --> 24:07.640 and Fidora speak enough started to do rebers of Fidora he's also here I'm not sure if he's in the room 24:11.880 --> 24:18.200 and he did mass rebers of Fidora and reached there in 1982 percent reproducible I'm not sure 24:18.200 --> 24:26.280 whether it's CIO rebers but it's a proof of concept at the moment and he also wrote ad determinism 24:26.440 --> 24:30.920 which I didn't mention we have in debut and we have stripped non-determinism where we remove 24:30.920 --> 24:41.960 non-deterministic stuff and he has the opposite they have a matrix standard Fidora so in theory 24:42.360 --> 24:51.160 we are done in practice we have shown that it can be done in theory now we need to close this gap 24:51.240 --> 25:02.280 and those 5% are still crucial or 1% at least 1% for debut it means 370 source packages 25:02.280 --> 25:08.200 so that's a lot because we looked at them for 10 years and didn't find the courses so they're still a bit to do 25:09.880 --> 25:15.240 and many projects support or aim for it to be today that is the huge success 25:15.320 --> 25:22.120 we wanted to change the software and I think we did think that's very good 25:33.240 --> 25:40.520 and the next is finish those last 1% to 5% or 8 and there are some dragons like profile guide 25:40.520 --> 25:49.960 optimizations there's other issues we need more infrastructure and processes and tools 25:49.960 --> 25:55.000 because it's still theory and there's also tool for both arts, Linux and Debian, Debian, 25:55.000 --> 26:03.480 Lepros, Datus, Arts, Depros, Datus which compares your packages but so there's 26:04.200 --> 26:12.040 and so it meets pro level consensus that the project wants Debian has changed policy 26:12.040 --> 26:17.160 much to how the arts processes are what others are, Fidora is a single person that has not 26:17.160 --> 26:19.800 committed so it's not only to happen 26:23.800 --> 26:28.760 yeah and this is again from 2014 it's basically the same today as we have today 26:29.160 --> 26:36.520 but now it's possible back then it was ideas so thank you 26:46.200 --> 26:48.760 do we have time for one or two more questions 26:59.080 --> 27:03.800 yeah thank you for the talk I had a question 27:09.800 --> 27:14.360 you've a closer so what's the reason you're not using QMU or something for 27:15.560 --> 27:20.600 the more out there architectures for the Debian rebuild instead of replying hardware 27:22.200 --> 27:27.640 basically because we started that three months ago and each other added to half the other architectures 27:27.640 --> 27:38.840 two weeks ago and we don't have the powerful hardware to run to Emo and if you have powerful 27:38.840 --> 27:40.360 hardware please stop to me 27:44.280 --> 27:51.960 do the upstream's care I mean can you can you go to I don't know the GHC folks and say hey guys can you 27:52.040 --> 27:58.440 make your stuff reproducible and or will they just like say we don't care most upstream's care 27:58.440 --> 28:04.520 but I know like there has changed yeah there's one two sopah who still owns and there I 28:04.520 --> 28:09.880 know what I'm doing and stuff but most have understood that they probably not completely know what they're 28:09.960 --> 28:19.960 doing for the has got stuff there's been recent patches for the has jealous jealous are set for 28:19.960 --> 28:25.480 the has got stuff there's been recent patches and we've also had cases where projects 28:26.200 --> 28:31.880 announced that they're reproducible like wire got was reproducible on f-roid placed or in 28:31.880 --> 28:36.600 something else and they just announced it on their mailing list but not to us they are just 28:37.000 --> 28:47.320 doing it there is a meta problem of reproducible bits which is now there are containers which 28:47.320 --> 28:54.440 are built on top of debion images and which is still packages and those containers need to be reproducible 28:54.440 --> 28:59.000 as well for various reasons for the same reasons that builds into be reproducible in order to do that 28:59.960 --> 29:06.440 timestamps again coming to play the installation time of a build of a package 29:07.640 --> 29:15.800 messes up with the bit that bit to visibility so other plans for distributions to help with the installation 29:15.800 --> 29:23.000 problem and the time's the thing that partly like this life images usually are installed 29:23.000 --> 29:31.960 packages so there's also work on that yes but it's a different class of problem and so for this 29:31.960 --> 29:37.640 bit out of scope but there's other doing that there's also the related project is good strapping 29:37.640 --> 29:43.560 that the compilers can be good step which is also related but not the same so there's more 29:45.880 --> 29:50.520 projects tackling related problem spaces all right thank you so much 29:53.000 --> 29:55.000 See you