WEBVTT 00:00.000 --> 00:10.080 Hello, welcome everybody in the distribution day room for people who were in the preface talk, 00:10.080 --> 00:15.520 you know what I was repeat for the newcomers. Here we go. First let's start with some 00:15.520 --> 00:21.200 safety instruction in case of fire or whatever would happen. Nothing will happen. We have 00:21.200 --> 00:25.760 the exit door at the back, first in one exit in the front or on the wings. I don't know that's 00:25.760 --> 00:32.960 for the planes, sorry. For the rest, if you want to leave the room just with quietly, 00:32.960 --> 00:37.760 if possible, because both the doors and the seat are producing a lot of noise and it can 00:37.760 --> 00:43.200 disturb the audience. For the human section at the end, raise your hand, wait for the microphone 00:43.200 --> 00:48.560 to show up so that everybody can listen and hear your question because it's also recorded in 00:48.560 --> 00:55.760 the screen. So that will be correctly captured. And now I will let the stage two, 00:55.760 --> 01:00.960 a couple of sceneries that I just told you before that I can call you Carlos, because we are all friends 01:00.960 --> 01:07.120 here. And so the topic will be fixing CV and DBM almost everything you should know about it. 01:07.120 --> 01:18.480 Cool, so hi guys, thanks. I feel the slamer is here. The first one, the stock is 01:19.120 --> 01:26.240 mostly targeted for end users and people want to know a bit more about DBM and how we manage 01:26.240 --> 01:33.920 vulnerabilities and if people are safe using DBM stable, yeah, like they are, but I'm going to go through 01:33.920 --> 01:40.160 it today. If you guys are after something more specific about how we do that in DBM and like 01:40.160 --> 01:46.960 the DBM specific things, I would recommend you to watch some while talk, unless you're a 01:46.960 --> 01:54.160 debconf, which he goes in a lot of more details on our specific process. And the second disclaimer 01:54.160 --> 01:59.600 I have to do, I'm not actually from the security team, so security team is a team inside DBM, 02:00.400 --> 02:11.760 but I've worked with DBM, I've fixed CV is on DBM, so actually my last year thesis, 02:11.840 --> 02:18.400 last year to graduate, based on a case study on how we do CV management on the DBM, that's why 02:18.400 --> 02:25.120 I have this stock here today. So again, this slides are available, maybe everyone can see, 02:25.120 --> 02:33.280 I'm not sure, on people.debn.org is slash tilde Charles, which is my nickname. Usually people 02:33.280 --> 02:40.880 call me Charles, so feel free to. Today we are going to like go through some things. The first 02:40.960 --> 02:46.560 it's going to be a bit of introduction. I want to present myself and I also want to know a bit 02:46.560 --> 02:53.680 about you guys. Then we'll talk a bit about the DBM project, then CV is and how we deal with CV is in 02:53.680 --> 03:00.080 DBM and then how we fix those CVs in DBM and we'll have some diagrams that hopefully will help you 03:00.080 --> 03:05.760 to understand everything. And hopefully I will have five or ten minutes for questions. 03:06.480 --> 03:14.000 So let's get into it. I'm Charles, I'm a computer engineer from Brazil and that explains 03:14.000 --> 03:24.240 why the long lives. I don't like this code. Back in university, those two topics, 03:24.240 --> 03:32.320 infos that I can free software were like my passion and see it. I did a lot of work with other 03:32.320 --> 03:38.800 students who we created this in Brazil we call extension groups. Like it's made of 03:40.800 --> 03:46.480 of students and we do a lot of lots of stuff. The free software group, this photo is from 03:47.440 --> 03:51.600 said other really is party but I had to like put my identity shirt and go there talking about 03:51.600 --> 03:58.560 DBM because yeah that's what I do. And also I'm a DBM contributor, mostly packaging, so 03:59.040 --> 04:05.200 I help to maintain curl, I do maintain neomot and some other packages and some other team 04:05.200 --> 04:11.840 maintain packages. But I also try to help on localization so getting DBM into Brazilian 04:11.840 --> 04:18.000 Portuguese because if we want to be the universal operating system, we need to be available on 04:18.000 --> 04:24.400 every language in the world. So I'm also a DBM developer and the first time I attended a 04:24.400 --> 04:29.840 DBM conference which is our annual conference was 2022 in Kosovo and I'm there. 04:32.400 --> 04:39.840 And today I work in embedded systems, companies which is called turrets and I do embedded stuff 04:39.840 --> 04:48.160 there related to DBM but not exactly. So now it's time for questions for you. So if the answer is yes 04:48.160 --> 04:55.200 you please raise your hand. Is this your first time hearing about DBM? Hopefully no one. Oh okay, 04:55.200 --> 05:02.720 one. That's nice. I get to introduce DBM to another person so it's nice. Have you ever used 05:02.720 --> 05:11.840 that in? Perfect. Have you ever installed that in so I would say so. Yeah. Okay. Do you know what 05:11.840 --> 05:22.000 CVs are? Okay. Have you ever seen a CV? Okay so I will leave in. You know everything. So 05:24.080 --> 05:31.760 okay so do you know how DBM fixes CVs and the process? That was the last question. If you all 05:31.760 --> 05:39.440 guys knew it, so let's talk a bit about the DBM project first so we can like get a sense of what it is. 05:40.400 --> 05:47.040 DBM project is a free software. It's like the operating system that we developed. It's a Gino Linux 05:47.040 --> 05:53.440 based distribution basically and it's a free software in the sense that we have a social contract 05:53.440 --> 06:01.360 that every contributor abide by when they are contributing to DBM and that specifically social 06:01.440 --> 06:09.760 contract says DBM will remain 100% free so that's on our core and also we have some our 06:09.760 --> 06:17.360 DBM free software guidelines that dictates what we consider free software and what can be distributed 06:17.360 --> 06:26.800 and maintained by DBM in our mirrors. Usually we call it DFSG. It's a collaborative project so 06:26.880 --> 06:32.320 we have developers from all around the world. I think for sure more than 1,000 people. 06:34.160 --> 06:40.080 And as with 1,000 people from all over the world we need some sort of constitutions so things don't 06:40.080 --> 06:48.480 get messy. But it's also a volunteer based and this constitution was developed in the 90s through 06:48.480 --> 06:55.920 some discussions in my list actually lots of months of discussions on mailing list but it 06:56.640 --> 07:03.600 worked out quite well. And also on our social contract we say that we don't hide our problems. 07:03.600 --> 07:09.680 So almost everything that is transparent to the users you can like go to our mailing list where 07:09.680 --> 07:16.400 discussions is happening. You can see what people are discussing on IRC channels and also our 07:16.400 --> 07:24.640 infrastructure is like the rules to create them. So the puppeteer I will for the name of the other 07:25.280 --> 07:30.480 our restrictions are public so you can see everything that's happening in that bin. 07:32.480 --> 07:38.000 Our goal is to release the next table version always and take care of the current one but 07:38.960 --> 07:46.160 our end goal is to always release this new stable version and currently it's book 07:46.160 --> 07:53.680 Army. It was releasing 2023 which means that this year we will have a new stable release. 07:54.560 --> 08:01.040 How do you get there? We freeze our development. Basically we have a development release which is called 08:01.680 --> 08:08.320 a stable or seed. Our new patches get uploaded there. Oh I didn't tell you but I'm going to 08:08.320 --> 08:15.760 lie a bit for you today because I don't want to make it very complicated. So don't trust me 100% but 08:17.120 --> 08:23.920 our new patches go to a stable which is our development release and after they 08:24.720 --> 08:32.240 pass some automated tests that we call auto-pkg tests and other stuff and after a few days they 08:32.240 --> 08:38.240 migrate to our testing release. This testing release is going to be the one frozen and release 08:38.320 --> 08:46.640 does the new stable release. Things happen quite a lot on those two. These are like the 08:46.640 --> 08:53.280 stable and the testing release but after the release is done and everything is fine we can release 08:53.280 --> 09:01.040 an inversion. We have the release of bookworm and previous we had the old stable release which was 09:01.120 --> 09:07.680 blue eye and blister and so on and so forth. Well let's talk a bit about the stable because 09:07.680 --> 09:14.080 it's frozen time in the sense that it's shipped with the upstream versions that were available 09:14.080 --> 09:22.320 at the time of the freeze. So for bookworm that means upstream software from 2023 and we don't 09:22.320 --> 09:26.720 want to introduce any big changes on the stable version because people rely on the behavior 09:26.720 --> 09:32.480 in there so we don't want to break an API or something like that or change the behavior of 09:33.200 --> 09:38.480 the curl software because people are writing scripts with it so we don't want to change a lot 09:38.480 --> 09:44.960 of stuff there but we need to change eventually because it might have a severe bug that we 09:44.960 --> 09:52.000 discovered only after the release. So we have this proposed updates mechanism where we are going to 09:52.160 --> 09:58.400 queue up proposed updates and eventually they are going to be released as the new point release 09:58.400 --> 10:07.440 of that in. Usually we use the proposed updates to fix bugs but there's also another need 10:07.440 --> 10:17.280 which is the security stuff. For a severe bug a severe vulnerability sorry we don't have the 10:17.280 --> 10:23.760 security feed. The security feed is enabled by default on all automated installation and user-based 10:23.760 --> 10:31.920 installation. So everyone has access to it and as soon as the package hits it it's available to everyone 10:34.560 --> 10:42.240 as long as you do an APT update and APT upgrade or you have the automatic updates enabled. 10:42.800 --> 10:49.680 So everyone has access to the security feed by default and as quickly as possible but there are 10:49.680 --> 10:58.080 some of them that I not that severe for that we use the proposed mechanism update that I've talked 10:58.080 --> 11:06.800 before. And basically every two months we release a new point release so everything that is 11:06.800 --> 11:15.440 queued on the proposed update to become the new point release. So let's briefly talk about the 11:15.440 --> 11:23.520 CVE and what they are. Basically it's a bit of overloaded word but the CVE ideas like this global 11:23.520 --> 11:32.160 identity fire that's unique for each vulnerability and it follows this format CVE ear and a unique 11:32.320 --> 11:41.680 number for that year. The CVE itself contains more metadata so it has like the title which 11:41.680 --> 11:52.000 is an issue the CVE published in the description and it is basically the main data source word 11:52.000 --> 11:59.040 wide for a CVE management and security stuff. I think so to consider it's a crowd source effort 11:59.040 --> 12:05.440 so a lot of people working on it they are mutable so you have to to consider that things can 12:05.440 --> 12:11.680 change during the last time of that CVE so be careful to that and also sometimes it can contain 12:11.680 --> 12:17.280 misleading information like we are not perfect made someone missed a variation that introduced 12:17.280 --> 12:25.360 the vulnerability or something so it needs to be updated. How can I get a CVE because that's an 12:25.360 --> 12:34.000 important question. You get a CVE contacting a CVE number of authority or CNA. I said look I found this 12:34.000 --> 12:42.240 thing I think it's a vulnerability can you issue a CVE defer it. They work in a higher right cool system 12:42.240 --> 12:49.120 so we have the top level route that is in my trip and we have some sub-SUNIA CNAs attached to it. 12:49.120 --> 12:56.240 I have the route CNAs and then I have some more sub-SUNIAs attached to these ones. I have to say that 12:56.240 --> 13:04.400 Debion is one sub-SUNIA so we have the power to assign CVE IDs but we only do it for Debion 13:04.400 --> 13:10.640 and for specific things for Debion so if you find a vulnerability on I don't know if I 13:10.640 --> 13:17.600 thought you shouldn't report to us you should report to probably there's an open source 13:17.600 --> 13:25.920 CNA that's more general we should report to. But there are a few things that every CNA has a 13:25.920 --> 13:32.880 poor CVE IDs that they can issue and they can grant this IDs as they see fit but they can be 13:32.880 --> 13:41.200 disputed by other people so also be careful about that. Well we know about CVE is we know a bit of 13:41.200 --> 13:51.360 Debion now let's see how Debion deals with CVE's. The main people that are going to take care of 13:51.360 --> 14:00.720 this stuff is the security team currently they are the security teams composed by 10 people those are 14:00.720 --> 14:08.800 the members they are delegated by our project leader and these are the members per se of the 14:08.800 --> 14:14.720 the security team. But the security team has some contributors that gather around and like 14:14.720 --> 14:22.160 try their stuff and help and they also have a assistance that can do this work for them. They have 14:22.160 --> 14:29.840 many many tasks so let's break down a few of them. They reduce our security advisories which are 14:29.840 --> 14:36.720 basically an advisory saying oh look in Debion stable we have this vulnerability it has this potential 14:36.800 --> 14:44.640 impact this this version that we've really fixed it in Debion stable and you should update it 14:44.640 --> 14:53.040 so much possible and there's a lot of data there like the CVE ID the DSA ID and more stuff that you can 14:53.040 --> 14:59.520 follow links and discover things so the security team does that part of writing it. They also manage 14:59.600 --> 15:09.040 embargoed vulnerabilities. When vulnerability is to spread or to dangerous for everyone it's kept 15:09.040 --> 15:17.520 private until multiple parties can agree on a date to turn it public and so during this private 15:17.520 --> 15:24.320 time we call it embargo time so only people that really need to know about it get to know about it 15:24.320 --> 15:31.520 and people try to fix it as so as possible and release the fix before the date that it's going to be 15:31.520 --> 15:40.000 made public. We also have the proposed updates that we might use to fix CVE minor CVE is so the 15:40.000 --> 15:47.200 security team has to manage that too so which version fix what when it's going to be released and 15:47.280 --> 15:55.360 all that. They also handle some special packages a few examples are those big patches that we 15:55.360 --> 16:04.480 can like stay on a fixed version Linux is one of those we update to every stable release that 16:04.480 --> 16:11.360 Greg makes. Firefox trauma where also one of those there are a few more but I'm not going to go 16:11.440 --> 16:19.760 through all of them. They also have to handle the income males because there is a male that you can send 16:19.760 --> 16:25.440 that only gets to the security team and they have to try it because they have received information 16:25.440 --> 16:34.480 of vulnerabilities in there and they also have a private key so it can send encrypted mail to them 16:34.480 --> 16:39.680 and only the members of the security team have access to the private key so only they can 16:40.480 --> 16:48.720 read those emails and they have to manage this key and they also manage our poor ID of CVE 16:48.720 --> 16:56.240 that we can issue and they manage the security tracker that is the most important thing I think for us. 16:58.480 --> 17:04.960 I still found it quite crazy. Why the security tracker is important because it's the thing that 17:04.960 --> 17:13.920 manages that do our vulnerability management. It's basically a gift repo of TXT files following some 17:16.400 --> 17:23.360 default formats for us. They contain information about everything the CVE is on the 17:23.360 --> 17:30.320 Adion, all the CVE is even the ones that don't affect the Adion and it's constantly updated through 17:30.320 --> 17:37.760 the CVE list that Mitre makes available and so we get around ADCVS per day which we need to 17:37.760 --> 17:42.960 try and see oh is this does it affect that end does it not should it fix should it not 17:44.800 --> 17:51.280 and also they receive information for Mitre, these are that open all things and you can directly 17:51.280 --> 17:58.960 mail them through this email address. We also generate a website with that information that you can see here 17:58.960 --> 18:07.600 it's I think this is the CVE for a curl I'm not sure but you can have access at security-tracker.dev.org 18:09.920 --> 18:17.120 and every vulnerability needs to be evaluated and we need to have a call to action on what to do with them 18:19.120 --> 18:26.080 and if we need to to release a DSA obviously anyone can contribute to so you can try 18:26.080 --> 18:32.560 exchange for for the security team they will review your work it can fix the vulnerabilities in 18:32.560 --> 18:37.760 that and it can submit a marriage request for the package maintainer through the BTS or our 18:37.760 --> 18:44.160 GitLab instance which is also and the security team has to make a decision if they are going to 18:44.160 --> 18:51.280 release a DSA or not in the end. So first of these we have basically three options we can 18:51.280 --> 18:59.120 apply the apply the fix and release the DSA this is usually the things that happen for our 18:59.120 --> 19:06.800 severe vulnerabilities so the XE or everything that remote code execution we are going to manage it 19:06.800 --> 19:13.440 and fix it before it's public and then on the day that it's going to make public we also make it 19:13.440 --> 19:17.120 available on our security feed and send the DSA to inform everyone. 19:17.360 --> 19:26.480 For the CVEs of vulnerabilities that are not that important like oh it's a local 19:27.680 --> 19:34.960 a local vulnerability that has to have user interaction to become a problem we don't have time to deal 19:34.960 --> 19:40.480 with everything so we document it we contact the maintainer opening a bug in the bug and 19:40.800 --> 19:45.120 system say oh here this package has a vulnerability you should fix it. 19:46.880 --> 19:52.400 Or sometimes it's just too complicated or involves a lot of stuff we are going to break a lot of 19:52.400 --> 19:57.280 reverse dependencies we've introduced this fix and there is a mitigation available so 19:57.280 --> 20:02.720 the system and the link and like mitigated we just document it and we don't do anything. 20:04.240 --> 20:09.040 We do fix this vulnerability via backparting those changes because 20:10.800 --> 20:16.560 the upstream they we see this information oh you have a vulnerability it's on this part of the code 20:16.560 --> 20:21.520 they just they make a new commit fix it and release on the new version of their software. 20:22.320 --> 20:29.440 For us it's a bit different because we have some other versions that are kind of frozen in time 20:29.440 --> 20:36.960 when the stable release was released so we don't have the flexibility to just grab the new 20:36.960 --> 20:42.800 code and inject there most of the time so we need to backpart those patches so see the difference 20:43.440 --> 20:50.560 see how it fixes the cv and then make the change and release a new them release we are not changing 20:50.560 --> 20:57.600 the upstream release we are just changing the them release so usually that shows you that you have 20:57.600 --> 21:06.640 an updated stable package. The process is pretty straightforward like we we discover the cv 21:06.640 --> 21:14.480 somehow someone infirm does we get from the cv list and then it needs to confirm in fact 21:14.480 --> 21:20.480 in the five effects apply the patches if they don't apply correctly because it's the older version 21:20.480 --> 21:25.120 it's not the same as the new version so the hooks don't apply you have to modify the patch 21:25.120 --> 21:30.720 document this changes because you are changing you want to make sure that everyone knows 21:31.120 --> 21:37.200 what to change and why you change that. Then someone ideally another debent developer or someone 21:37.200 --> 21:45.200 from the security team or someone else will review this backparting changes you have to test if you 21:45.200 --> 21:52.400 actually did fix the vulnerability because it's not just importing the patch to fix it and call it 21:52.400 --> 21:58.000 a day you have to make sure that everything is fixed and then you can submit the fixed package 21:58.080 --> 22:04.240 to them to make it as they won't own them also make sure that you watch for regulations 22:04.240 --> 22:09.600 we try to do that as best as we can because you know we are not perfect sometimes it happens 22:10.880 --> 22:16.880 and the process here for a severe vulnerability the security team is first informed 22:18.000 --> 22:25.680 then it's going to evaluate the that cv so if it's a very severe cv they will fix it before 22:25.680 --> 22:33.760 the embargo is is lift and they might contact the the maintainer via encrypted mail because we 22:33.760 --> 22:41.200 we all have our gpg so we can do encrypted mail it's not always the case but they can contact the 22:41.200 --> 22:46.080 maintainer too okay we are going to change that is it okay for you what do you think about those changes 22:46.080 --> 22:53.840 do you want to help us and then they can have those those encrypted conversations and the security team 22:53.840 --> 23:02.320 will upload to our FTP query to to be for the package to be built with the fix then the package is 23:02.320 --> 23:07.840 built it's going to the security feed when it's time to to lift the embargo and they craft the 23:07.840 --> 23:14.000 BSA and send the mail when the patches are available on the security feed. One thing to note 23:15.840 --> 23:21.200 that all happens on security teams infrastructure it's not the same as our regular build these 23:21.200 --> 23:27.520 so everything that happens in there only the security team has access so we can comply with the embargo 23:27.520 --> 23:37.680 with powder problem for our severe vulnerability. For the the minor vulnerability the security team 23:37.680 --> 23:44.720 just received those informations they evaluate the cv they say okay it's fine the fix is there but it's 23:44.720 --> 23:52.320 it's a minus vulnerability we are not going to fix it ourselves so they don't do contact the 23:52.320 --> 23:57.760 maintainer via the bug tracking system in the area so the maintainer can know that there is 23:57.760 --> 24:06.080 vulnerability and they can fix it when the maintainer wants to fix it here we go through the 24:06.080 --> 24:11.840 vulnerability see the fix try to apply on the package test everything okay it's fine 24:12.560 --> 24:19.760 he's going to upload to our proposed updates kill it's going to get built and it's going to be available 24:19.760 --> 24:26.800 our proposed updates feeds so those those who are like the main ways of doing this this stuff via 24:26.800 --> 24:34.960 the proposed updates which is a regular process that we do to also fix another bug or via the 24:35.120 --> 24:43.360 security infrastructure the SA everything else that that's available to everyone as someone it's 24:43.360 --> 24:52.320 released so I think that's the the main idea I wanted to give you it's a bit difficult because 30 minutes 24:52.320 --> 25:00.400 to talk all that it's way too short I will be here at falls day into 6 7 pm so if you guys want to 25:00.400 --> 25:15.360 talk to me go ahead and do we have questions or comments a lot 25:15.360 --> 25:20.800 you know since the state that the security team is 10 people and you sort of they're managing 25:20.800 --> 25:26.000 the inbox and they're like they're watching the list of them with like the 18 new CVEs per day 25:26.000 --> 25:31.520 so I mean this four ten people even with the two that maintain the security of WNC but this 25:31.520 --> 25:37.520 house a full-time job of four ten people so it's this that we know that they are somehow 25:37.520 --> 25:42.880 working like full-time for the deviant project or it's this that you know voluntary basically so 25:42.880 --> 25:51.360 that is yeah I think it might be a mixed bag of things but I would say in general it's 25:51.360 --> 25:57.040 probably volunteer based I I'm not sure I know I don't know they answer for that but yes I agree 25:57.040 --> 26:02.480 that it should be a full-time job and someone should pay these people to do this because a lot of 26:02.480 --> 26:07.920 companies rely on deviant a lot of servers were in deviant so it would be really nice to have that 26:07.920 --> 26:20.240 support do you have complete separate built environment for all the security 26:20.240 --> 26:25.680 build for all so you have for each platform support in deviant you have your own build machine 26:26.400 --> 26:32.560 yeah I I'm almost certainly about that I'm not exactly sure if it's really really like that 26:33.120 --> 26:40.480 for like X390 or like power PC I'm not exactly sure maybe they share something but I wouldn't think so 26:49.280 --> 26:56.320 the question is can regular user speed CVs or maybe other companies that are used the base 26:56.320 --> 27:02.240 deviant base to customize distribution what what is the first part of the question is 27:03.200 --> 27:14.320 if any regular user of deviant can create the CV it's not that simple but if you think you find 27:14.320 --> 27:20.720 found vulnerability you can mail the security things that look I think this is a vulnerability 27:20.720 --> 27:27.360 can you help me probably they will forward you to the right place because probably it's not 27:27.920 --> 27:34.080 the deviant CNA that has to issue the CV but yeah they they can help you 27:37.200 --> 27:43.040 another question some vulnerabilities were not this ignored are you transparent why they 27:43.120 --> 27:51.760 have been stuck like this yeah I mean like if the the vulnerability is not on the embargo you can 27:51.760 --> 27:57.200 like check the deviant that the deviant that security or c-channel you can mail them you can like 27:57.200 --> 28:03.600 take a look on the security tracker because they also do git commits and in the commit message 28:03.600 --> 28:10.240 the their explanations to why the thing was treated or not or evaluated as a viewer or not 28:10.880 --> 28:18.240 yeah it's it's public for embargoed stuff usually it goes in via encrypted mail these 28:18.240 --> 28:29.520 shows that open wall so it might not be that transparent to us do you have more questions 28:29.840 --> 28:39.200 hi so heavy from the almalinux build system team and I've been working on erata stuff 28:39.200 --> 28:45.200 recently and I'd like to talk with you later if you don't mind cool cool yeah okay we think I'll 28:45.600 --> 28:59.840 later so I think oh okay another one last question isn't it sometimes more risky to backport 28:59.840 --> 29:06.640 and to just actually put the major version of some software project let's say stay as a project 29:06.640 --> 29:12.800 like true yeah as I said there are some some software that does not completely follow this 29:12.800 --> 29:19.920 in deviant so like Firefox the kernel from you a few libraries also pick up new versions like 29:19.920 --> 29:30.160 I have my open system might be one of those but we do have a lot of automated tests going on 29:30.160 --> 29:38.560 this so ideally we can try to catch up these regressions yeah I mean like it's not a perfect 29:39.200 --> 29:49.440 sometimes we will make them steak but yeah I mean like it's a topic for another whole top about this 29:51.840 --> 29:56.720 time is up so thank you everyone and I'll be here