WEBVTT 00:00.000 --> 00:13.960 Hello everyone, welcome to our next speaker, I'm here, who will be talking about 00:13.960 --> 00:15.560 out-of-think open-source. 00:15.560 --> 00:22.000 All right, thank you, thank you so much. 00:22.000 --> 00:24.440 Hello, good afternoon, everyone. 00:24.440 --> 00:27.840 Thank you for lasting this long and being here in this afternoon talk. 00:27.840 --> 00:33.440 I'm really excited to speak with you all today and also kind of live my former 00:33.440 --> 00:38.000 life stream of being a college professor, being at a university, so this is a lot of fun. 00:38.000 --> 00:40.760 So yeah, thank you so much for being here. 00:40.760 --> 00:46.680 I'm a mere Montezary, I'm the Managing Director of OSTIF, that's the open-source technology 00:46.680 --> 00:52.520 improvement fund, and I'm going to talk about security auditing as a practice and the success 00:52.520 --> 00:59.720 stories that we've seen by applying the best practice of security audits, but in a open-source 00:59.720 --> 01:07.400 centric way that is focused solely on, or it's primarily focused on empowering maintainers 01:07.400 --> 01:18.760 and helping maintainers with the issues that they face and fulfilling their security needs. 01:18.760 --> 01:31.000 So security audits are typically, no, are defined as code reviews where experts review code, 01:31.000 --> 01:40.040 an analyzed code looking for vulnerabilities, testing code for logic bugs and other issues. 01:40.040 --> 01:45.840 And according to a lot of research, it is a best practice, both in the software development 01:45.840 --> 01:53.320 space, but really just a best practice in general, to have a independent expert come and review, 01:53.320 --> 02:02.240 validate, and help address issues, because as we have seen, a lot of open-source maintainers 02:02.240 --> 02:09.120 are not security experts themselves, and typically security is not always at the forefront 02:09.120 --> 02:13.960 of development or as at the forefront of the focus of development. 02:13.960 --> 02:22.400 So there's a lot of technical debt and help that especially open-source projects can use, 02:22.400 --> 02:30.060 and by applying the best practice of a independent security audit, we're able to do that 02:30.060 --> 02:35.720 best practice for projects. 02:35.720 --> 02:45.080 So a typical audit process or a more traditional audit process involves kind of a top-down approach 02:45.080 --> 02:54.120 where you have a boss or C-suite or some kind of force working with a audit team, which 02:54.120 --> 03:02.520 lots of times can be big consulting firms, which tend to be more generalists and lack some 03:02.520 --> 03:11.520 of the specific expertise that some projects would need, and working with an internal 03:11.520 --> 03:13.120 software development team. 03:13.120 --> 03:21.040 So essentially the auditee and the funders are part of the same organization, they kind 03:21.040 --> 03:27.520 of have the same objectives, there's a little bit of the stick, so to speak, when it's 03:27.520 --> 03:35.920 your boss, for your company, and this is typically how these processes are. 03:35.920 --> 03:44.280 That being said in the open-source way of what an open-source audit can and should be is 03:44.280 --> 03:51.880 having an open-source centric audit team, which typically works with under-resourced either 03:51.880 --> 03:58.480 solo maintainers or small group of maintainers and then a funding body. 03:58.480 --> 04:05.440 What's really important here is having a champion or somebody to manage this process from 04:05.440 --> 04:13.680 start to finish because unlike the first example, maintainers can work for different companies, 04:13.680 --> 04:20.200 they can have kind of different objectives, lots of times they are very much overworked 04:20.200 --> 04:28.840 and have enough on their plate already, and working with open-source centric audit teams, 04:28.840 --> 04:35.760 so teams that understand the needs that maintainers face is really, really important. 04:35.760 --> 04:41.800 And again, having somebody to manage or champion as I like to say the process from start 04:41.800 --> 04:50.040 to finish is really, really important because when we do that as part of our audit process, 04:50.080 --> 04:56.320 we are solely responsible for carrying the audit from start to finish and being that advocate 04:56.320 --> 05:03.880 for moving it along because as we've seen in other examples, when there is that missing 05:03.880 --> 05:12.160 piece of someone to essentially own the process from start to finish, it can lead to less 05:12.160 --> 05:20.000 than ideal outcomes in audits and this way, having someone, again, solely focused on 05:20.000 --> 05:27.120 completing this work, has been very successful. 05:27.120 --> 05:34.880 So similar to a director of a film, so director of a film isn't necessarily an expert 05:34.880 --> 05:43.000 in costume design or lighting or special effects, but they kind of are in charge of the vision 05:43.000 --> 05:49.920 of moving a film through to completion and kind of play that role as the facilitator 05:49.920 --> 05:58.560 and the kind of, again, champion of the process to create the outcome that is either 05:58.560 --> 06:07.480 the film or in this example, a successful security audit. 06:07.480 --> 06:16.800 So to share a little bit about our experience here, we started about 10 years ago, just celebrated 06:16.800 --> 06:24.480 our 10 year anniversary mid last year, but in the early years, it was very much proving 06:24.480 --> 06:31.240 the idea and proving the concept, so we were able to work on some early stage projects 06:31.240 --> 06:41.440 such as the Veracript Project and OpenVPN to kind of validate this idea that this best 06:41.440 --> 06:48.800 practice of independent security audits can, in fact, apply to open source projects and 06:48.800 --> 06:54.720 the kind of nuances that come in open source. 06:54.720 --> 07:01.520 After a growth period, it was a period of slow but steady growth, we were able to get more 07:01.600 --> 07:12.320 experience, more case studies and examples of applying a security audit to open source projects 07:12.320 --> 07:19.520 and so much success over those years, but it was a great opportunity to really hone in on a 07:19.520 --> 07:27.680 process and understand what the typical needs of maintainers are and how to fulfill those needs. 07:27.680 --> 07:35.200 All while building a network of researchers, advocates, audit teams that are open source 07:35.200 --> 07:43.440 centric, that could fulfill the needs of maintainers and in the last two years, we've seen 07:43.440 --> 07:52.880 significant growth in auditing many more projects, projects like Git, Curl, PHP, Drupal, 07:52.960 --> 08:02.320 to name a few and many more currently in flight that are expected for 2026 and a really interesting 08:02.320 --> 08:09.840 actually, a really interesting thing that we've been doing in more recent years as well is 08:09.840 --> 08:18.800 more holistic security improvements that apply to whole ecosystems and have far reaching impacts 08:18.800 --> 08:25.280 beyond just for example helping one individual open source project with a security audit. 08:26.160 --> 08:32.160 Those results will be coming out this year, so I don't have a lot to talk about on that quite yet, 08:32.160 --> 08:39.440 but it's been a really nice journey over the last 10 years of starting with this idea that we can 08:39.520 --> 08:50.640 apply this best practice to open source, learn a lot of examples of efforts that try to do similar 08:50.640 --> 08:59.120 things to varying degrees of success and learn from those efforts and observed very closely 08:59.840 --> 09:07.040 those efforts and take all of that into really hone in on a process to be a 09:08.080 --> 09:14.800 neutral independent advocate for improving the security posture of projects. 09:17.760 --> 09:27.600 As the title suggests, it's time to audit open source so why now, it's always been of course extremely 09:27.680 --> 09:33.440 important, it's been a problem since the beginning of software, but especially now with 09:34.400 --> 09:41.040 sophisticated and escalating bad actors who are out there to cause trouble, 09:42.880 --> 09:50.240 increased geopolitical tensions, pretty much all I've heard this weekend is the concept of 09:51.200 --> 10:00.320 sovereignty and EU sovereignty and very understandably so as well as the global digital 10:00.320 --> 10:10.880 interdependence that software is developed in. So taking all that into account and the fact that we 10:10.880 --> 10:21.040 have a method, a process and a dedicated organization that is solely focused on doing this 10:21.040 --> 10:31.600 process and doing it well in a way that is effective, that is transparent, it makes now an ideal 10:31.600 --> 10:40.720 time to really scale this work up and do it for multiples of projects than we are currently doing. 10:45.120 --> 10:52.320 As I mentioned earlier, we've been doing this for 10 years now and have marked over 800 10:52.320 --> 10:59.200 security vulnerabilities found and fixed because going back to working with open source 10:59.200 --> 11:07.680 centric teams and interestingly enough, I learned this weekend about some of the first efforts 11:07.680 --> 11:14.000 and doing this kind of work of doing security audits for open source projects and seeing what 11:14.000 --> 11:21.840 those pain points were and how those outcomes were less than ideal for what they originally intended. 11:22.720 --> 11:31.040 One of those kind of consistent factors was again kind of applying that commercial approach 11:31.040 --> 11:38.400 to open source where as working with audit teams that are fixed oriented that will work with 11:38.400 --> 11:46.560 maintainers on proofs of concepts and providing fixes made it so that this process was 11:46.560 --> 11:57.520 much more effective for the project. So because of that, we have almost well over 90 11:57.520 --> 12:05.200 percent fixed rate, all of the high medium and higher risk vulnerabilities we have closer, 12:05.200 --> 12:12.640 much closer to 100 percent fixed rate. So we're not just out there to dump kind of bug reports on 12:12.720 --> 12:19.680 projects, there's already enough groups out there doing that and thanks to AI, that's just 12:19.680 --> 12:25.840 exploding and overwhelming maintainers even more than they already are. So by having this very 12:25.840 --> 12:32.640 maintainer focused approach, by working directly with maintainers and communities on 12:32.640 --> 12:38.640 holistic improvements to software, we've been able to be very effective at applying the 12:38.640 --> 12:48.320 best practice to open source and that's also been marked with over 20,000 hours of independent 12:48.320 --> 12:56.880 expert security review and close to 200 projects directly improved and thankfully we have been 12:56.880 --> 13:06.800 scaling up so it is of course it's not something that can be kind of fully scaled like a lot of 13:07.760 --> 13:15.040 other technical solutions but there is the talent out there and the process out there to really ramp up 13:15.040 --> 13:26.320 this work and do instead of 200 total do close to 200 per year. So I'm feeling very optimistic about 13:26.320 --> 13:31.840 the future especially given sometimes a little bit of struggle or a little bit of, 13:32.560 --> 13:40.560 yeah I guess struggle is a good way to put it can really spur action and incentivize 13:41.600 --> 13:46.080 people to really think about the security of the open source projects that they depend on, 13:46.080 --> 13:54.640 that they use, that their organizations use and groups like us like many other organizations that 13:54.720 --> 14:05.680 are working towards solving this problem have the means to do so. So I am actually wrapping up with 14:05.680 --> 14:10.160 a material that I had I was not able to look at my notes while presenting which is why it was a bit 14:10.160 --> 14:19.040 sparse so we can turn the rest of the session into Q&A for anyone that has questions or things 14:19.120 --> 14:25.520 that I might not have been able to address during the talk. So yeah let's hear I see a lot of hands 14:25.520 --> 14:34.960 up so this is great and definitely open to doing some discussion so oh thank you very much. 14:39.440 --> 14:46.080 Thank you for a great presentation I really like the concept the motivation and it's very much 14:46.080 --> 14:56.800 needed my question to you is as you talked about scaling this is there a role of AI in doing 14:56.800 --> 15:04.240 security analysis and will that help being adding a middle layer on top of your layer you been on 15:04.240 --> 15:10.640 the top and some degree of AI in here for doing security auditing just to ease out the process 15:10.640 --> 15:18.400 make it more committed and making it more scalable. That's a very good question and I'm not 15:18.400 --> 15:27.520 surprised it was the first one. I think the answer is that it depends I mean AI is still very new 15:27.520 --> 15:36.400 in terms of its development and from what I have seen so far it looks like AI is currently doing 15:36.400 --> 15:42.960 more harm than good in that it's a great concept but at the end of the day it's in the implementation 15:42.960 --> 15:51.920 and execution so when the incentives are just to create AI that can just find bugs that kind of 15:51.920 --> 15:59.920 reward function you know there's a mismatch there and that's why we're seeing very public instances 16:00.000 --> 16:07.920 of projects just being absolutely overwhelmed with AI bug reports that from what I have seen 16:07.920 --> 16:16.080 are have a very low rate of being actual issues so I think there is potential and I think 16:16.800 --> 16:24.400 over time AI could be used to augment the audit process but the where AI falls short and where 16:25.360 --> 16:32.880 what makes this process so meaningful and powerful is the context so having auditors who really 16:32.880 --> 16:41.280 understand what they're reviewing because they'll be able to actually find issues find classes of bugs 16:41.280 --> 16:46.080 and I think over time that could be augmented but I don't think we're there yet. 16:46.560 --> 16:55.600 How does the prioritization work for you go about where's the funding for for selecting 16:55.600 --> 17:00.960 a project which project to audit could you say something about that? Absolutely it's a great 17:00.960 --> 17:06.400 question as well and it's a question that comes up a lot actually where sometimes we end up spending 17:06.400 --> 17:13.360 more time talking and more time and resources talking about which projects to audit when we could 17:13.360 --> 17:18.080 just be auditing you know we could agree at least on some projects and actually start doing the work 17:19.120 --> 17:28.160 generally in our experience it has been very demand driven so the funding the bodies that 17:28.160 --> 17:36.960 fund the work typically dictate which projects to review however as we've grown we've gotten a 17:37.040 --> 17:44.080 little bit more flexibility with project selection so we will do undergo our own analysis 17:44.880 --> 17:53.280 and come up with projects that are especially showing signs of benefiting from this work but 17:53.280 --> 17:59.040 from our experience I mean we've audited projects that are you know your stereotypical kind of 17:59.120 --> 18:08.000 single maintainer project you know we always think of the XKCD graphic you know the little block so 18:08.000 --> 18:14.400 we've audited those and we've audited projects as big as git and Kubernetes and others and 18:15.600 --> 18:20.960 really from our experience all of them have benefited from this work and I think it's just a 18:20.960 --> 18:30.960 testament to the under-resourced nature of open source and the fact that there isn't enough 18:31.680 --> 18:40.720 funding and enough diligence really going into open source projects I do hope that over time as we grow 18:41.360 --> 18:48.400 we can get to a place where we have essentially our own discretionary fund to help a lot of those 18:48.400 --> 18:55.040 edge case projects because so much focus is going on critical infrastructure and understandably so 18:55.840 --> 19:01.520 but there's a lot of edge cases and those projects that you don't necessarily seem like they would 19:01.520 --> 19:09.760 be important like no one I like who knew what XZ was before 2023 okay more than I thought but 19:11.120 --> 19:16.720 but I think that's a find example where there's a lot of edge cases there's a lot of small projects 19:16.800 --> 19:23.360 that just simply don't get enough attention and we have a way to improve those projects and find 19:23.360 --> 19:28.960 those vulnerabilities that sometimes make the news so we should be going out and fixing them 19:30.800 --> 19:37.040 next question hi you mentioned the importance of heavy and champion and can you be more specific for 19:37.600 --> 19:45.280 who selects or volunteers to be the champion yeah how does it work absolutely yes I'll go back 19:45.280 --> 19:57.120 to the slide so what I mean by championing is let's say I'm a maintainer of a very popular open source 19:57.120 --> 20:04.960 project and I am under a foundation and I want to get a security audit done or get some security 20:04.960 --> 20:14.720 help just the process of finding professionals or finding audit teams let alone reaching out to 20:14.720 --> 20:24.560 them let alone working with them on getting proposals or getting calls for proposals and analyzing 20:24.560 --> 20:33.520 all of those proposals vetting different vetting the vendors and let alone all of the contracting 20:33.520 --> 20:39.600 and administration that goes into the process it's it's staggering it's it's very high and it's 20:39.600 --> 20:46.320 why this process is not more widespread because it's it's hard essentially it's a lot of what we 20:46.320 --> 20:53.840 call cat herding you know even with projects where we have the funding in place we we have the 20:53.840 --> 21:00.480 maintainers on board the amount of administration and project management to make this this work 21:00.480 --> 21:09.360 so valuable is a very high and typically without having that dedicated resource the process just 21:09.360 --> 21:17.600 does not go as well as intended so there are some organizations that have the capacity they have the 21:17.600 --> 21:25.280 people who can undergo these processes reach out to vendors build relationships with them have a 21:25.280 --> 21:31.920 contracting process have a invoicing and accounting process and they have those in place and 21:31.920 --> 21:38.800 they have been able to procure security audits successfully but those examples are very rare compared 21:38.800 --> 21:44.480 to the norm which is we don't even know where to start we don't know who to talk to let alone 21:45.520 --> 21:50.000 something we've heard a lot today a lot of open source projects are just individuals they don't 21:50.080 --> 21:56.640 have a bank account or a company associated with them or really any way to procure this kind of work 21:56.640 --> 22:03.840 and that's exactly why we created the the the nonprofit that we did because it does all of that 22:03.840 --> 22:12.880 essentially primarily on behalf of the maintainers with the mission of improving the software 22:12.880 --> 22:19.600 and doing it in a transparent way so again it's it's extremely important to have somebody 22:19.600 --> 22:26.000 dedicated to this because when it's somebody's fourth fifth job to to go and do this it's just 22:26.000 --> 22:33.120 not going to happen effectively so I think we have time for one one or two more I see a hand up oh sorry 22:33.120 --> 22:39.680 yes so you mentioned hiring experts to do the review for the security audits but in fact they're 22:39.680 --> 22:45.760 also using tools that you can see tools fuzzing etc so in your experience with all these projects 22:45.920 --> 22:50.160 are the tools that's you would recommend that's a broad category of tools that's 22:50.160 --> 22:56.000 I've been profitable in these hundreds for that yeah that's an excellent question 22:57.280 --> 23:03.600 I don't necessarily have opinions on which tools are better than others I can say that I know 23:03.600 --> 23:10.880 a lot of our audits have involved a significant amount of static analysis tooling I know some 23:10.880 --> 23:17.680 graph has been used in a number of our audits by the audit teams for for fuzzing as well that has 23:17.680 --> 23:25.680 grown a lot in in impact over the last few years and so we work with teams that are very well 23:25.680 --> 23:34.000 versed in OSS fuz and and our when applicable can build fuzzing sweets and actually build the tooling 23:34.000 --> 23:40.800 for the projects to help them not only be more secure in that moment but from every moment on 23:40.800 --> 23:47.200 words after the audit so tooling is a huge part of our audit process so I'm glad you brought that up 23:48.560 --> 23:54.400 and I know for sure we've worked with a lot of different ones but we don't we're not really in the 23:54.400 --> 24:00.320 business of proprietary tooling so anything we would use or recommend would be open source 24:00.400 --> 24:12.400 so yeah so I'm with Dr for a little bit quite interested in this on this type of auditing 24:12.800 --> 24:20.640 we have code we have things that look more like configuration we have processes around the code 24:20.640 --> 24:26.320 and then we've got infrastructure so just wonder sort of what is the scope of what you can address 24:26.320 --> 24:31.200 with these or this is it just the code or is it that kind of extraneous stuff around as well 24:32.240 --> 24:37.600 that's the next one question as well I think in a perfect world there would be enough funding to 24:37.600 --> 24:42.720 really dig deep into essentially all aspects of the project even down to things like 24:44.320 --> 24:52.480 mitigating social engineering risks or any kind of indicators of maintainer burnout or 24:53.200 --> 25:02.000 really any risks to the project overall in reality funding is very sparse so typically audits have to 25:02.000 --> 25:12.080 be scopes down to the most high-risk areas the most critical functionality and be audited to that 25:12.080 --> 25:20.640 but again in a perfect world and as we gain more influence on scoping and and funding these audits 25:20.720 --> 25:27.120 then we can dig much deeper but for now it's it's more focused on again most high-risk 25:27.120 --> 25:33.520 areas areas and is entirely risk based so I see my times up so thank you all again and have a great 25:33.520 --> 25:36.160 pause