WEBVTT 00:00.000 --> 00:11.000 Okay, hello, we are on time. Please vote for Mayba. 00:11.000 --> 00:16.000 Fire, Ron. 00:16.000 --> 00:21.500 So, I'm Eva's Hurricane New. I'm excited to be here, and for the next 20 minutes, we will 00:21.500 --> 00:27.200 talk about open-source dependencies and how to monitor vulnerabilities. Before we start 00:27.200 --> 00:31.720 up, she thinks about myself. I leave the product security to matter most, matter most 00:31.720 --> 00:36.720 in case of some of you don't know, it's a collaboration platform for mission critical work. 00:36.720 --> 00:40.840 I was previously adult zero, acquired by all the principal product security engineer, 00:40.840 --> 00:46.320 they're living a few different places I'm currently in Athens, Greece. And with that, this 00:46.320 --> 00:52.600 is our agenda. So, we'll start by explaining why third-party dependencies and their vulnerability 00:52.600 --> 00:57.760 is matter. We'll evaluate different software. Well, we will explain a process to evaluate 00:57.760 --> 01:02.120 different software composition analysis tools. We will see how to roll out such a tool 01:02.120 --> 01:08.200 and finally, how to create the findings. The reason we have this agenda, it's because it's mapped 01:08.200 --> 01:13.520 to what I call a dependency management journey or a secure, a secure dependency management 01:13.520 --> 01:18.840 journey. Before we start talking about any SCH solution, we first need to understand 01:18.840 --> 01:25.880 why we need to do that and we need to understand the risk that vulnerabilities in third-party 01:25.880 --> 01:30.600 code carries. Once we can do that, we can start saying, okay, this is a problem we need 01:30.600 --> 01:36.080 to solve and we can start gathering our requirements and based on those, evaluate some tools. 01:36.080 --> 01:41.320 Once this is done, then we can start evaluating a tool, pick one, roll it out and then 01:41.320 --> 01:48.040 do something with the findings. And we will cover this journey today. So, why do we have 01:48.040 --> 01:53.160 this talk today? Why do we talk about third-party dependencies and their vulnerabilities? 01:53.160 --> 01:59.160 I'll start by saying that it's important to keep in mind how a software is comprised. I would 01:59.160 --> 02:03.720 say that in big projects, it could be half-and-half in smaller ones, the third-party code can 02:03.720 --> 02:10.040 even be higher than the in-house code. What does this mean? It means increase the tax surface. 02:11.080 --> 02:15.560 Now, there is a valid point to be made that any time you add new code, you add new 02:15.560 --> 02:21.720 tax surface, either that's in-house code or third-party code. There is however one significant 02:21.720 --> 02:28.600 difference between those two, which is control. And in case of third-party code, lack of control. 02:28.600 --> 02:35.000 So, for your in-house code, you can control your security processes, a security review. You can have 02:35.000 --> 02:40.840 static analysis tools scanning your PRs. So, you can be proactive around the code that you 02:40.840 --> 02:46.840 ended up having in your software. You don't have the option in third-party code, 02:46.840 --> 02:51.960 the other use it or you don't use it. And this is where the this increase that tax surface 02:51.960 --> 02:59.560 comes into play. Now, it's also important to understand that managing vulnerabilities and third-party 02:59.560 --> 03:05.560 dependencies is not just a security best practice, they carry real risk. And the best way to 03:05.560 --> 03:13.080 remember is a log-for-J issue back in 2021, where I create a vulnerability in a library 03:13.080 --> 03:22.200 actually called severe issues in software using this library. So, usually we spend more time 03:22.200 --> 03:28.920 into our in-house code and the vulnerabilities what we call in our software. And we often like 03:28.920 --> 03:33.960 make a second-class citizen, the dependencies, and their vulnerabilities. And examples like 03:33.960 --> 03:40.040 LLK4J remind us how important it is to also keep track of our third-party dependencies and 03:40.040 --> 03:47.480 their vulnerabilities. Now, without setting stone, before we go into evaluating different 03:47.480 --> 03:53.880 solutions, we need to define what are our goals. And I split them into two categories, 03:54.600 --> 03:59.800 proctivate and reactivity. And for me, these are the two pillars of a security programming 03:59.880 --> 04:05.720 general. Again, if you think about your in-house code, you usually do some proctivate, 04:05.720 --> 04:11.560 as you have a proctivate approach, security reviews, some security testing maybe. And because 04:11.560 --> 04:15.800 you are also released, they can you know that things will get missed. You also have a reactive 04:15.800 --> 04:22.600 approach, we usually call this vulnerability management. It's a similar process here. So, 04:22.600 --> 04:28.680 for reactivity, what we're looking is it's pretty obvious. Address dependencies that the vulnerabilities 04:28.760 --> 04:34.280 that we have from our third-party dependencies. And for the practice part, we want to make sure 04:34.280 --> 04:41.160 that once we start working on this like building a process and a program around this, that we don't 04:41.160 --> 04:46.360 keep adding vulnerable dependencies. We will come back to this slide a bit altered to explain 04:46.360 --> 04:52.280 more in detail how we can do that. But this is something to keep in mind going forward. 04:53.000 --> 04:58.600 Without said, let's talk about, well, I forgot to mention, SEA is software composition analysis, 04:58.600 --> 05:04.680 I'll just be using SEA going forward. Why to evaluate such a tools for this solution? Well, 05:05.320 --> 05:09.560 imagine like going over all your dependencies, all their transitive dependencies, and then 05:09.560 --> 05:13.880 finding out if they have vulnerabilities. There are tools out there that can help us with that. 05:14.440 --> 05:20.760 So, let's figure out what's best for us. Now, one thing that often, I think there is 05:21.480 --> 05:27.880 we often do something which I don't agree with with different tooling. As we have a problem, 05:27.880 --> 05:32.120 we go out, we find we're the different popular tools, open source commercial, I'm not going to 05:32.120 --> 05:38.520 suggest any. And then we try to understand which fits better for us. I'm actually suggesting the exact 05:38.520 --> 05:44.520 opposite so that we remove these initial bias of looking online, which is, understand your problem, 05:44.520 --> 05:50.200 define and document your requirements and be clear about them. And then in the next two slides, 05:50.280 --> 05:56.040 I'll actually show what kind of requirements these could be. Now, it also helps your evaluation. 05:56.040 --> 06:01.400 If you assign them a label, this will help when you actually evaluate tools and you know, 06:01.400 --> 06:07.160 well, this is a must-love. That's pretty nice. I could leave without it. And once you've done 06:07.160 --> 06:11.400 what I call your homework and you're clear about what it is that you really need, 06:11.400 --> 06:18.680 then you can start looking for different tools to evaluate. Usually, the requirements you've created, 06:18.680 --> 06:25.640 you can answer them from the docs for the biggest part. So, you can do that and then I say pick three. 06:25.640 --> 06:32.600 The reason I do, I say this is time and effort like restrictions for further evaluation. 06:33.560 --> 06:40.120 Now, an important aspect, what we mean about evaluation. Another common mistake we're doing is 06:40.120 --> 06:49.000 we evaluate the details, but not the core. What I mean by that is SEO solutions give us a 06:49.000 --> 06:55.560 very hard promise that they can give us complete an accurate results about all the vulnerabilities 06:55.560 --> 07:01.560 of all the transitive dependencies. This is a hard promise. So, when you evaluate, I would say 07:01.560 --> 07:07.320 get your source code, include dependencies, ideally dependencies that have a transitive dependency 07:07.320 --> 07:13.000 with availability and test if the tool will find it. Like, really test the SEO engine, 07:13.000 --> 07:16.680 you can also test all the other requirements, but pay attention to that part. 07:20.120 --> 07:25.960 These are some examples about requirements. I'm not going to go over them, but you can get an idea 07:25.960 --> 07:31.960 the core goes to the spreadsheet with the requirements. Just a few notes, usually languages and 07:31.960 --> 07:37.960 package managers are your must have requirements. Your organization uses some like package 07:37.960 --> 07:43.720 management, there are some languages. If a tool doesn't cover them, it doesn't really cover your needs. 07:45.480 --> 07:49.720 And what I have in the third one, it's zero to minimal findings in consistency. This goes 07:49.720 --> 07:56.840 back to the robustness of the engine that actually gives you the results. You can have some 07:56.840 --> 08:04.040 like more details depending on your requirements. And again, one common for the reachability. 08:04.040 --> 08:10.040 It's very difficult to get it right. If you think that you need reachability, which means making sure 08:10.040 --> 08:16.440 that your code reaches a vulnerable function or the vulnerable dependency, test it out. 08:19.160 --> 08:26.360 So, we're always so far. We discuss what is the risk. We discussed what are our requirements 08:26.360 --> 08:32.680 and what is a process to evaluate tools out there. Now, what are we going to do about rolling, 08:32.680 --> 08:38.920 we selected a tool and now we're ready to roll this out. How can we do that? I'm listening to 08:38.920 --> 08:45.160 different options here. The first is roll everything at once. What I mean by that is you can 08:45.160 --> 08:51.320 integrate your tool with all the repositories and then start re-ausing. There is one caveat 08:51.800 --> 08:56.360 if you have many repositories and they have many dependencies and many vulnerabilities, 08:56.920 --> 09:04.840 you'll get a huge number. It feels chaotic. I've been there. You see where I'm starting from. 09:06.600 --> 09:11.880 Another option is what I call a phase approach. I'm going to give an example here and then I'll 09:11.880 --> 09:16.680 actually tell you what we did at a mile or most. A phase approach could look like the following. 09:17.320 --> 09:22.920 You make a list of your more important repositories. Max5 is just a random number. You need to 09:24.200 --> 09:28.360 again, like time box is set for it and then you can have maybe a second phase. 09:30.200 --> 09:36.040 When you start the roll out, you integrate your first repository. You wait X amount of days, 09:36.040 --> 09:43.640 X depends on how many efforts you can put, how many vulnerabilities it ends up having. 09:44.520 --> 09:49.640 After that time, you can integrate your second repository and by that time, you make sure that 09:49.640 --> 09:58.040 your critical and high findings are mitigated. You go on. Some factors that you can 09:58.600 --> 10:04.680 consider to when designing those two options are different teams responsible for different 10:04.680 --> 10:11.960 repositories. If so, you can do them in parallel. Yes, as a security team, the security team will 10:12.040 --> 10:17.800 have some a bit more work to help the engineering teams, but engineering teams working on mitigating 10:17.800 --> 10:23.160 the finding if they're different, we can parallelize that. How many vulnerable dependencies is 10:23.160 --> 10:30.360 repository has? If there is a culture of regular maintenance updates and you have very few findings, 10:31.480 --> 10:35.880 again, you can either group them in the phase roll out or you can just do it all at once. 10:36.760 --> 10:41.400 All at once also means how many repositories do you plan to integrate? Are there a few? 10:41.480 --> 10:45.640 Maybe it's okay if you do it like the all at once option. You have too many. 10:47.160 --> 10:53.480 Somehow group them and then phase them, like do them in phases. And again, to be realistic, 10:53.480 --> 10:59.960 was a team like capacity in workload both the security team and the engineering teams. So, 10:59.960 --> 11:08.360 depending on those factors, you may go for all at once, a phase. We went for a phase, but not like 11:08.440 --> 11:13.000 the one by one. We actually, like, stank ranked our repositories in terms of 11:13.960 --> 11:19.640 priority and the most important ones. Then we said, okay, which team owns which repository we found 11:19.640 --> 11:24.520 out that we can do some parallel work and we actually ended up having three small phases, 11:25.080 --> 11:29.240 but if you think of phases is, well, not phase. It's actually one. It's a phase approach where 11:29.880 --> 11:35.880 we did three buckets of repositories. Three works for us because of how our teams are structured 11:35.960 --> 11:41.240 and how the repositories own to those teams of structure, but that's an example. 11:43.240 --> 11:47.400 I'll go back to this slide, the reactive approach and the proactive approach. 11:48.440 --> 11:52.280 Because this is when we roll out the tool and we need to think about those two goals we 11:52.280 --> 11:58.680 discussed in the beginning. In the proactive approach, once you integrate, make sure that 11:58.680 --> 12:04.760 use can the PR level. So, one thing is to scan the your main bronze and see, okay, I have this 12:04.840 --> 12:08.440 dependencies, I need to fix them, which is actually the reactive approach I just said. 12:09.160 --> 12:14.520 But for the proactive one, you need to make sure that you start working on what you have as 12:14.520 --> 12:20.200 vulnerabilities, but you don't add more. And to do that, you can have a check at your PR. 12:20.920 --> 12:26.760 It's easy if your manifest file has a change inversion. If it has, it calls your SEA tool. 12:27.320 --> 12:32.840 If there is a vulnerable dependency, I would say fail to check and discuss, like, the security 12:32.840 --> 12:37.400 and with engineering team and find this solution. Otherwise, you would end up 12:38.040 --> 12:43.320 keeping reactive as more vulnerabilities get added. Now, for the reactive approach, 12:43.320 --> 12:49.080 if you solve a phased approach, I said, I showed before, I said highs and criticals. 12:50.120 --> 12:55.080 This is just for our risk-based approach. Like adding a few repositories and going through all the 12:55.080 --> 13:00.120 findings, depending on the regardless of severity, means that you are leaving other repositories 13:00.200 --> 13:06.040 with highs and criticals to wait. This is why here, it mentions a risk-based approach. 13:07.720 --> 13:13.080 Now, for let's go to where are the phase of preusing the findings? 13:14.120 --> 13:20.840 How to do it? I'm usually suggesting a time box investigation that aims to answer two different questions. 13:20.840 --> 13:25.560 The first one is very simple. Is this a test dependency? If yes, we can skip it. 13:25.960 --> 13:33.640 And the second one, which is difficult to answer and goes back to the retability discussion we 13:33.640 --> 13:42.680 had earlier, is it exploitable in our systems? Is it exploitable dependency or even better the 13:42.680 --> 13:49.560 vulnerable function of the dependency reachable from our code? Now, the second question can be 13:49.560 --> 13:55.960 part answer, especially if you think of transitive dependencies of a N-depth, and that's why I'm 13:55.960 --> 14:01.160 saying time boxed. And also, this is why I suggest that you balance the time between doing the 14:01.160 --> 14:07.320 investigation and actually going and updating the dependency. Like, it may not make sense to even 14:07.320 --> 14:10.840 do this time box if it's, I don't know, a minor version that you're going to update it. 14:12.680 --> 14:17.880 Another thing to keep in mind is that updating is not always an option. Well, there are many 14:17.880 --> 14:26.520 times where there is no fix available. What can you do there? Notify the maintainer. We are in an 14:26.520 --> 14:33.960 open source conference, let's work with the maintainer, notify them and see what's their appetite. 14:33.960 --> 14:39.960 Maybe they would like to work on, they don't have time contribute. Like, you can work on addressing 14:39.960 --> 14:45.880 the finding, contribute, and then both your project and all the other projects will benefit from that. 14:47.960 --> 14:53.160 Sometimes, maintainers are not responsive. Maybe there is an unmetane library. It's possible that 14:53.160 --> 14:58.120 you may need to explore alternative dependencies, offering the same functionality. But these are 14:58.120 --> 15:05.800 things that we can keep in mind. Now, slowly closing, how to do this dependency management 15:05.800 --> 15:14.280 effort, like a sustained thing. Track the findings and track their resolution for two reasons, 15:14.280 --> 15:20.440 first to make sure that you work on them. Secondly, because you give a signal to your organization 15:20.440 --> 15:25.240 that those findings are equally important to your in-product vulnerabilities. Like, make them a 15:25.240 --> 15:30.840 first class citizen. Whatever process you have for your in-house vulnerabilities, use a very similar 15:30.840 --> 15:39.640 one for your vulnerabilities and dependencies. The second one is that once you do a live initial 15:39.720 --> 15:45.960 rollout, you may think it's your done. Because if you think about it, I mitigate all the existing 15:45.960 --> 15:52.120 vulnerabilities. I added a PR check, so no new ones, but wait, the existing ones, we may find 15:52.120 --> 15:57.960 new vulnerabilities. So this reactive approach never really ends. The initial rollout is usually 15:57.960 --> 16:04.600 the heavily-fting, but you will still get new vulnerabilities that you would need to mitigate. 16:05.480 --> 16:11.800 And that's actually why it's good to have a regular maintenance schedule. This will also help 16:11.800 --> 16:17.480 you a lot with those lows that is very difficult to get any buying to fake, especially their 16:18.200 --> 16:25.080 dependencies. So having a regular maintenance schedule of updating dependencies in general across 16:25.080 --> 16:33.960 your repositories will help a lot. So this is all the journey we covered. As you see, you can 16:33.960 --> 16:40.280 do the first three phases once. The third one is a repeatable process. I have a lifting in the 16:40.280 --> 16:47.720 pro in the beginning, but a repeatable process. And with that, some key takeaways, choose the tool 16:47.720 --> 16:53.480 wisely. If you need to do some initial integration between your tool and your systems to it, 16:53.480 --> 17:01.480 like in the initial rollout, the more integrated your tools are, the more easy it is for the 17:01.480 --> 17:11.320 engineering teams to feel like to be better, like to see the tool better, like to feel it to 17:11.320 --> 17:17.960 feel it better. The other thing is, what we discussed before, when you roll this out, have a clear 17:17.960 --> 17:25.320 strategy for addressing the findings and for having this proactive approach. But our lymphocytes 17:25.400 --> 17:33.240 before you do anything like have a plan. It can become stressful when you get all the findings. 17:35.000 --> 17:40.520 And as you probably saw in this presentation, this is an org wide effort. It's not a security 17:40.520 --> 17:46.680 team only. It's not engineering team only. It's an org wide effort. It needs time. It needs investment, 17:46.680 --> 17:52.680 especially in the beginning. Get agreement, get buying that for the next few months, we will need 17:52.680 --> 17:57.640 to allocate some time and effort in this initiative. That's all I had. 18:06.040 --> 18:14.200 Seven months for questions. Regarding reporting to a tool like zero, I have made some 18:14.200 --> 18:21.720 observations first, the nature of the findings. I mean, to yourself see me numbers. 18:22.920 --> 18:30.120 And really being slow, et cetera, the very often replaced by a OSB or a DSB first. 18:30.920 --> 18:37.720 But so there was a dynamic nature of those findings. And then if you report to zero in such tools 18:37.720 --> 18:44.280 like a fire and progressive approach, it might happen in the complex environment. It might happen 18:44.280 --> 18:53.240 that people talk about various identities. If in case you don't keep those tickets dynamic 18:55.080 --> 19:01.880 and the second thing is if you have a real complex application, it pays off to have a look at. 19:01.880 --> 19:11.400 So vulnerability tickets, which have a supervisory approach. Both management and developers, 19:11.400 --> 19:18.680 because they can see how far some teams are already fixing things. In case it's not trivial, 19:18.680 --> 19:24.760 both from management perspective and let's look which colleagues we can ask for help. 19:25.720 --> 19:28.440 Yeah, that's true. 19:31.720 --> 19:34.280 And I think one thing that this reminded me is, 19:35.160 --> 19:41.080 Girish tickets should not be a fire for a get. They should come with some sort of SLAs. 19:42.680 --> 19:47.640 If you have some SLAs for your in-house product vulnerabilities, maybe you can relax them a bit 19:48.360 --> 19:52.360 again because you need a cream in from engineering, but don't forget them. 19:55.080 --> 19:57.080 Yep. 20:01.080 --> 20:06.360 Two small questions is, what is your opinion on the state of the furnace available, 20:06.360 --> 20:11.080 but not the needs to be part of the digital state of the furnace on relatively diaspora 20:11.080 --> 20:17.000 to the degree of the automobile industry and dependencies. Also, have you considered, 20:17.000 --> 20:22.600 you can see open it is a scorecard based use of dependencies in the strategy. 20:23.560 --> 20:29.800 With, I used to, like a year ago, I was in the NSSF one working group. 20:29.800 --> 20:35.560 With the evaluated scorecard, we haven't had the time to roll this out. 20:36.680 --> 20:43.240 For tools and retability, I'm very, very skeptical. I'm yet to find a tool that does what it promises. 20:46.200 --> 20:51.080 Yeah, it's a very difficult problem with retability. I can understand direct dependencies 20:51.160 --> 20:55.960 and retability. I'm yet to become convinced about the transitive ones. 21:08.680 --> 21:13.400 That's a good question. They often don't understand the risk. 21:14.440 --> 21:17.320 So they don't understand why they need to spend time on this. 21:18.120 --> 21:21.880 When you have an in-house product vulnerability, you have steps to reproduce, 21:21.880 --> 21:27.480 you can, like, they can actually see things happening. When there is a transitive dependency, 21:27.480 --> 21:33.880 especially the transitive, we're proving their risk is very time consuming for a security team. 21:35.320 --> 21:42.920 Making, like, persuading and engineering teams to work on that, it creates friction. 21:43.160 --> 21:51.000 And the other one is, if there are no, like, major upgrades is a friction. 21:51.000 --> 21:56.200 Like, if it's a minor or a patch, usually people are fine, when you tell them to upgrade, 21:56.200 --> 22:00.360 like, to two major versions happen, they need to do loads of code refackering, they don't like it. 22:03.080 --> 22:08.520 Or, if there is no available fix, so they need to go and see alternative dependencies, 22:08.600 --> 22:13.560 and then maintain it as we said, is not responsive. So they need to go final alternative dependencies. 22:13.560 --> 22:18.600 This means that they also need to do some code changes. In any, in any, like, the more effort, 22:18.600 --> 22:25.720 the more friction. Because the risk involved is not that clear as with a in-house vulnerabilities. 22:38.520 --> 22:45.240 I mean, at the source, like, say, like, hey, you know, get talk and go talk to them, 22:45.240 --> 22:51.320 like, maybe go to use so much for the sort of, you know, a fair part, you look for a little east version of that. 22:53.080 --> 22:55.960 Is that even a gadget? Can you repeat the question? 22:55.960 --> 23:01.320 Well, I'm so sorry, I'm so sorry. As I get my gadget, I've had a chance to, maybe, 23:01.640 --> 23:07.560 um, thank you for what fancy to use? Yes, this is a very good one. That's a very good point. 23:09.160 --> 23:16.120 The question is that addressing vulnerabilities and dependencies in what is one part, 23:16.120 --> 23:24.120 but shouldn't we really educate engineers or on what dependencies to use and what not to use? 23:24.120 --> 23:28.520 And this, actually, I think goes back to the scorecard where, like, the open is a set, 23:28.600 --> 23:35.560 the OSSF1 where vulnerabilities, like, dependencies, you get a scorecard depending on your, on different 23:35.560 --> 23:42.600 metrics. They're higher, the, like, it can be maintained, um, it can have security advisors or 23:42.600 --> 23:49.240 many different factors in there. So yes, this is a very valid point. This, like, dependency management, 23:49.240 --> 23:55.080 security dependency management is one part, uh, but educating what kind of dependencies you introduce 23:55.160 --> 23:59.160 your code is, it's, it's, it's equally important. Yeah, that, thanks for the point. 24:20.440 --> 24:23.800 Yes, uh, so the, if understood correctly, the question is, 24:23.800 --> 24:30.040 at matter most, um, how many people need to be involved in this process that I described. 24:31.400 --> 24:34.520 It's definitely our team, which is the product security team. We do the trius, 24:35.320 --> 24:40.760 we at some times do also some heavy lifting, like, if there's no fix available, we may do some 24:40.760 --> 24:46.920 research on this and the, again, the maintainer is not responsive. What else can we use? We, 24:46.920 --> 24:53.560 some, and because we're, like, we started the, we did a more systematic approach to this a 24:53.560 --> 25:01.960 year ago, um, we do some initial work. So it's the product security team. And then we do regular 25:01.960 --> 25:10.120 maintenance of dates. This helps a lot. Um, and the other thing is the way we, uh, our engineering 25:10.120 --> 25:15.400 team operates, usually there are specific teams for specific, specific repos. And within those teams, 25:15.480 --> 25:20.200 there is usually a person who's has taken, like, the, the ownership of, of taking the 25:20.200 --> 25:29.320 dependant. The good that, um, there is less context switching for the whole team if there's 25:29.320 --> 25:35.400 one person. So the most fun thing to do, uh, let's, that's how we do it. Okay, let's thank 25:35.400 --> 25:41.000 you.