WEBVTT 00:00.000 --> 00:13.840 Now, we have gates to it, the vice president of the Penable Embedded Systems at the Linux Foundation. 00:13.840 --> 00:18.720 She has several years of experience in the software industry and has been instrumental in 00:18.720 --> 00:25.120 the success and launch of several projects that you may have heard of like Zephyr, SPDX, 00:25.680 --> 00:33.440 Kales and more. Today, she will be talking to us about how Zephyr applied best practices 00:33.440 --> 00:40.960 from the Linux kernel community to achieve more than 100,000 comments with focus on safety and 00:40.960 --> 00:43.920 security certifications. Peace, welcome, Kate. 00:44.800 --> 00:54.640 Thank you. Thank you very much. Just for, let me meet the room since we're in person, 00:54.640 --> 00:58.800 which I adore. How many people have heard of Zephyr before this talk? 01:00.880 --> 01:04.800 Okay, so we've got some new people who haven't heard of Zephyr, so I've got a little bit of background, 01:04.800 --> 01:09.440 but I won't spend a lot of time on it and you're more than welcome to come and ask me more questions 01:09.520 --> 01:16.640 afterwards. Who am I? For those who don't know me, my focus of my career has pretty much been embedded 01:16.640 --> 01:24.080 open source and the current challenges I'm really focusing on is how can we make sure that open source 01:24.080 --> 01:30.320 when it's being used and safety critical applications can be analyzed and made sure that we can 01:30.320 --> 01:34.960 get a safe system working at the end of the day. So you'll see me in Zephyr and you'll see me 01:35.120 --> 01:41.920 in the Elisa project and we're starting working on a space-grade Linux project there and 01:41.920 --> 01:49.280 remontere to with me and so he's a one wearing a penguin with a spaceship top. So we've got 01:50.000 --> 01:56.400 a fair amount of things that we're sort of caring about in this space because open source is very high quality. 01:57.440 --> 02:01.840 So amount of review and everything else is there, but it's not done in a way that the 02:01.840 --> 02:07.520 traditional safety analysis process is going to deal with it. So part of the challenge is going to be 02:07.520 --> 02:13.200 how do we actually make this stuff accessible and how do we actually work it from this perspective? 02:14.400 --> 02:19.520 So these are sort of the projects you'll probably see my name associated with and I've been volunteering 02:19.520 --> 02:29.040 on the SPDX front since it started in 2009. We've been working on S-bombs and you will see evidence of S-bombs 02:29.040 --> 02:35.200 in CS-bombs for those who are in the last talk. We've been generating seed-based S-bombs in 02:35.200 --> 02:40.880 Zephyr for the last four years as we've been trying to do best practices in the industry all the way along. 02:41.920 --> 02:49.920 And in terms of hobbies for me, I like photography, I like traveling and I like taking 02:49.920 --> 02:54.960 separate types of interesting places now. Unfortunately last March they would not let me fly that 02:54.960 --> 02:59.680 kite down there because the bird flu had started and the first signs of the bird flu had made 02:59.680 --> 03:04.000 it down to Antarctica. So I could carry it on and hold it up, but I couldn't fly it. I was 03:04.000 --> 03:10.080 dreaming of being able to fly the kite down in Antarctica. However, word of mouth, so it's not confirmed. 03:10.080 --> 03:18.480 I don't have details. They are using Gen2 penguins with separate-based trackers for monitoring 03:18.480 --> 03:23.760 global warming and seeing the changing patterns of, you know more about that detail? No, okay, 03:24.560 --> 03:28.960 because I've gotten a little bit of hints of this from a couple people, but I don't know the details 03:28.960 --> 03:36.800 and I'd love to tell that story. Anyhow, contact me anytime. There's my, you know, there's my linking, 03:37.680 --> 03:44.080 but for those who don't know Zephyr, it's a real time operating system. And we started the 03:44.080 --> 03:50.000 project off of safety and security in mind. The projects launched formally in 2016 about this 03:50.080 --> 03:57.040 time in 2016, so we're about nine years old now. And we've asked a bit by bit, we've grown the 03:57.040 --> 04:03.680 community and when we looked at it and looked at security at the start, we needed to be different 04:03.680 --> 04:09.760 than what was out there already. Back in 2015, for those who might remember that period, the joke was 04:11.680 --> 04:18.240 security as the SNIOT, applying it wasn't there. And so we really wanted to make sure we could 04:18.320 --> 04:23.120 do security best practices with this project, pretty much all the way through. And so I'll go through 04:23.120 --> 04:28.320 some of that, but it's very modular. Anyone who's familiar with embedded Linux will find it 04:28.320 --> 04:35.280 pretty familiar. It's not Linux code base, but it's structured like Linux. So it's got K config, 04:35.280 --> 04:40.960 it's got device tree, and this lets you be very flexible about just putting in what you actually need. 04:41.840 --> 04:46.240 So the lose kernel doesn't get smaller than about three meg these days, when you really 04:46.400 --> 04:51.520 put it down. The first starts at 10k and goes up, depending on what you want to do with it. 04:53.280 --> 04:57.520 So it was structured really differently for the embedded space, and we have people who have a 04:57.520 --> 05:02.000 long amount of embedded experience who are very focused on keeping it small and not letting things 05:02.000 --> 05:08.640 creep into it. We've also learned a lot from Linux, and I'm going to some more of these lessons, 05:08.640 --> 05:13.760 but things like having an LTS, so that people who are making products don't have to keep up with 05:14.400 --> 05:19.920 if they do not need to. Or they may keep up with tip for a while, and then things aren't changing, 05:19.920 --> 05:25.840 so they just worry about security fixes. The LTS is good for that. And we are actually going after 05:25.840 --> 05:31.760 a formal 61, 508 certification, this effort, for a subset of the kernel, and I'll talk a little bit 05:31.760 --> 05:40.960 about that. And as I'm talking, if, like I say, if something's really burning you and you want to 05:41.840 --> 05:48.240 ask a question, go for it, and those who ask a question get a kite. Okay, and otherwise we'll 05:48.240 --> 05:53.200 just have to put you in the end, whatever's the easiest for people. So where is effort today? What's 05:53.200 --> 06:00.880 going on? Well, we just finished doing some analysis, and thanks to Benjamin and Susan and me, 06:02.480 --> 06:09.280 we've actually had 1100 contributors last year into this project. 50% were first-time contributors, 06:09.440 --> 06:16.320 so the community is growing fairly strongly. And there's a blog post out on our site that 06:16.320 --> 06:24.080 goes into more details. The 150 boards were added to this repo. There's over depending on how you 06:24.080 --> 06:31.280 count it, and where you're counting it. There's probably around 700 and more boards sitting there. 06:31.280 --> 06:35.040 So if you've got a board sitting at home, there's a pretty good chance as a zephyr port available 06:35.040 --> 06:39.600 for it. So you can go and play with and just serve experiment. And we have a lot of samples, 06:39.600 --> 06:46.080 and I can show you some examples of that later. And just last year, Susan, who's in the audience, 06:46.080 --> 06:50.400 started a meet-up program for us, so that developers and regions can get together. 06:51.040 --> 06:56.320 Like, fossil-happens once a year embedded in it, world happens in November, you know, once a year, 06:57.520 --> 07:02.160 and then there's effort to follow for some it. This wasn't all we needed, and people getting 07:02.160 --> 07:06.800 together and talking about things and sharing ideas and having a community is what helps a community, 07:06.800 --> 07:11.680 that helps a community grow to be able to have the interaction in person. And the meet-ups have 07:11.680 --> 07:19.200 definitely been part of helping to accelerate zephyr in the last year, my mind. So to put things 07:19.200 --> 07:25.520 in perspective, where zephyr is these days, the lens foundation has hundreds of projects. 07:26.480 --> 07:33.520 And the CNCF folk, because they're, um, see, because they're going after Linux, 07:33.520 --> 07:39.440 so these are two logarithmic scales, okay? Just to put in the context, this is logarithmic scales, 07:39.440 --> 07:47.200 it's commits on the bottom, and then it's interactions on the side, okay? And then the circles 07:47.200 --> 07:55.680 of the contributor base. Um, Linux is by far still the most active in the successful project. 07:57.040 --> 08:01.120 However, Kubernetes has been coming up with a closer, like, inching up closer. 08:01.120 --> 08:06.000 This is why that community looks at this type of metric. And as a sign effect, 08:08.240 --> 08:12.560 zephyr is now showed up, zephyr starts showing up, lower and lower, but now we're 08:12.560 --> 08:20.160 at the fifth most active project of the Linux foundation, okay? So we're seeing, like, there's 08:20.160 --> 08:27.760 that 1100 developers serve showing up. And so the project didn't measure this another project 08:27.760 --> 08:32.400 measured it. So, lies down lies in benchmarks, but at least it was a neutral source. 08:34.240 --> 08:39.440 And then something that's maybe really excited this last year is, okay, it's one thing to be 08:39.440 --> 08:45.680 a parvalist foundation, but there's a much wider open source ecosystem out there. And so they 08:45.680 --> 08:50.240 also look and see where Kubernetes is in that ecosystem. And they just look at it for the top 30. 08:50.800 --> 08:57.600 And they serve graph that out. Well, does this year zephyr's in the top 30 list? In fact, 08:57.600 --> 09:03.200 we're number 25 in terms of velocity. And the fact that we're sitting up there with things like 09:03.200 --> 09:10.640 mix OS and so forth is like, I was just blown away personally. It's something that has been 09:10.640 --> 09:16.640 served like, okay, when we served, sorry, evidence that we were down here at the bottom, but when 09:16.640 --> 09:22.240 they just finished measuring it last month, we're sitting at 25. And so how does it all play 09:22.240 --> 09:33.520 into an open source ecosystem? The effort here is this first line. Okay, community controls 09:33.520 --> 09:40.560 its commits, the license is Apache. And you'll see now there's almost, well, we'll be heading 09:40.560 --> 09:49.840 to 2,500 contributors, I think, within the next month or two. But compared to the other projects 09:49.840 --> 09:54.960 and the total commits and the commits each month, you know, we're three or four times. 09:56.160 --> 10:01.680 So actually, this is how we do the calculations for that chart you saw. I used to get up, 10:02.240 --> 10:06.720 and I just go and actually mine the data for all those numbers. So you can go and do this 10:06.720 --> 10:10.800 year on analysis and prove me wrong or find out what the did stats are today because these were 10:10.800 --> 10:19.360 from yesterday. But the methodology is fairly straight, but I was starting looking at this back in 10:19.360 --> 10:25.520 2017, 2018, because at that point in time, it's ever wasn't anywhere near this level. 10:26.800 --> 10:32.320 And I wanted to see, okay, when did we cross the commits, when did we cross the contributors? 10:33.600 --> 10:38.800 And, you know, being able to measure things has been one of those features as you're trying 10:38.800 --> 10:43.440 to get growth in your community. So you know what's working, what's not working, but don't expect 10:43.440 --> 10:47.440 it to change like in a month's time or something like that. You have to be able to be willing 10:47.440 --> 10:52.000 to look over years. Look at trend lines over years, and that's what you can tell after you're 10:52.000 --> 10:57.680 actually making a difference or not with some of the things you're trying. So obviously, 10:57.680 --> 11:05.120 this current is right, it's up there. I was never comparing. Well, last year, there was a 11:05.120 --> 11:10.480 six-eight kernel came out, corrected the calculations. They had like, you know, for the one 11:10.480 --> 11:15.760 released, they had, you know, 1900 contributors, but they were doing about nine changes an hour. 11:16.640 --> 11:20.880 And then you saw in the last slide, there was a first two point, two changes an hour. So it's 11:20.880 --> 11:26.720 about a quarter of the rate of the kernel, and so forth, you know, a nine-year-old project, 11:26.720 --> 11:35.200 and pretty happy with that. But I was around when when it started, okay? And when it started 11:35.200 --> 11:46.480 in 1991, it was a very fragmented ecosystem. So we had, you know, Linux, BSD, all these commercial 11:46.480 --> 11:53.520 versions of Linux, and so forth, out there, which kind of resembled this landscape, when's 11:53.520 --> 12:02.640 effort started? Why do we need another R toss? And so Linux was successful, and so when we were 12:02.640 --> 12:06.720 starting to think about creating this project and working with this project, you know, we 12:06.720 --> 12:14.400 sort of looked at what worked for Linux, and what could we apply? And so, Linux today, it's pretty much 12:14.400 --> 12:18.880 dominating most of the markets, it's not including a bet it, but there are places that are embedded, 12:19.440 --> 12:29.760 where Linux will never fit. So the zephyr is designed to go into those spaces, okay? But in 2016, 2017, 12:30.720 --> 12:37.280 the Linux kernel report, which is available for anyone to go back and read, focused on what was 12:37.280 --> 12:43.200 working for that community. So we looked at that, we did our homework, we looked at what was working, 12:43.200 --> 12:49.200 and we figured, okay, when we set the governance for this project, and we, you know, what can we 12:49.200 --> 12:55.920 apply? Things like short release cycles have worked before, you know, how many people were involved 12:55.920 --> 12:59.920 with Linux before the two six stuff, and it was varying a lot with this really cycles, 12:59.920 --> 13:06.160 but once I went to a regular cadence, that was a start of it accelerating. It pretty much is plateau, 13:06.160 --> 13:11.760 but it's plateaued in pretty high level right now, so we'll have to see what's going to change 13:11.760 --> 13:19.040 it and shift it eventually. But, you know, you need a hierarchical model for your developers, 13:19.040 --> 13:25.600 because one person can't be the funnel, you have to delegate trust, and the hierarchical model is 13:25.600 --> 13:30.080 something that was working for Linux, and so we, as we were thinking about it, we wanted to make 13:30.080 --> 13:37.280 sure we could do something like that, too. Another thing is tools matter, you want to keep your 13:37.280 --> 13:44.880 developer experience as easy as possible. After all, Venus came up with Git, make sure we actually 13:44.880 --> 13:53.280 had a flow, right? And so, when things are frustrating, I'm working it that way, so remembering 13:53.440 --> 13:59.440 that tools matter, and then remembering also that a consensus oriented approach tends to get you 13:59.440 --> 14:06.400 farthest and more stability over time. And, you know, the kernel itself had a very strong, 14:06.400 --> 14:11.040 no regression rule. If something was causing a bug, causing regression, it just got pulled out, 14:11.760 --> 14:18.720 until things got fixed. And this led things move forward faster over time. And then, another 14:18.720 --> 14:23.680 factor that, you know, the new kernel has is the corporate participation in the funding, 14:23.680 --> 14:32.000 it's part of what made the kernel possible, okay, in a stable fashion. Now, when we do the analysis, 14:33.280 --> 14:39.760 and it's primitive, but basically if you look at some of the, the stats from LWN and so forth, 14:40.400 --> 14:46.640 the estimate is about 15 to 20 percent of people working on the kernel of these days are doing it as a hobby. 14:48.800 --> 14:55.040 The rest are getting paid by companies using it in products. And as we set up a virtuous cycle, 14:55.040 --> 14:58.480 this means that people are getting jobs and people are working on things. So, we want something 14:58.480 --> 15:03.040 where people can actually have careers, some degree of independence, and there shouldn't be boundaries, 15:03.040 --> 15:12.400 like you should be able to commit to a laptop. Maybe able to commit to the memory subsystem one day, 15:12.400 --> 15:16.800 and then go commit to another driver in a different day. You should not have to have strict boundaries 15:16.880 --> 15:20.880 of these other places you have to go into. It's open for any contributions anywhere in the kernel. 15:21.840 --> 15:29.600 So, these were the key lessons we took out of that report, and some other ones that we saw along the way 15:29.600 --> 15:34.240 is, you know, keeping things vendor neutral as part of the reasons the links kernel succeeded, 15:35.360 --> 15:42.160 okay. You want to have a mix of companies scratching their own inches. You want to streamline 15:42.240 --> 15:47.440 things and make the easy for developers with the DCO, and, you know, do public reviews. 15:48.480 --> 15:52.800 People are going to review things and sign off on it. That's an indicator of quality, 15:52.800 --> 16:00.400 because their name is associated with it. And so, public reviews and sign-offs help build trust. 16:01.760 --> 16:07.120 And so, this concentrated model in the kernel is doing an email and in person summits. 16:07.760 --> 16:13.760 And then the heart of the adult model signed off by is what's indicating that to some extent. 16:14.960 --> 16:19.600 The intro boundaries we've talked about, tools matter, and so forth. So, these are sort of the lessons 16:19.600 --> 16:26.960 learned. I was focusing on when we were setting up governance. And always, I've been keeping it 16:26.960 --> 16:32.640 throughout my career. When developers get frustrated and irritated, they get really good about creating solutions. 16:33.360 --> 16:41.120 So, keeping it so that the creativity works for your project, not against it. It's important. 16:42.800 --> 16:52.800 So, how did we apply these lessons? Okay. Our vision start, and still is, we're striving to develop 16:52.800 --> 17:00.080 the best and class art class for connected resource constrained devices and build to be secure and safe. 17:00.640 --> 17:07.280 That's what we're aiming for. And we've been pretty true to that vision, and you'll see most of these things fall in line with that and 17:07.280 --> 17:13.360 executing to that. So, one of the things you'll notice in this effort project is the developers' 17:13.360 --> 17:22.400 design directions. RTSC has an elected chair, and the chairs elected every two years. So, it's 17:22.400 --> 17:30.720 affirmation by peers, and the developers look at what they're working with, and if they don't like 17:30.720 --> 17:38.320 something, they have a way of changing it, and we've changed it a few times. You know, when we started 17:38.320 --> 17:46.560 it, we were using cake and cake build, we changed to cake and frequency make. We had basically an 17:46.560 --> 17:50.640 NN on micro-corneal, and we went to a unified current. So, a pretty significant change in the early days. 17:51.840 --> 17:58.080 Initially, we were started on Garrett and Jira, and the developers wanted to go over to get 17:58.080 --> 18:05.920 have an issues for communication and transparency. And they basically determined we were going to have a 18:05.920 --> 18:11.920 code of conduct. We put that in place, and then all the APIs, the house, we've architecture groups. 18:12.000 --> 18:17.040 So, we've got places where people can analyze and talk to each other as part of the project, 18:17.040 --> 18:23.440 but also we make sure we have events and things like that too. So, these are the directions 18:23.440 --> 18:28.480 that we're serving decided. So, now I want to look at these lessons I was talking about. 18:30.400 --> 18:36.080 Vendor-neutral decision making. Yep, we've got multiple companies, and I can show you a list 18:36.080 --> 18:42.080 of the end here of our members, and we've got a lot more than our members right now, 18:42.080 --> 18:49.040 who are actually participating in this tree. The next one was the companies and the individual 18:49.040 --> 18:56.800 team participate. As you can see from this, across the years from 21 to now, we're still 18:56.800 --> 19:02.640 seeing steady contributions from some. We're seeing others increase their contribution like an 19:02.640 --> 19:08.800 XP's been doing. And then we see this other, which is, you know, it could be basically obvious. 19:09.920 --> 19:16.960 From the developer surveys we run, we're finding that about 10 to 15 percent of people who fill 19:16.960 --> 19:22.240 in the developer survey are obvious as well. So, it's not just a corporate environment. There are 19:22.240 --> 19:26.480 people who are actually interested in it for their own hobbies. And, like, the code base. 19:27.520 --> 19:31.680 Peacown, for instance, is one of the ones that is being done by someone who is not being paid 19:31.680 --> 19:35.680 to work on it. If you really want to have a tiny bit library, that's aligned with this 19:35.680 --> 19:39.920 things. If you want to have it common across a couple of our classes. So, if you work with Zefer 19:39.920 --> 19:46.400 on this, we've got people doing home automation systems, home gardening systems, 19:46.400 --> 19:51.520 and using Zefer, and we've had that pretty much since the start. As you can sort of see, 19:52.560 --> 19:56.240 we've got a fair number of companies, some of which are members, some of which are not, 19:56.240 --> 20:02.240 doesn't matter, in this way, but we've got the input. And so, as we're sort of changing things 20:02.240 --> 20:09.040 around and evolving, we do have that, and we have our TSC has people from companies as well as 20:09.040 --> 20:13.680 community participants in there, making sure that the community view is calved. For instance, 20:13.680 --> 20:18.880 as part of the discussion. Contribution guidelines, we've been pretty public about it. 20:18.880 --> 20:22.880 We have really haven't changed that much. As you can sort of see here, if you pay attention to 20:22.880 --> 20:27.680 the details, this file hasn't changed for six years, and it's still pretty accurate. 20:27.680 --> 20:34.560 So, it's been working for us from that side. So, we've streamlined that upstream process. 20:34.560 --> 20:39.600 It seems to be working. One of the things I'm going to highlight, which is a personal pet, 20:39.600 --> 20:45.440 well, it's a pet. So, box of mine, I guess, is the best way to put it. Developers certificate 20:45.440 --> 20:51.920 of origin is a very lightweight way to let people contribute to your community in a structured 20:51.920 --> 20:59.360 fashion. So, you're not signing a CLA and signing over copyright. So, that someone can't 20:59.360 --> 21:03.200 license it from underneath you. So, you're going to build trust by knowing that the licensing 21:03.200 --> 21:08.480 is not good change. It's been very effective for the Linux kernel. And so, it was one of the things 21:08.480 --> 21:13.760 that I really wanted to see we did. You know, a lot of lawyers wanted to see CLAs, and I think 21:13.760 --> 21:20.320 DSEO is a more equitable thing for developers on a personal level. And we've used it in Zephyr. 21:21.280 --> 21:26.480 Public code reviews. Yep, we're doing that. Everything's up on GitHub. You can watch the history, 21:26.480 --> 21:33.280 as things change. You can put in issues, you can track things. They're to follow the consensus 21:33.280 --> 21:38.880 oriented decisions. We try to pretty much work by consensus. If things get escalated, 21:38.880 --> 21:44.800 we have a TSE that can vote. There's no one to cater here. It does go to votes when things 21:44.800 --> 21:50.160 are problematic. Or there's two different strong sides and things like that. But those sorts 21:50.160 --> 21:54.640 of votes only happen maybe a couple of times a year. So, for the most part, it's pretty good 21:54.640 --> 22:00.960 in consensus side. And it's been working well for us. We have a maintainer's file that lists 22:00.960 --> 22:04.880 all of our maintainers and the hierarchical subsystems. Again, following the lesson from the 22:04.880 --> 22:10.480 Linux kernel's maintainer's file seems to work. It's working for us here. And anyone can make a 22:10.560 --> 22:16.880 poll request in any area. No problem there. And the distributed version of troll. The 22:16.880 --> 22:25.600 contributing explains that. And in terms of the releases, we are doing periodic releases. We've 22:25.600 --> 22:32.080 been doing it pretty steadily since the 2.0 release. We were figuring out what our cadence was 22:32.080 --> 22:39.440 in the 1.0 series. And the 2.0, we've been pretty much four months. And every two years, we put out 22:39.440 --> 22:44.960 an LTS. That's the cadence that seems to be working for this community. The Linux kernel has 22:44.960 --> 22:53.280 like a 2-week merge window. We actually use a three month window and then a month of stabilization. 22:54.160 --> 22:59.040 And that's working with this one. We'll revisit it if we need to down the road as things 22:59.040 --> 23:05.600 mature more and more. But this this cadence has been working for us. And then when we come off 23:05.600 --> 23:13.280 that with that LTS, it's every two years. And it's maintained independently for 2.5 years, so we 23:13.280 --> 23:17.360 have an overlap. This is one of actually one of the lessons I learned from Ubuntu, which kept 23:17.360 --> 23:22.960 that overlap on the LTS' because it made it easy for people to have a transition window. And so 23:24.000 --> 23:27.760 when this was a doc, like I said, when we decided to try to start figuring out what the LTS policy 23:27.760 --> 23:32.480 was going to be, this was a practice. Which is if you're going to stabilize on something for a 23:32.480 --> 23:36.800 period of time, make sure that it's a good transition window for people who have to upgrade. 23:36.800 --> 23:40.160 And as we move more and more towards like the CRA and longer periods of time, 23:40.960 --> 23:46.240 I think there'll be more companies coming in with business opportunities to maintain things 23:46.240 --> 23:50.640 for these longer periods of time. I don't think you'll see the community maintaining things longer. 23:50.640 --> 23:54.560 Although there is discussion with Defer of extending this another year, possibly two. 23:55.600 --> 23:59.520 But that's the discussion and we have to figure out a way to be efficient and smart about it. 24:00.080 --> 24:06.480 And so this is working progress. So are we going to try to make more two to four weeks 24:06.480 --> 24:13.120 stabilize and our long-term support release? Actually, you'll notice that our LTS-2 is pretty much 24:13.120 --> 24:19.920 EOL. So I'm expecting this page to get updated very shortly and it'll all be updated by the time 24:19.920 --> 24:29.120 we put our 4.1 release out. But I don't know if there's going to be another point release or not 24:29.120 --> 24:34.000 with some of the security issues. I've got to talk to someone about that one. But as you can see, 24:34.000 --> 24:42.960 we came out with the 3 LTS-3 in July and that's being maintained and 4 came out in November 24:43.760 --> 24:50.240 and it's going to be maintained until the end of the next like until the 4.2 comes in. 24:50.240 --> 24:58.160 So we'll be putting out the 4.1 early March, brings the cadence and working from there. 24:59.120 --> 25:04.880 And then we'll be keeping things marked as end of life. So as you can sort of see, we have applied 25:04.880 --> 25:12.240 the lessons from Linux to the large extent in Zephyr and it's been paying off for us. So okay, 25:12.240 --> 25:16.560 let's happen with security. You said you wanted to be in security and we do best practices there, right? 25:20.320 --> 25:26.160 Well, when we set up the project, we pretty much came up with this diagram and we wanted to not have 25:26.160 --> 25:33.040 blockage on contributions upstream to tip and we wanted to release this going there. But the long-term 25:33.040 --> 25:40.560 stable would happen periodically. And so when development is happening on tip and things can 25:40.560 --> 25:46.800 go into stable and then any processes can be applied from that. Actually, realistically, they're 25:46.800 --> 25:53.360 being applied into development now. But there's a subset that's going to use for audits. And that's 25:53.520 --> 26:00.160 kind of the model we've had from the start when we're keeping to it so far. So the 4.0 releases are 26:00.160 --> 26:06.720 latest released. We've put a lot more stuff into it after we've, you know, with the last LTS out 26:06.720 --> 26:11.280 and the start of things they'll be showing up for the next LTS. And so the release notes are there, 26:11.280 --> 26:18.800 everything is public and we're moving forward. Our last LTS had a new hardware model and actually had 26:18.800 --> 26:22.800 the precision time protocol stuff supported, which is going to be good. And we actually extended the 26:22.800 --> 26:30.640 S-bomb support to actually support the pearls and CPEs as well as SPX 2.3. So that's been there. 26:32.480 --> 26:37.760 The S-bomb support has actually been in Zephyr since 21 and I've got a slightly 26:37.760 --> 26:42.560 little slides in a minute about that. But as I say, we have a support window. We're probably 26:42.560 --> 26:46.240 about it. We're transparent. How long we're going to do it as a community and we've got 26:46.240 --> 26:53.360 buy-in from the community to do this. And the key for these are, it's product focus, there's 26:53.360 --> 27:00.560 updates, security updates. So we actually have a P-Cert team with this project. Volunteers that is 27:00.560 --> 27:06.160 actually monitoring the vulnerabilities and enhance the things. And we've been, you know, 27:07.280 --> 27:12.800 we have a testing window and we're working with this. And then we put out the point releases 27:12.880 --> 27:17.520 during the course of the support window with the security fixes. And if you actually go and 27:17.520 --> 27:23.600 look at our charter, it was right in there from the very start that we would have a security committee 27:23.600 --> 27:29.680 that would be focused on these types of issues. And setting up your governance so that these places 27:29.680 --> 27:37.280 have a clear role was one of the lessons learned. Okay? The other thing was, okay, how do you do 27:37.280 --> 27:45.040 security well? Well, back in 2016, the CI, our core infrastructure initiative project was starting 27:45.040 --> 27:49.680 and they came up with the best practices badge. So our security team looked at it after 27:49.680 --> 27:54.880 documenting the threat models and some of the other basics. And so I said, okay, let's look 27:54.880 --> 28:04.320 at this and see what's like. And, you know, can we get it? And so what these criteria were for 28:04.320 --> 28:13.840 was to effectively improve open-source projects for best practices and letting people self-certify 28:13.840 --> 28:19.120 to meet criteria. And we actually found it a really good guide to improving our processes. 28:20.240 --> 28:29.520 So it moved into the OpenSSF so it's there still today. But it's now OpenSSF best practices badge, 28:29.520 --> 28:37.360 curls there, zephyrs, ends there, the curls there. And so it's possible for projects to go 28:37.360 --> 28:41.840 through this. And you'll probably see a reasonable correlation between these projects that have 28:41.840 --> 28:46.160 gone through it and having best practices as well as a big very popular because they can be trust, 28:46.160 --> 28:49.520 there's an element of trust because you've got public access stations about your policies. 28:51.840 --> 28:56.400 And so there's currently now there's a passing silver and gold when the project started, 28:56.400 --> 29:03.200 there was just the passing. And at the higher levels, there's slightly more strict best practices 29:03.200 --> 29:10.480 that are getting invoked. And then the criteria is, you know, there's only 23 gold at this point 29:10.480 --> 29:16.640 still. Or there may be, I have to double check that number beforehand. But, you know, looking at 29:16.640 --> 29:21.680 the criteria, there's 23 criteria on top of the silver. So there's 55, and so the passing 29:21.680 --> 29:28.240 you have to go through a 67 criteria. And I test one way or the other on those criteria. And then 29:28.240 --> 29:34.720 another 55, I'm top of that, and then another 23. And if you can do all those and have it all 29:34.720 --> 29:39.520 probably get tested, you can be quote-and-quote gold. Some of them may come straight, pretty, you know, 29:39.520 --> 29:44.320 some of them take different works. So we started doing this bit. Okay, yes, it's like a good 29:44.320 --> 29:52.960 idea, let's go forth. And when it launched, and so pretty much within six months, we achieved 29:52.960 --> 29:58.560 the passing for Zefer. So it's like, okay, good. We're passing. Yay, we're doing the right thing. 29:59.600 --> 30:03.920 We had a good security documentation. At that point in time, and the security documentation is 30:03.920 --> 30:08.320 something, you know, we keep up to date, keep working with the project, and we want to focus 30:08.320 --> 30:12.800 around security development, security design, and security certifications. And so this is something 30:12.800 --> 30:17.360 that we consider an ongoing process rather than a point in time. So our security committee 30:17.360 --> 30:26.080 keeps looking at this type of thing over time. And price and price, we stopped passing, 30:26.080 --> 30:34.000 in 2017. Remember me showing you that developers controlled decisions? The infrastructure 30:34.000 --> 30:40.800 changed. The tool went in and was looking for links to keep working for the evidence and stop 30:40.960 --> 30:46.960 working. So since we finished all our migration, we got ourselves passing again. By updating 30:46.960 --> 30:52.240 a lot of links and things like that. And so six months later, we were interesting. And we went, 30:52.240 --> 30:57.280 okay, and failed us for a while, we can come back. Let's try for silver. Let's see what's going to 30:57.280 --> 31:03.440 happen there. So we did that, and the roles in response, please, had to be documented. Again, 31:03.440 --> 31:10.560 these are all good practices for anyone to be secure software. And we had to have a longer road map, 31:10.560 --> 31:17.200 and we had to document in certain places. When we started looking around by, we become a DNA. 31:18.160 --> 31:21.360 So I don't know, how meant to show hands, how many people are familiar with CVD numbering 31:21.360 --> 31:31.840 authorities or CNAs? That many. Okay. These are the ones that assign, these are the people. 31:31.840 --> 31:39.280 So CVEs are the ones that assign the vulnerabilities. And they basically have control over the 31:39.280 --> 31:45.920 vulnerabilities. And there's not that many open source projects that you're doing it. But it really 31:46.800 --> 31:53.920 lets you have control as opposed to an integrator trying to guess. So if you've got a big enough 31:53.920 --> 32:01.840 team that someone that cares about this, it's something to look at. We basically had to change 32:01.840 --> 32:07.120 our processes a bit. We changed our processes in a useful way. You know, that we had to review 32:07.680 --> 32:12.800 basically, we had to change some review processes a little bit. We had to create some new email lists. 32:12.800 --> 32:16.720 So in that sense, it wasn't too hard. And then we had to interact with their system, 32:17.280 --> 32:22.400 which is automated since then. So we defined our scope, created our value of contact. 32:23.200 --> 32:31.200 We have a spot for contact from minor. And we have ways of saying who should be added on to lists. 32:31.200 --> 32:38.880 So 2017, we got announced as CVEs number 30. So hey, we've got our goal passing bad. We've got 32:38.880 --> 32:46.000 our, sorry, we've got our passing badge. You know, we're CNA. Cool. We were listed in the MBD, 32:46.000 --> 32:50.560 which is a national vulnerability database for those who are in the last room before. And so you can find 32:50.560 --> 32:55.040 us there still. And this is what it looks like if you actually are curious what an entry looks like. 32:55.920 --> 33:01.760 And we have a product security, a project security, not product, incident response team. 33:02.480 --> 33:08.880 And our security architect leads it. And we have volunteers from our committee who, you know, 33:08.880 --> 33:13.680 basically, help with the triaging and then interacting with the maintainers to make sure we get the fixes in. 33:14.880 --> 33:20.320 And so, so it's satisfying the program rules and requirements. So we kept going. 33:21.040 --> 33:25.920 We were almost silver 2018 and we finally got there at towards the fall of 2018. 33:27.040 --> 33:32.000 And part of the challenge for us was this program assumed you were writing an application. 33:32.560 --> 33:38.400 An operating system was not what it was thinking about. So we had to write a little application 33:38.400 --> 33:42.800 so we could catch one in the criteria, which has been a sore point for me as you can tell. 33:42.800 --> 33:53.760 But we kept on and then, okay, Zepher has been very good since the start of putting the licensing 33:53.760 --> 33:57.920 in each file, which is one of the things that people stumble about at the gold level. 33:58.480 --> 34:02.160 And so we already had that. So it's a pretty easy thing for us to get to gold at this point. 34:02.880 --> 34:08.480 And so we actually got gold in 2019. We were the fifth project to get it. 34:08.960 --> 34:12.800 We got it before the Linux kernel did, just saying. 34:15.200 --> 34:21.520 And then, you know, we're pretty smart. And then someone paid NCC group to audit Zepher. 34:23.120 --> 34:32.000 And we got our first bulk vulnerability report at 26 issues. Okay, secure your team crank in. 34:32.000 --> 34:36.400 So we actually addressed this all. You can see this in the LTS histories and so forth. 34:37.280 --> 34:43.280 And they're all addressed. And most results is one or more TVs from what they got. 34:43.280 --> 34:48.400 So it was a good for the project. If you ever have the chance to get your project 34:48.400 --> 34:54.000 audited by a third party, do it. Because you'll find things that don't find things that you don't 34:54.000 --> 35:02.160 overlook over time. And so, you know, they came up with a report after we finished with 35:02.800 --> 35:07.760 them. Most issues were fixed in a reasonable time. It was a nice thing to see for an open source project. 35:09.120 --> 35:13.520 One piece we decided to disable feature. That's the most sensible approach. 35:13.520 --> 35:20.160 So we did increase our embargo window to 90 days. So that we had 30 days as a project to fix things upstream. 35:21.360 --> 35:30.400 And then we wanted to give people doing prod ducks with our project, 60 days to fix things. 35:31.360 --> 35:35.920 Because they've got customers to interact with sometimes. So we were trying to keep 35:35.920 --> 35:41.680 products and customers in mind right from the start. And, you know, we had to improve some processes. 35:42.560 --> 35:49.600 And, you know, we did this and we've documented our website, etc. And so learning from working 35:49.600 --> 35:53.600 these things and learning from the experiences we've kept on improving our best practices. 35:54.640 --> 35:58.320 And we wanted to make sure that we were first classes and for product makers. 35:59.200 --> 36:05.040 And so this vulnerability registry was also created where if someone can show us evidence that 36:05.040 --> 36:09.840 they've got a zephyr in a product, we will add them for free. They do not need to be a member of zephyr. 36:10.880 --> 36:16.640 So they can basically say, hey, let me know, here's this product I'm shipping. Let me know 36:18.000 --> 36:22.320 if there's any vulnerabilities. So we have a way of pushing information to them so that they have 36:22.320 --> 36:28.640 them alert and can work in their embargo windows. And so that has been very useful. 36:29.760 --> 36:36.400 The issue is, you know, people letting you know that they're using your product is interesting 36:36.400 --> 36:40.800 sometimes. They don't want to. And some part of my fun, part of the fun, you know, as I have, 36:40.800 --> 36:45.600 is put it embedded. It's trying to find where zephyr is being used. I get lots of interesting 36:45.600 --> 36:51.200 things. I'll show you some of the products later, but we also had challenges before the concept of 36:51.200 --> 36:59.200 a vex came in where we didn't actually use the files in fnet that had the vulnerability. 36:59.200 --> 37:03.120 If you had just looked at the component level, you would have thought zephyr was vulnerable to 37:03.120 --> 37:08.320 unusual 33. Because we had the same fnet version or some of the files from the fnet version, 37:08.320 --> 37:12.800 but it really wasn't in there. None of the files were in there. And there was no way of signaling 37:12.800 --> 37:19.680 you were not impacted. Other than writing a blog post. So when the whole concept of vex came out, 37:19.680 --> 37:23.760 I was definitely supported because I lived through the reality of not having a way of signaling 37:23.760 --> 37:27.760 not impacted. And if you're just looking at the components, too many false positives, 37:28.880 --> 37:32.720 too many problems. And you don't want to fix things and upgrade things and cause perturbations 37:32.720 --> 37:39.760 when you don't need to. So this is also an element for the s-bomg generation. And if you work with zephyr, 37:39.760 --> 37:45.760 you'll find it can generate three or four s-bomgs depending on what you've got in your system. 37:46.080 --> 37:52.080 And the way you do it, you can do one for the zephyr sources, a source s-bomg for the sources. Here, 37:52.880 --> 38:00.080 a source s-bomg for your SDK, a source s-bomg for your app. And then you can have your build s-bomg 38:00.800 --> 38:07.840 linking and referencing that to your source s-bomg as to how everything is connected. So you have a very 38:07.840 --> 38:12.240 precise source of truth, which is what we're going to need for safety down the road just 38:12.400 --> 38:16.240 foreshadowing here. As well as quite frankly, it gets through the whole class of false 38:16.240 --> 38:22.560 positives for security folk. And if you want to generate these s-bomgs with zephyr today, 38:23.920 --> 38:27.920 this is it. That's all you need to do. You just initiate some variables when you're doing your 38:27.920 --> 38:35.600 builds with west. And these will come out automatically. And if you actually, there's one of 38:35.600 --> 38:41.600 our members and my crew has a re-node simulator and they use zephyr as a test suite for their 38:41.600 --> 38:53.200 simulator. And so they build 875 boards with it. And everything in there that has built past or 38:53.200 --> 38:59.520 generated has three s-bomgs. And if you're mouse over, you'll see download s-bomg and you can download 38:59.520 --> 39:07.200 those s-bomgs across those apps. So yes, we can do it at scale with c-programs for those who care about that. 39:07.280 --> 39:12.320 The other place that is doing it at scale for c-programs is a s-bomgs project. They've been 39:12.320 --> 39:18.800 doing it as well. And Joshua here is a life of implemented at all. So you may want to find 39:18.800 --> 39:28.160 him after the call. Thanks. But this is what it looks like. Okay, we put in links, symbolic links 39:28.160 --> 39:32.880 to the other files. And then we refer to exactly which stuff sees me to.is and which 39:32.880 --> 39:39.840 studies made it into the alpha images. And so we can be that precise. And that just comes out of the 39:39.840 --> 39:46.320 build information. And so our vulnerability infrastructure went to GitHub in 2021. This was one of 39:46.320 --> 39:51.200 the decisions that our security committee made. We wanted to do that because things were more easy 39:51.200 --> 40:00.400 to get stats and metrics. And so this gives a picture of our CV publishing history and C-WE breakdowns. 40:00.400 --> 40:03.760 For those who understand the weakness in the iterations associated with the vulnerabilities, 40:04.880 --> 40:09.280 this is there. And then we can look at scoring and so forth. So we have a ways now starting to 40:09.280 --> 40:17.360 look at metrics in this area. And we decided that we wanted to branch out wider than just a committee. 40:18.880 --> 40:25.280 And so the security team basically, the chair and the architect decided to start working group as 40:25.280 --> 40:32.320 well. And they wanted to start doing public work on some of the security standards. So Etsy and 40:32.320 --> 40:37.600 303645 was one of the things they were working on. And so that's open to anyone to join 40:37.600 --> 40:45.760 whether you remember or not. And they did work at this on 223. And you can look at the public documentation 40:45.760 --> 40:50.000 of each provision and the test station, which is sort of like what you'd see in the best practices. 40:50.080 --> 40:57.920 It's basically public assertions about the provisions. And last year, the project decided to 40:57.920 --> 41:03.120 actually say ask NCC group to do another audit on us, to see if things have changed, things have 41:03.120 --> 41:10.000 regressed, et cetera. And we wanted to find out, are we better, we've got challenges, you know, we 41:10.000 --> 41:16.560 missed things. And so they actually went and looked at our 3637, which was our LTS. And two 41:16.640 --> 41:22.480 most of the ready issues were found and went informational. So they weren't finding things as 41:23.760 --> 41:28.320 that initial traumatic report was for us. So in that sense, it was a good result. In the sense 41:28.320 --> 41:34.400 that we really had not regressed and we kept the quality level high. And designing the scope of these 41:34.400 --> 41:40.000 things is hard. But it does increase confidence if you've gone through through party audits. So 41:40.000 --> 41:48.080 if your project has the ability to do it, I consider it. If you're lacking funding to do it, 41:48.080 --> 41:55.280 OpenSSF's AlphaMega project is willing to probably support, if you can make a good case for it 41:55.920 --> 41:59.920 and put an application in. So there's funding potentially there too. Because they're trying to increase 41:59.920 --> 42:06.880 security for open source overall. And if you're interested, Flavio and David, they give a really 42:06.880 --> 42:12.560 good talk at the Zefford and all of our summit. And I encourage you to listen to more details 42:12.560 --> 42:19.920 from the experts, not me. But as you can see, we're doing all of these things. And the S-plum 42:19.920 --> 42:25.040 generation is very much lining up for what we're going to need for CRA down the road. So what about 42:25.040 --> 42:31.760 safety? Well, we've got a subset. We're going to be looking at for Zefford from the auditable. 42:31.760 --> 42:37.280 And for Zefford, we want to try to actually get something that's going to work with a traditional 42:37.280 --> 42:40.640 vehicle model. Because we're in our class because we're using small things that are usually 42:40.640 --> 42:45.280 done bare metal by security experts. We want it to be able to interface with that community. 42:45.280 --> 42:50.320 So we start our prototyping things about four years ago. And we've designed the security 42:50.320 --> 42:54.480 committee's side of the initial certification focus is just going to be the kernel. But we're 42:54.480 --> 43:00.960 doing the analysis out in the open. So people want to extend it. They can. And we've basically 43:00.960 --> 43:04.400 worked through our background with the collateral we need to put together. And we have a 43:04.400 --> 43:12.000 functional safety manager in the project now. And we've made a strategic decision to actually 43:12.000 --> 43:18.560 document the requirements in public as well as to do the linkage in public. Now the artifacts 43:18.560 --> 43:24.880 specifically will be for the members. But the working group is doing all of the requirement 43:24.880 --> 43:29.600 traceability. And if there's something that you care about that's outside the scope, 43:30.720 --> 43:40.160 even now I know the methodology. And we actually, in December, we got our concept to prove 43:40.160 --> 43:44.400 from to sued on our methodology that we're using. So they think if we can do what we say we're 43:44.400 --> 43:48.800 going to do, they'll be able to certify us. We just have a lot of work to do between now and then. 43:48.800 --> 43:55.520 And so that's, we just sort of mentioned that in announcement yesterday. So what are the results 43:55.520 --> 44:02.320 from applying all these best practices? Lots and lots of new products. They're showing up all over 44:02.320 --> 44:08.880 the place in various things from wearable rings for biometric monitoring. Livestock trackers, 44:08.880 --> 44:14.800 Chromebooks, the framework laptop. Some of these are actually at this effort stand. So if you want 44:14.800 --> 44:20.400 to look at some of this stuff, just over there, you can go and talk to them and then you've got 44:20.400 --> 44:26.320 some of the devil or person with some of their demos. They're happy to talk about that. But from 44:26.320 --> 44:30.480 winter binds to hearing aids, things that need to ever source constraints, it's a good fit for. 44:31.440 --> 44:38.160 We're seeing about 6.9 almost 7,000 forks in the wild. And things like winter join systems 44:38.160 --> 44:42.880 and so forth. And in fact, when I was looking at this last night, I actually spotted Tesla there. 44:42.880 --> 44:50.560 So it was so like, okay, Tesla's now starting to work upstream. That's cool. We're supporting 44:50.560 --> 44:55.840 all the main arch to recharge hardware architectures. We've got a pretty complete stack. The developers 44:55.840 --> 45:00.080 are generally referred to this as batteries included. If there's a component technology, there's 45:00.080 --> 45:05.680 a few that are missing that I'd love to see. But a lot of the key ones are there already integrated 45:05.680 --> 45:11.520 into the effort. There were 220 sensors, so you just can figure this information in and build it 45:11.520 --> 45:16.640 and then you can start working with it. And there's over 700 boards in our repo at this point now. 45:17.440 --> 45:23.280 And it continues to grow. We are seeing our average number of unique contributors per month. 45:23.280 --> 45:29.440 See my comments about looking at over time. Our, you know, and actually you saw in the other chart 45:29.440 --> 45:38.720 where 300. So when we start seeing 225, it'll be seeing us at 300 level. And you can sort of see 45:38.720 --> 45:43.440 that our average commits. This is more contributors. There's more commits. You'll surprise there. 45:45.040 --> 45:53.360 But we've got the various history and the stars. So the developers like it enough to give 45:53.360 --> 45:58.400 a stars for what metric that's worth. But we are seeing each release. We're seeing about third, 45:58.400 --> 46:04.240 new contributors usually happening now, which is kind of cool. And then we're seeing lots of 46:04.240 --> 46:09.760 floating and visiting activity going on. This doesn't give us why are the products are. We have 46:09.760 --> 46:13.760 to use other clues for that. But at least it starts to less know people are using it in certain ways. 46:14.720 --> 46:19.600 And our ecosystem is grown, significantly in the last two years. We're getting a lot more 46:19.600 --> 46:24.640 ideas working with SEFER. And we just got another compiler, IAR's compiler working with SEFER, 46:24.640 --> 46:30.000 which is a safety certified compiler. So that'll be part of the whole safety story. We've got a wide range 46:30.000 --> 46:37.520 of debugger and tracing tools, as well as emulators now. And people are giving training. AC6, 46:37.520 --> 46:42.560 dualists is just a new one. It's just joined us. And they're doing training, as well as people 46:42.560 --> 46:47.840 do consulting. So if you don't have SEFER skills in health, there's someone people who work with you. 46:48.800 --> 46:53.120 We've got a fair amount of security stuff. We're coming with a trust in firmware, PSA, 46:53.840 --> 46:58.640 and VEDTLS with SEFER. We've got some tiny ML situations going on, as well as various 46:58.640 --> 47:02.560 language run times, including Rust. A lot of work's going on with Rust right now. 47:03.280 --> 47:07.440 And Micah Python. And then we've got a variety of other firmware instead of a recently 47:07.440 --> 47:14.000 that have been integrated in. A lot of people are using working with SEFER for remote management 47:14.000 --> 47:19.120 of systems, a IoT devices. And so various companies do a lot of work with it, and those are there. 47:19.120 --> 47:24.400 And as well as we're now starting to see graphical interfaces showing up. So we were seeing LVGL, 47:24.480 --> 47:28.960 but we've just finished having Micah Edge and QT group join us as members, and Slintis there 47:28.960 --> 47:33.840 now too, as well as some of the robotics. So we're continuing to grow our ecosystem around SEFER 47:33.840 --> 47:39.360 as well, which is also helping make it more accessible. These are our platinum members. They're 47:39.360 --> 47:44.320 the ones who are, you know, funding quite frankly the safety work, because this isn't cheap 47:44.320 --> 47:50.960 to put them out of like. And we have a fairly wide range of silver members now, including one or two 47:50.960 --> 47:55.200 that are not willing to have their logos up there yet, but I'm hoping as soon as they ship products 47:55.200 --> 48:02.880 they will. So what's next? We've got focus areas. We've done a lot of best practices, and there's 48:02.880 --> 48:08.080 still more to go. It's a journey. It's not a done deal. The growth on maintainers has been a stress 48:08.080 --> 48:15.120 point, because as more consumers come in more reviews, maintainers are fragile resources. And so 48:15.120 --> 48:20.160 figuring out how we can be scale things more humanely for everyone is a challenge and still get 48:20.160 --> 48:25.040 responses and reasonable ways. We need to do our own benchmarking. We learned that when the hard 48:25.040 --> 48:32.320 way last year, when the digital real-time performance benchmarking, and we've got work on our 48:32.320 --> 48:38.320 testing for structure we want to rework. And then quite frankly, the CRA readiness is a big topic. 48:38.880 --> 48:43.280 The heart of the SEFER community is here in Europe. Okay? About two, there's the developers 48:43.280 --> 48:47.760 that respond to surveys that are coming in from Europe at this point. We are spreading out 48:47.760 --> 48:53.600 other geographic regions, but CRA is going to be a big consideration, and making sure that 48:53.600 --> 49:00.320 as a project we're doing everything we can to make it easy for people. And then, as we go after 49:00.320 --> 49:05.600 safety, we require requirements. Well, developers don't know how some developers do, but most 49:05.600 --> 49:08.720 developers you can't assume know how to formulate requirements, it's a different mental discipline. 49:09.520 --> 49:14.000 And so getting people who know how to work with requirements, ecosystem participating with us, 49:14.560 --> 49:19.520 is a current challenge. And we've got some outreach going on there. So getting them working here, 49:19.520 --> 49:24.320 possibly getting them working with Linux, these are things that might see us start to do. And then 49:27.600 --> 49:30.720 something that's near and dear to my heart is we still need to keep on improving diversity. 49:31.680 --> 49:39.200 So, if you consider yourself a member of an underrepresented group in some way, shape or form, 49:39.200 --> 49:43.600 it's a short time in survey, and kind of no one know what the blockers are. 49:45.280 --> 49:48.240 So, what is the blockers? What's stopping? 49:49.760 --> 49:54.880 And I have questions. I have kinds for any one who has questions. Any questions? Sure. 49:55.440 --> 50:03.840 Yes, I think if the presentation was very interesting. My question is given your proximity to Linux, 50:03.840 --> 50:08.720 how did you manage to avoid catching the email workflow? 50:11.120 --> 50:17.360 The developers decided they wanted a workflow that would actually track and have things linked 50:17.360 --> 50:20.400 whether rather than trying to do it afterwards. So, it was a developer decision. 50:21.040 --> 50:25.120 So, do you think that affected the outcome your success? 50:25.760 --> 50:31.280 I think so. I think there's a factor. A lot of the kernel developers have muscle memory. 50:32.080 --> 50:35.360 I'm working with email flows and have filters and a bunch of other things. 50:35.360 --> 50:39.040 The people coming in into the newbies and so forth have to get trained on this. 50:39.040 --> 50:40.720 And some of that barriers into your now. 50:41.840 --> 50:43.360 Okay. Thank you. 50:43.360 --> 50:43.760 Sure. 50:53.680 --> 50:59.280 I think you were saying that some devices and manufacturers that use effort are a 50:59.280 --> 51:02.720 look sent to disclose that. Why? 51:04.720 --> 51:11.920 So, I was talking to a one of the award winners at CES at the start of the month. 51:13.120 --> 51:18.320 And you had a really interesting device. Okay. One an award. 51:18.880 --> 51:23.120 And so, I went, this looks like an it might have it in there. So, I went and started talking to him. 51:24.080 --> 51:29.680 And he was playing dumb initially and then he realized I was with Zephyr and said, yeah, we're using 51:29.680 --> 51:33.680 Zephyr. I said, well, you'd be visible about it. And he went, no, it's my secret sauce. 51:36.240 --> 51:42.640 So, as an open source project, the fact that people are feeling like it's proprietary edge for them 51:42.640 --> 51:47.200 to be using Zephyr, it's good in some senses, but it makes it hard for people to see that. 51:47.200 --> 51:51.200 On the other hand, I met other people at CES. They were doing some really cool things. 51:51.280 --> 52:00.560 So, they were quite happy to be visible about Zephyr. One of which was a harvesting of motion for energy. 52:01.600 --> 52:06.480 And they were using Zephyr for buttonflix, catching up the energy with an induction coils for 52:06.480 --> 52:11.760 general energy. And then being able to send stats over Bluetooth to another device. 52:12.880 --> 52:18.320 Another was putting a device on an engine and the vibrations generate power. So, you don't have the 52:18.320 --> 52:23.520 back of the batteries. And then you're still sending a stat. So, there's, you know, there's some 52:23.520 --> 52:27.920 people who are very interested being cool. A lot of the venture capital are happy to be visible, 52:28.960 --> 52:33.840 but some of the others, the big manufacturers in particular sometimes are a bit reluctant to be visible. 52:37.040 --> 52:41.840 And if you know of any products, running Zephyr, and you want to talk to me about them and let me know 52:41.920 --> 52:45.040 how would love it. Thank you. What's we going there? 52:47.280 --> 52:55.200 Sorry, hey. Hey, thank you for the talk. You mentioned if you, if we can show a project 52:55.200 --> 53:01.360 using Zephyr, you can get on the list for the security firm. What's here a product? 53:01.360 --> 53:07.680 Is my homegrown home automation system in Zephyr? Only I am using also a product or 53:08.400 --> 53:12.880 yeah, if you've done a, like, if you've got a homegrown project that you do at home, 53:12.880 --> 53:18.400 we would very much welcome a blog explaining how you do things and to share that knowledge with others 53:19.200 --> 53:25.760 and point them at your code bases and whatever you want to do. So, we would welcome the blogs and if you've 53:25.760 --> 53:29.760 got pictures of your thing, we would probably put it up on, and project sister shell people. 53:29.760 --> 53:35.440 It's pretty as surprising. Yeah, now we want this much to decide and give it a share. Go for it. 53:36.080 --> 53:45.840 You, sorry, just for what. In the beginning, you taught how the Zephyr community adopted a lot of 53:47.120 --> 53:53.280 good practices from other communities. Is there anything that you say that is completely different 53:53.680 --> 54:01.040 within the Zephyr community of the mainstream? I don't think anything is different on 54:01.040 --> 54:06.560 of the sun. There's all variants from different places. I think we've listened, well, 54:06.560 --> 54:11.920 the thing that's different about Zephyr's were actually embedded. And there's not that much 54:11.920 --> 54:17.440 really, like I say, the embedded ecosystem is not your big name. The fact that we're actually 54:17.440 --> 54:23.920 showing up on the top 30 open source projects, overall was like completely mind blowing to me, 54:23.920 --> 54:28.720 because I've always been used to embed a piece of thing off to the side. Like I say, we're