WEBVTT 00:00.000 --> 00:10.240 Well, thanks a lot for coming. This is a very optimistic talk. Quite tricky to write 00:10.240 --> 00:15.120 it turns out. It's quite tricky to push your testing upstream. You might or may not know 00:15.120 --> 00:21.040 my name. From talking about no, I'm an open QA testing. This isn't the same talk again 00:21.040 --> 00:25.440 talking about no, I'm an open QA testing. Even though I am going to relate it to no because 00:25.440 --> 00:29.760 that's what I'm involved in. And I'm going to relate it to open QA because that's pretty 00:29.760 --> 00:35.040 much the best tool we have. But it's more about kind of a vision of the future than what we have 00:35.040 --> 00:43.040 today. Firstly, we've come a long way, right? The desktop Linux world. In the early 2000s and 00:43.040 --> 00:47.440 in the late 90s we've installed the Linux distribution and we'd go, okay, like the graphical 00:47.440 --> 00:52.000 desktop doesn't work. Let me fix my X config and then move it up. It'd be nice to have 00:52.000 --> 00:58.720 Wi-Fi. One of why that doesn't work. And then if you were lucky you'd have audio as well. So today 00:58.800 --> 01:02.240 things were a lot better. I mean that they used to be a distro as well where you couldn't do 01:02.240 --> 01:06.960 desktop grades. They'd say delete your old installation and install the new one. It'll work better. 01:07.600 --> 01:14.400 And we've come a long way. Now we have much better testing. We test things like upgrades and 01:15.600 --> 01:21.280 I like to use this example as one of the things that's really horrible to test. If you open a 01:21.280 --> 01:27.280 MP4 file in Fedora and you try to play it doesn't work because you don't have the right codec. 01:27.280 --> 01:31.840 So this dialog box looks very simple. It just pops up and says, oh, you need a codec. 01:31.840 --> 01:37.520 But the code underlying this to make it work is hugely complex. You have a G-streamer plug-in, 01:37.520 --> 01:42.000 you have something in the video player to handle messages from the plug-in and you have 01:42.000 --> 01:49.360 known software which is incredibly complex in itself. A testing app is quite tricky, right? 01:49.360 --> 01:55.440 Because you need all of these bits integrated together. So I'm a big fan of end-to-end testing. 01:55.440 --> 02:01.920 Of the kind that OpenQA does, where you test the entire system, you test the entire operating 02:01.920 --> 02:07.040 system and either on hardware or on a virtual machine. So you have your current, you control the kernel, 02:07.040 --> 02:13.920 you control the user space and everything. And another key part of this type of testing is that you 02:13.920 --> 02:19.440 navigate the system like an end user would. So you don't connect to the serial console and type serial 02:19.440 --> 02:25.200 commands. That's not how most people interact with the desktop machine. You inject keyboard events 02:25.280 --> 02:30.880 and mouse events because real users use a keyboard and a mouse. We never go to UI by looking at 02:30.880 --> 02:38.400 what's on the screen, right? So if you want to test the OS like a real user, a real user, 02:39.680 --> 02:43.760 if they're trying to find the console button, they look for a shape that looks like a button and says 02:43.760 --> 02:48.880 cancel on it. Doesn't matter if it's here or if it's here, really, just they're looking for that. 02:49.840 --> 02:54.480 And OpenQA has really good support for looking for something that looks like a button and says 02:54.480 --> 02:59.440 cancel. And if the non-developers decides and the next release it needs to be a pair instead of 02:59.440 --> 03:03.840 on the right, the test might still pass because the user could still find the button. 03:04.880 --> 03:10.880 So who's doing this kind of testing today? I'm going to open the floor to questions. 03:10.880 --> 03:13.200 Shout out to the angels and distros that are doing this. 03:13.280 --> 03:22.720 Fordora? Yeah. One of the earliest to adopt OpenQA. Open Susa, definitely, the original. 03:23.360 --> 03:27.520 Anyone else? I'm a Linux. Anyone else? 03:28.800 --> 03:32.800 Debbie and yes, Debbie and actually has a quite a well-maintained OpenQA instance, 03:32.800 --> 03:40.320 many times by a couple of volunteers. So I have a list. I don't think this is a complete list of 03:40.320 --> 03:49.840 about 10 OpenQA instances out in the wild. If I missed anyone, then let me know after. 03:50.640 --> 03:54.800 Each one's kind of operating independently from the others, so there's not enough collaboration 03:54.800 --> 03:59.520 between these today, which is what I'm going to go into in the talk. But it's very good to see, 03:59.520 --> 04:05.760 right? We've come a long way. Well done, everyone. If you work on downstream tests, have you ever 04:05.760 --> 04:09.440 had anyone to upstream help you write the test or help you maintain the tests? 04:11.760 --> 04:15.920 Has anyone ever said, oh, in the next version of, you know, we're going to replace GED with 04:15.920 --> 04:20.080 non-text editor, so I'll help you update some new tests for non-text editor. 04:22.560 --> 04:28.960 I think we're still lacking in this area, right? The downstream QA pipelines are still kind of 04:28.960 --> 04:34.320 invisible to upstreams, unfortunately. And a lot of people upstream of volunteers, everyone's 04:34.400 --> 04:38.720 overworked, so this is kind of understandable, but can we do anything about it? 04:42.640 --> 04:46.800 So what developers want is fast-food back on our changes, right? If you develop an app, 04:47.360 --> 04:53.360 it's not much fun to get a bug report a year later. You have a contributor, they send a PR, 04:53.360 --> 04:56.400 it has a bug, you merge it, you didn't spot the bug in a year later, you're going to bug 04:57.200 --> 05:01.440 the contributor who sent the PR is probably disappeared and they're doing something else, and 05:04.720 --> 05:09.360 it's all about time, right? What developers upstream want is quick feedback on the changes, 05:09.360 --> 05:14.960 and that's how we kind of have this idealized distro model, which at the moment is experience 05:14.960 --> 05:20.960 and friction has, you know, end users want to talk to developers directly and developers want 05:20.960 --> 05:25.360 to ship binaries to end users directly and the sort of traditional model that we've had for 30 years, 05:26.080 --> 05:34.480 has some friction. So being involved in, no, I put some thoughts into how we could 05:35.120 --> 05:40.240 move testing upstream, and the first thing we did was a few years ago, now back in 2021, 05:40.240 --> 05:45.680 and we set up an open QI instance, which is difficult for a few reasons. Firstly, 05:45.680 --> 05:52.560 known is, and has always been independent from any distro, so we have more than zero paid contributors 05:52.560 --> 05:58.960 from at least four companies, do I miss anyone? I always want to make the point that companies 05:58.960 --> 06:04.080 should be more loud when they pay people to work on upstream projects, because it's very important 06:04.080 --> 06:09.760 that companies do that, but we have more than zero paid contributors from each of these companies, 06:09.760 --> 06:16.720 and most of them are competitors, right? So we don't want necessarily to favor one over the other, 06:17.360 --> 06:23.680 and it's meant to be a neutral space. If we add end to end testing based on Ubuntu, 06:24.400 --> 06:28.400 and only Ubuntu, then it's kind of a slap in the face to red hat, and the same if we add end to 06:28.400 --> 06:34.560 end testing based on only Fedora. Also, we don't have much funding for infrastructure in 06:34.560 --> 06:41.680 home, and always images quite big. So the way we made this work is we don't pick anyone's distro, 06:42.400 --> 06:47.840 in the end we have our own distro called NomOS, which is a very simple, simple distro, 06:47.840 --> 06:52.880 which is intended at this point, just for testing. Everything in NomOS is built from Master, 06:54.160 --> 06:59.280 or Main, so it has all the latest books, breaks regularly, and that's the point, right? 06:59.280 --> 07:03.280 And breaks regularly, and then you go, oh, this broke, why did this break? So, 07:04.560 --> 07:07.600 like I said, I've talked about this before, so I won't go into it in too much detail, 07:07.600 --> 07:15.680 but it was working. We've got five test suites now. It's integrated to the CI for the 07:15.680 --> 07:21.520 NomBuild Meta repo, which is like the integration repo of Nom, so when someone releases Nom48, 07:21.520 --> 07:27.280 that exists as a tag in this repo that then points to all of the modules that make up Nom48. 07:27.280 --> 07:33.680 So that's where we run the tests, and you can go to OpenQA.com.org and see them. Here's a screenshot of 07:33.680 --> 07:44.240 one of the tests that we just test, does every app start. It's maintained by volunteers, 07:44.240 --> 07:50.560 so sometimes it works, sometimes it doesn't. And this is the list of contributors for the 07:50.560 --> 07:54.880 Needles repo, the screenshot's repo. You can see the first one is me. The second one is Jordan, 07:54.880 --> 08:00.880 who's one of the main people volunteering time on NomOS. The third person is me again, 08:01.040 --> 08:07.680 and the fourth person is Dorothy, who's one of the outreach interns. So this list isn't long enough. 08:08.720 --> 08:13.120 The QA initiative is still fairly invisible within Nom, and again, much of maintainers are 08:13.120 --> 08:20.320 most volunteers, then all of the works. So when I chat to them, they say, well, this is great. I really 08:20.320 --> 08:25.280 wanted to do this kind of testing, but I needed to be in my repo. I don't want the test to be 08:25.280 --> 08:32.240 in a separate repo, and the pipeline breaks after I've merged a PR. I want the testing to happen 08:33.040 --> 08:38.160 when somebody opens a merge request in my project, and I want the tests to be maintained as part of 08:38.160 --> 08:48.880 my project, which seems reasonable, right? So can we do it? Well, last year we worked on a prototype, 08:49.200 --> 08:57.760 and this was part funded by the STF, and partly by code think, and we worked on trying to do testing 08:57.760 --> 09:02.640 upstream for NomShell. So the reason we chose NomShell is because it's the pretty much the most 09:02.640 --> 09:08.880 difficult case. I previously did a proof of concept for a flat pack app, and that's fairly straightforward, 09:09.440 --> 09:15.440 but NomShell is the difficult case, right? It has multiple moving parts. It's actually split across 09:15.440 --> 09:22.480 multiple repose. It certainly can't be deployed as a flat pack. So it's hard mode, and if we can 09:23.040 --> 09:28.480 get a prototype of that, we can potentially apply it to every module in known that has a graphical 09:28.480 --> 09:32.720 component. So again, it's still quite difficult. We still don't have enough infrastructure, 09:33.280 --> 09:38.560 and if we try and build a new OS image for every commit to NomShell, then we're going to go crazy, 09:38.560 --> 09:43.520 right? So that's 4 gigabytes times, however many commits there are to NomShell every day. 09:44.480 --> 09:49.760 We also need the tests to be fast if they're going to be in upstream CI pipelines, and 09:51.200 --> 09:55.200 if we do it the sort of naive way, then they are not going to be fast, right? Building an image, 09:55.200 --> 10:01.680 you can't build a Linux desktop image in less than about 45 minutes, like IO just isn't fast enough, 10:02.480 --> 10:07.680 and then you have to download it onto the CI runner, boot, running the install that takes a while, 10:08.560 --> 10:13.360 booting machine takes a while. This is hopefully this will work. I'm going to show you a video 10:13.920 --> 10:18.400 of what it looks like to install NomOS in a VM. So this is what I test to doing. 10:22.480 --> 10:27.360 That's fed up. It actually takes about 3 minutes, but that's what's happening. 10:27.920 --> 10:33.360 And at this point, we could actually test the NomShell, right? But we've already spent almost an hour 10:33.360 --> 10:39.840 getting to that point. With a few tricks, we can make this a bit better, let's go to the next slide. 10:39.920 --> 10:45.520 Thank you. So with a few tricks, we can make this a bit better. Firstly, we can't make image building fast. 10:45.520 --> 10:51.680 So let's instead of building a fresh image, we already have a stock NomOS image, and we just 10:51.680 --> 10:58.160 need to overlay whatever's in the pull request. And systemD has a really cool way of doing this, 10:58.160 --> 11:02.720 called the systemD6, which is fairly new. Of course, it's great because it's 11:02.720 --> 11:08.800 just a neutral, you only have to adopt systemD, which most of us have. The way a system works is 11:08.880 --> 11:12.240 it's a little bit like a package except for none of the things we expect in a package, right? 11:12.240 --> 11:15.520 It's a bunch of binary files. It doesn't track its dependencies. 11:18.080 --> 11:23.200 It's no use for deploying to end users, but very useful for testing, because you get the binary, 11:23.200 --> 11:29.760 you overlay it on your router fast with an overlayFS in one command. Then let's say we're testing 11:29.760 --> 11:36.960 NomShell, so you just restart NomShell, it'll start the new version. And then you can remove it again 11:36.960 --> 11:42.720 when you want to shell it works. And you can also drop them into the file system, 11:43.840 --> 11:50.160 add or during early boot, so you can mount them as disks, use SMS and BIOS magic to have them mounted 11:50.160 --> 11:54.960 at early boot, and then the system just boots into an OS with whatever change you want overlaid. 11:54.960 --> 11:59.600 So this is really good, and it should work for almost anything apart from the kernel. 12:01.040 --> 12:05.920 That saves us most of the time. Downloading images, unfortunately, is an infrastructure problem, 12:05.920 --> 12:11.120 which we're not going to be able to solve without more money for infrastructure. So we still have 12:11.120 --> 12:16.400 a kind of 10-minute delay there. We don't actually have to run the installer, because we can build 12:16.400 --> 12:21.520 a pre-installed OS image when we build NomShell, so that saves a couple of minutes. And we don't actually 12:21.520 --> 12:28.000 have to pre-create a user, we can create a user on boot, we can do that via the SM BIOS. 12:28.000 --> 12:33.120 Or we kind of can. There's still a system D patch that needs completing before we can do that, 12:34.000 --> 12:41.520 the pieces are there. So we got this kind of working to a prototype standard. This is the first 12:41.520 --> 12:47.440 part of building a 6-ext on every commit. That landed in upstream NomShell, the second part, which 12:47.440 --> 12:52.400 is running the tests in NomOS, didn't yet land, and there's more work needed before it can land, 12:52.400 --> 12:58.000 but we made a proof of concept, and it proved the concept, so I'm quite happy. This is a screenshot 12:58.000 --> 13:03.280 of the real upstream NomShell CI pipeline, so this is what NomShell developers are looking at every 13:03.280 --> 13:13.280 time they review a merger request. And we can add in a test-to-sex job, which let me explain this 13:13.280 --> 13:19.280 diagram. I could have done a better job with the diagram, but I was in a hurry. The input to the 13:19.280 --> 13:26.560 build-to-sex job is a container image with the build tools you need to build NomShell. Now system 13:26.560 --> 13:31.760 D6-ex, a binary, so you have to think about ABI compatibility. You're not going to be able to 13:31.760 --> 13:37.920 download the C6-ex and run it on Debian 9 because it's built using a much more, you know, newer tool chain 13:37.920 --> 13:46.160 and NomOS. So what we do need to be careful of is that with the container we used build the binary 13:47.360 --> 13:53.760 is the same ABI as the OS we're going to test it against. But that's not a hard problem to solve, 13:54.320 --> 14:00.960 so we build a binary C6, it's available as an artifact from the GitLab CI, and then the test 14:00.960 --> 14:08.800 C6-ex job downloads the C6, downloads the disk image, overlays the two, and starts an open QA test. 14:08.800 --> 14:13.040 And then we can test the app in, you know, if we solve the download problem a couple of minutes. 14:14.560 --> 14:19.920 And the really nice thing about this is that once it works for NomOS, it can actually work for 14:19.920 --> 14:27.120 any OS, any system D based OS, could do the same, right? If you have a container with the build tools, 14:27.120 --> 14:32.720 you take a postmarker to OS container, you build a C6 that will run against postmarker to OS, 14:32.720 --> 14:40.080 and then you boot it against the postmarker to OS image. Once the system D support lands in postmarker 14:40.080 --> 14:45.920 to OS. So it works pretty well, I'm quite happy with the prototype. It's definitely not ready 14:45.920 --> 14:55.360 for prying time yet, and for a few reasons, partly technical reasons, partly infrastructure reasons, 14:55.360 --> 15:03.360 and partly stuff that requires a bit more thought, such as, I did an open QA workshop at 15:03.360 --> 15:07.360 Waddick, the known conference this year, and it was clear that it was the first time that most of the 15:07.360 --> 15:12.480 people in that room had, if it looked at Open QA. So there's a lot of work to do, both to make 15:12.480 --> 15:17.360 Open QA more accessible to people who aren't using it every day, and for people who want to do 15:17.360 --> 15:23.600 this kind of testing to learn about what tools we have and how they work. So where do we go from 15:23.600 --> 15:28.240 here? This is what I really wanted to talk about today. I hope you all with me so far. 15:29.840 --> 15:36.560 Part of what we need is money to pay people to do work. A lot of this stuff isn't very fun, right? 15:36.560 --> 15:43.680 So it'll be great to volunteer, step up and help, but in general it's not going to fit into 15:43.680 --> 15:49.040 some of the weekend stuff that requires a bit more engineering and probably will have to pay people 15:49.040 --> 15:54.160 to do it. The system de-patch may be quite easy, if anyone's bored and wants to look at 15:54.160 --> 16:02.800 a system de-patch, the link is in my slides. But the next problem I want to tackle is 16:02.880 --> 16:10.880 having a shared base of how to write these tests. Because if you think about it, you go back 16:11.440 --> 16:16.960 when I was looking at all the Open QA instances in the world. We have about 10, maybe 10 different 16:16.960 --> 16:22.560 Open QA test suites, like the suites, the fedora tests, the deviant tests, the Yamalynics tests. 16:24.080 --> 16:30.320 No one has about 200 modules, not all of them make sense to do graphical testing on, but if a 16:30.320 --> 16:35.520 100 of them have Open QA tests, we've now multiplied the number of test routes in the world by 10. 16:36.080 --> 16:41.600 So what works today in that we have quite a lot of copy-paste and duplication between the 16:41.600 --> 16:46.960 repos is not going to work when we have 100 such test routes. So we're going to need to work 16:46.960 --> 16:52.160 on collaborating a bit more and figuring out what stuff can we abstract. I'll go more into that 16:52.160 --> 16:57.520 in a bit. In fact, I'm going to go into all of this in a bit and it slides for each of these topics. 16:58.240 --> 17:04.960 So on the topic of the shared base, when you look at an Open QA test in the abstract, 17:04.960 --> 17:09.680 it's quite nice. The DSL is really straightforward and simple. But behind the scenes, 17:10.720 --> 17:17.440 there is quite a lot of pull with quite a lot of history in it. This is the code that I copied 17:17.440 --> 17:23.840 from the suites and to the numberless tests to wait for the machine to boot, which it does 17:23.840 --> 17:29.680 by checking the serial console to see if Shell Promptors appeared yet. And there's all kinds of, 17:30.640 --> 17:34.800 there's someone obviously spent a long time figuring out that they need to play all of the 17:34.800 --> 17:42.480 content. I don't know. Technical difficulties. 17:45.280 --> 17:50.080 Okay, I hope that still sounds okay. Now that's feeding back a bit. So somebody obviously 17:50.160 --> 18:01.920 spent a lot of time in energy trying to, hopefully they'll stay put. So a lot of blood 18:01.920 --> 18:08.240 sweatin' tears has gone into this. Ideally, we could share that and everyone who's doing Linux 18:08.240 --> 18:13.680 system-d-base testing can share the same code. Now we realize that's difficult because 18:14.240 --> 18:19.520 trying to re-base thousands of existing tests on a new base is not going to be straightforward. 18:19.520 --> 18:27.440 But at least we can start by saying, what should be the shared base that we build on? 18:28.320 --> 18:33.600 The current base test class in OS Auto-Inst, which is the test run or an open QA, 18:34.160 --> 18:38.240 is completely generic. So you could use it to test Windows as much as you could use it to test 18:38.240 --> 18:44.160 Linux. But that means you need all the support for testing. You need all this custom code to 18:44.160 --> 18:49.280 wait for the boot and every Linux system has nine virtual frame buffer consoles, right? So you 18:49.280 --> 18:57.280 have to define each of those that you want to use. We could share all that stuff. And hopefully, we will. 19:00.480 --> 19:05.600 I've been in talks with Santiago Zalati in particular who works at Susa, who's quite interested in 19:06.240 --> 19:11.840 building this common base. A spoiler for the end of the talk, we have a monthly call in which we 19:11.920 --> 19:18.400 try to work through this stuff, which I'm going to invite you all to at the end, but that's my spoiler. 19:19.440 --> 19:24.400 In terms of infrastructure, there's everyone heard about the free desktop.org problem. 19:25.520 --> 19:31.040 So free desktop.org has a fairly major issue, which is one of the infrastructure sponsors, 19:31.040 --> 19:36.320 who are generously donating infrastructure, have to stop doing that. And it's going to cause 19:36.320 --> 19:41.360 some pretty major disruption, because where do you get a whole load of new infrastructure from 19:41.360 --> 19:46.720 with three months notice? This is the reality of doing testing and open source projects, right? 19:47.280 --> 19:52.000 It's quite hard to get an ongoing stream of funding and an ongoing stream of 19:52.000 --> 19:56.800 paid people who maintain infrastructure. Again, volunteers, thismadmins are quite thin on the ground, 19:58.000 --> 20:03.040 quite unreliable because they have lives and people need to be paid to do this work if we're going to 20:03.040 --> 20:08.640 have professional quality development. In no, we also have a very small infantry team. We're very 20:08.640 --> 20:13.520 grateful to Red Hat for sponsoring half of the person to maintain the infrastructure and the 20:13.520 --> 20:18.400 non-foundation pays for the other person. And we have a few hard responses. Most of the 20:18.400 --> 20:24.400 infrastructures currently moving to AWS based on free open source project credits from AWS. So again, 20:24.400 --> 20:31.680 that's very nice. But for QA testing, we're going to have more requirements. And we're going to need 20:31.680 --> 20:41.200 someone to contribute more hardware so that we can run more tests. Also, my dream would be a hardware 20:41.200 --> 20:46.640 test lab, so we can actually test, you know, on real laptops and on devices. This is something 20:46.640 --> 20:53.360 a known cell team really wants. But again, we need to figure out how to do that and where to host the hardware. 20:53.520 --> 21:01.920 With more capacity, we could actually build more stuff as well, so we could conceivably start 21:02.720 --> 21:07.840 providing stuff with different distros. Another thing I'm really interested to talk about is 21:07.840 --> 21:13.840 making open QA more accessible for newcomers. So the DSL that you write testing is excellent, right? 21:13.840 --> 21:18.240 It's very straightforward. Here's an example. If you want to send a key to the virtual 21:18.320 --> 21:24.960 to the machine on the test, how do you do it? So you write send key and then you put the name of the key 21:26.000 --> 21:30.960 and that's everything. You need to put a semicolon after. It's not too difficult, right? It's really 21:30.960 --> 21:36.160 really straightforward to explain this stuff. You need four or five commands to do 90% of the testing. 21:37.840 --> 21:43.840 But when you look at the real tests today, you realize this isn't quite a DSL because you can do anything 21:43.920 --> 21:48.480 you can do in Pearl, and so of course people do do anything that you can do in Pearl, such as 21:48.480 --> 21:54.720 importing 42,000 lines of help a code to do different things that isn't part of the test API. 21:57.040 --> 22:01.040 I can see why things have got to this point, but to make things more accessible, 22:02.160 --> 22:07.120 we really need to start breaking this up into libraries and sharing and documenting the libraries better. 22:07.120 --> 22:15.440 So another fun project. I also personally find that the developer experience for tests is quite 22:15.440 --> 22:23.120 hard. I didn't personally like running a web UI on my laptop to generate tests. I wrote my 22:23.120 --> 22:29.600 own command line tool that connects to ISO to video directly and lets me run tests from the command line. 22:30.720 --> 22:35.520 But that's my tool. I think it does more work before it's available and useful to everyone. 22:38.080 --> 22:43.760 So going even further into the future, right? If we get to a point where the tests are 22:43.760 --> 22:50.640 using a real DSL and they're fully encapsulated and they don't depend on stuff that's specific 22:50.640 --> 22:56.240 to a live directory that's only available for the open-source of tests, we can actually share them. 22:57.760 --> 23:03.680 So imagine the dream, right? The no file manager, Nautilus, provides a test rate that says, 23:03.840 --> 23:09.200 here are tests for all the core functionality, how to connect a network file system, 23:09.200 --> 23:17.040 how to search for a file. Those tests can run in its CI. Those tests, the same tests and run against 23:17.040 --> 23:25.280 normal S. And then downstreams can also use the same tests. Of course there might be some 23:25.280 --> 23:29.200 need to change the needles, right? If you patch, if you change the theme, then you'll need to update 23:29.200 --> 23:33.520 the needles, which is expected, right? And it also gives you a way to test that your theme is working, 23:34.240 --> 23:40.560 but this, this is a distant dream. I don't think we're going to get there this year, 23:40.560 --> 23:46.480 but I think we can, if we think about it now, we can set ourselves off on the right path 23:47.200 --> 23:52.560 to at least not block our way from getting there. And who knows, maybe in 10 years, we've 23:52.560 --> 23:57.600 all be there. Some of this stuff, some of this stuff is fairly simple technical problems, and some 23:57.600 --> 24:01.760 of this is what I call the hardest problem in software development, which is getting people to 24:01.840 --> 24:07.760 agree on things and talk to each other. So my contribution in this space is organizing a monthly 24:07.760 --> 24:14.400 call where it's a vendor neutral space that people who work on this stuff, upstream and downstream 24:14.400 --> 24:19.120 can meet and talk about and to end testing. It's kind of begun with a focus on no, 24:19.120 --> 24:24.800 I'm going to focus on open QA, but every test tool is welcome and every, every desktop is welcome. 24:25.760 --> 24:31.440 We meet once a month, because the world is quite big, there are 24 times on, so we meet 24:31.440 --> 24:37.040 one month it's European morning, Asian evening, and then one month it's US morning, and 24:37.040 --> 24:42.160 European evening, the details are on this, get lab repo, get a hub repo. I think you have to 24:42.160 --> 24:46.320 look in the poll requests at the moment. I'm hoping to merge the poll requests that has the info, 24:46.960 --> 24:54.480 but you're all welcome to join. The next call is this Thursday at 1600 to UTC, 24:55.200 --> 24:59.600 and I hope to see you there. And that's the end. Thank you very much. 25:06.880 --> 25:11.520 So we have five minutes for questions or a bit less, any questions? 25:16.400 --> 25:23.680 All right. Thanks for the talk. Just wondering, how are you integrating with the 25:23.680 --> 25:29.920 git forges? I've seen you are kind of tied to the, like, git lobby, I'm going to end this stuff, 25:29.920 --> 25:33.840 or how we can, for example, move it like to GitHub and other places. 25:33.840 --> 25:40.240 Right, that's something that could be improved in an open QA today. The design of open QA 25:40.240 --> 25:44.400 initially is that you have static workers maintained by an infrastructure team, right? So you 25:44.880 --> 25:48.800 have, Susan has, you know, I don't know how many dedicated machines that run tests. 25:49.520 --> 25:53.920 In no one, we don't have that much infrastructure. We don't have any more capacity to spin up, 25:53.920 --> 25:58.880 you know, more machines. So we wanted it to run on git lab runners, because we really have 25:58.880 --> 26:04.240 git lab runners. The way we do this is a bit of a hack, right? So each git lab runner 26:05.680 --> 26:11.920 boots, well, starts a container image. So we use the open QA worker container image as our container 26:12.000 --> 26:19.280 image in the git lab test. That runs a short cell script, which first registers with the 26:19.280 --> 26:26.800 open QA web UI as a worker. So it says, hi, I'm a worker. Then it submits the job and the job 26:28.000 --> 26:32.160 has a special, I can't remember how we do it, but we have a special parameter so that it goes to that 26:32.720 --> 26:37.760 runner. Runs the test sends the results back to the web API and then destroys itself. 26:38.720 --> 26:41.200 So, it's a bit of a hack, but it does actually work. 26:47.200 --> 26:51.920 Any other questions? Of course, you're welcome to join the monthly call and ask questions there. 27:00.080 --> 27:04.400 Well, I hope I've just tried to put the testing up food. Thanks a lot.