WEBVTT 00:00.000 --> 00:11.000 I'm supposed to present this talk together with a cash deep, but he cannot be here, so it's 00:11.000 --> 00:12.000 only me. 00:12.000 --> 00:14.000 And let's start then. 00:14.000 --> 00:15.000 So, who are you? 00:15.000 --> 00:16.000 Who are we people? 00:16.000 --> 00:18.000 I'm part of the team, which is called Continental Engineering. 00:18.000 --> 00:23.000 We are the team, which is responsible for infrastructure services, design, documentation, 00:23.000 --> 00:29.000 and leadership of the project, and send those and people and some other things. 00:29.000 --> 00:30.000 So, let's about it. 00:30.000 --> 00:35.000 I'm part of the team, and how do we choose for you? 00:35.000 --> 00:41.000 How do we get into the situation where we are going to switch to something new? 00:41.000 --> 00:50.000 So, as a project, for a few years, we understood that our current solution is not really 00:50.000 --> 00:51.000 maintained. 00:51.000 --> 00:56.000 We don't have enough resources to maintain it, and be developing yet another open source project 00:56.000 --> 00:59.000 is not necessarily a way to go in the open source world. 00:59.000 --> 01:07.000 So, our team was approached by Federal Council, and we were given a mandate to compare 01:07.000 --> 01:13.000 some open source forges, and choose or not really choose, we just created an analysis, 01:13.000 --> 01:18.000 and submitted it to the Federal Council body to decide what will happen. 01:18.000 --> 01:25.000 So, the solutions we compared were GitHub and for Joe. 01:26.000 --> 01:33.000 One thing that Council decided was that we are going to do it for Joe even though at that 01:33.000 --> 01:37.000 moment, for Joe it didn't have all the features that we need. 01:37.000 --> 01:43.000 It still doesn't have them, but it's way better for us as a project to be part of something 01:43.000 --> 01:50.000 open, and something which is very well alive, and it's being developed by somebody else 01:50.000 --> 01:51.000 who can help us. 01:51.000 --> 01:57.000 So, we can contribute, we can help the project grow, but we are not the one who is owning the code 01:57.000 --> 01:58.000 this itself. 01:58.000 --> 02:05.000 That was one of the reasons why this is in fail into the for Joe even though we were approached 02:05.000 --> 02:12.000 by our sponsors, and we were offered to use GitHub Enterprise Edition, but this is 02:12.000 --> 02:16.000 not really in a philosophy, etc. 02:16.000 --> 02:21.000 Using proprietary solutions to build open source project is kind of strange. 02:21.000 --> 02:26.000 So, we ended up with for Joe, and where are we now? 02:26.000 --> 02:32.000 It's been about seven months, when we are eight months, when the Council decided what 02:32.000 --> 02:34.000 are we switching to. 02:34.000 --> 02:41.000 So, currently we deployed for Joe in one deployment, which we call the forge. 02:41.000 --> 02:47.000 So, instead of how we do infrastructure, how we do things is that we have separate staging 02:47.000 --> 02:52.000 and production environments, where the staging is trying to be like one or one to one with 02:52.000 --> 02:55.000 the production stuff, so all the services are running there. 02:55.000 --> 03:03.000 We have a production clusters, which are there to provide the services alive in production. 03:03.000 --> 03:10.000 As a first, we deployed in staging, and started playing with it, we used different strategies 03:10.000 --> 03:11.000 to deploy for Joe. 03:11.000 --> 03:17.000 But eventually we ended up using upstream helm, and updating is a little bit 03:17.000 --> 03:22.000 because a bunch of the images that we were used are not really, we didn't really like 03:22.000 --> 03:31.000 them, so we changed them, updated some, modified the helm a little bit, and deployed it. 03:31.000 --> 03:38.000 In the way we were deploying it, and we started experimenting it, we needed to find 03:38.000 --> 03:42.000 the way how do we import all the work that we have already impagure. 03:42.000 --> 03:46.000 Happily, for Joe is absolutely amazing project because it has the structure for 03:46.000 --> 03:50.000 importing other repositories in it, so we just needed to contribute the structure of 03:50.000 --> 03:56.000 pagure, the data structures that are used there, and how it will be imported into 03:56.000 --> 03:58.000 the forge. 03:58.000 --> 04:06.000 We did that, as we did, we introduced some bugs, which we needed to fix, and as we were fixing 04:06.000 --> 04:11.000 it, we discovered smaller bugs, which were not introduced, but yes, so we started fixing 04:11.000 --> 04:17.000 them slowly, but surely, and eventually we came to a state where the forge was up and running, 04:17.000 --> 04:22.000 and we needed to provide our users with some additional functionality, because having 04:22.000 --> 04:27.000 just the same thing that you had previously, is not really moving anywhere, right? 04:27.000 --> 04:35.000 So we decided that we will enable the users to, the ability to ram the ground jobs, and 04:35.000 --> 04:39.000 even though at that time, for you actually didn't really recommend it for your 04:39.000 --> 04:44.000 actions, we decided to go the way, because we were not looking for a CI 04:44.000 --> 04:49.000 runner, we were looking just for automation tooling, we allow for the 04:49.000 --> 04:53.000 contributors and developers to automate some of the work tools that they have. 04:53.000 --> 04:57.000 So we deployed it, we deployed it as a virtual machine on the top of the 04:57.000 --> 05:04.000 hardware, where it's running a rootless podmem, and each organization in the forge has 05:04.000 --> 05:10.000 its own runner, so eventually if somebody starts behaving, we can just kill their 05:10.000 --> 05:12.000 containers, and we're good. 05:12.000 --> 05:17.000 The other things that we need to do was integrating for Joe into our 05:17.000 --> 05:22.000 our infrastructure, and that is one of the key features that we have in 05:22.000 --> 05:26.000 the infrastructure is for the messaging, which is a queue that all services use and 05:26.000 --> 05:28.000 just post messages there. 05:28.000 --> 05:32.000 For now, the, which is the simplest way that we can, that's for Joe's 05:32.000 --> 05:36.000 supports, and this is web hooks, so we use web hooks to just push messages 05:36.000 --> 05:42.000 to our queue to report the rest of the stuff that we have in there. 05:42.000 --> 05:48.000 So, this is the deployment that we call the forge. 05:48.000 --> 05:55.000 This is a place where people across teams in Fedora do things, but not 05:55.000 --> 05:57.000 necessarily the content for the distribution. 05:57.000 --> 06:01.000 The content for the distribution services, we call this getting Fedora. 06:01.000 --> 06:05.000 It's, it's a bit complicated and convoluted term because it means a lot of 06:05.000 --> 06:06.000 things to a lot of people. 06:06.000 --> 06:09.000 Let's just say, is the, is the place where the distribution 06:09.000 --> 06:10.000 sources list. 06:10.000 --> 06:19.000 We haven't yet deployed production early deployment here, because the use 06:19.000 --> 06:21.000 case is a little bit different. 06:21.000 --> 06:24.000 It's not just about having given repositories. 06:24.000 --> 06:32.000 We need our, our users and group permissions properly mapped into 06:32.000 --> 06:33.000 things. 06:33.000 --> 06:36.000 So, we currently are in a very early stage. 06:36.000 --> 06:37.000 We have a staging deployment. 06:37.000 --> 06:40.000 We did some changing into it. 06:40.000 --> 06:44.000 Again, we're using helm from upstream with a little bit of modification. 06:44.000 --> 06:50.000 And now, we are in a stage where we imported about 40k of repositories into 06:50.000 --> 06:54.000 it, and are experimenting with playing with, for Joe's, seeing how the 06:54.000 --> 06:55.000 application reacts. 06:55.000 --> 07:05.000 If, if it is ready to sustain the load that Fedora contributes, can make on it. 07:05.000 --> 07:09.000 So, how do we do that, how do we build the things that we deploy. 07:09.000 --> 07:15.000 So, we decided to use build system called conflux to build our container images that 07:15.000 --> 07:16.000 we then deploy. 07:16.000 --> 07:22.000 Since conflux does not really support anything else other than GitLab and GitHub. 07:22.000 --> 07:23.000 It happened. 07:23.000 --> 07:27.000 We had to do it a little bit, weirdly. 07:27.000 --> 07:31.000 So, what we are doing is that we have our fork of for Joe, living in 07:31.000 --> 07:35.000 codebarc, which is a service providing for Joe. 07:35.000 --> 07:39.000 Where we do the development, we create new features. 07:39.000 --> 07:44.000 If the features are acceptable to upstream, we will open a pull request and try to get 07:44.000 --> 07:45.000 it and merge upstream. 07:45.000 --> 07:49.000 If it's not, we are good with carrying on our own patches. 07:49.000 --> 07:53.000 If the functionality is no good for upstream, why to submit it. 07:53.000 --> 07:55.000 And this is basically the workflow. 07:55.000 --> 07:58.000 We push things to codebarc. 07:58.000 --> 08:01.000 They have a mirror, which mirror things to GitHub. 08:01.000 --> 08:07.000 And from GitHub, build actions are triggered and conflux builds our images. 08:07.000 --> 08:09.000 Push is done to our edge states. 08:09.000 --> 08:12.000 Then we pull the images and deploy them in our deployment. 08:12.000 --> 08:16.000 It's a bit complicated, but we work with what we have. 08:16.000 --> 08:21.000 And try to make it better. 08:21.000 --> 08:25.000 And a little bit of working with upstream, I would like to say that for Joe 08:25.000 --> 08:28.000 upstream is absolutely amazing. 08:28.000 --> 08:34.000 And we didn't expect that by your importer, which is very specific for 08:34.000 --> 08:38.000 there are some other deployments in the world, but there is not much of them. 08:38.000 --> 08:41.000 We will be merged upstream, but it got merged upstream. 08:41.000 --> 08:46.000 And as I mentioned, we introduced a bunch of bugs, which we needed to fix. 08:46.000 --> 08:47.000 So we fixed them. 08:47.000 --> 08:51.000 And as we are fixing them, the team was able to, I think we got a 08:51.000 --> 08:55.000 much like four or five pull requests across two releases. 08:55.000 --> 08:59.000 It was V13 and V14, which is their nice. 08:59.000 --> 09:07.000 Now, and again, working with for Joe upstream is absolutely amazing. 09:07.000 --> 09:13.000 People are very nice and happy to help solve our problems. 09:13.000 --> 09:17.000 So we got some guidance on changes that we would like to contribute. 09:17.000 --> 09:18.000 I would get to the later. 09:18.000 --> 09:27.000 But we are not sure how to approach it, because we do not own the cold base. 09:27.000 --> 09:36.000 So we currently, we are in a space where we moved. 09:36.000 --> 09:40.000 Our team is mostly working from the new forge. 09:40.000 --> 09:45.000 There's a few repositories that are still missing that we are working on moving 09:45.000 --> 09:49.000 them to the new place. 09:49.000 --> 09:53.000 And the idea is, in the next six weeks, to have all of the 09:53.000 --> 09:58.000 commenting and engineering teams working from the new forge. 09:58.000 --> 10:03.000 And in Idol World, at the Flock Conference, that Flock is for 10:03.000 --> 10:04.000 their contributors conference. 10:04.000 --> 10:06.000 You can scan the QR code and join us. 10:06.000 --> 10:09.000 And join us on the conference. 10:09.000 --> 10:14.000 We would like to see all the community teams working from the new forge. 10:14.000 --> 10:20.000 Again, this does mean just the teams that are not working on the content. 10:20.000 --> 10:25.000 The content of the distribution itself is elsewhere. 10:25.000 --> 10:28.000 So what the future holds for us? 10:28.000 --> 10:33.000 And we would like to do more upstream fixes. 10:33.000 --> 10:36.000 As you know, open source projects have their issues. 10:36.000 --> 10:39.000 In this case, it's about 1,100 of them. 10:39.000 --> 10:45.000 So about the community is nice about taking their issues. 10:45.000 --> 10:49.000 So we can find about 70 good first issues. 10:49.000 --> 10:53.000 If you never contribute, if you never contribute to jail. 10:53.000 --> 10:56.000 And you know how to write the goal and code. 10:56.000 --> 10:59.000 I would suggest if you want to contribute to something. 10:59.000 --> 11:02.000 I would suggest to go scan the QR code, go through the tracker. 11:02.000 --> 11:06.000 Find issues that are marked as good first issue. 11:06.000 --> 11:11.000 They are very nicely described and usually easy to work on. 11:11.000 --> 11:18.000 So please help us make for jail better for us and for everybody else. 11:18.000 --> 11:25.000 So about the new features that we would like to see in for jail that we are missing for our workflows. 11:25.000 --> 11:29.000 The biggest one we decided to call it by issues in public repositories. 11:30.000 --> 11:37.000 As it says, it's about being able to mark some issues as a private. 11:37.000 --> 11:46.000 Or being able to move private issues from private to public stage as a disclosure, for example, of some secret issues. 11:46.000 --> 11:50.000 This is the biggest feature that you are working on right now. 11:50.000 --> 11:55.000 We got some nice guidance from upstream on how to make this change. 11:55.000 --> 11:59.000 Not disruptive for them and for other people. 11:59.000 --> 12:07.000 So hopefully we will be able to contribute this new feature in upcoming weeks or months. 12:07.000 --> 12:12.000 The other features that we have noticed are not only coming from us. 12:12.000 --> 12:17.000 With our there are also in the tracker for jail. 12:18.000 --> 12:23.000 Nested organizations and projects which means a set is just about the structure. 12:23.000 --> 12:27.000 But structuring your content inside the for jail instance. 12:27.000 --> 12:34.000 Ticket transitions for jail allows us to use some rudimentary planning and combat boards. 12:34.000 --> 12:39.000 The problem is that the combat boards are disconnected from the issues themselves. 12:39.000 --> 12:43.000 So having a nice transitions where you move things around the combat board. 12:43.000 --> 12:47.000 Being able to transition the issues automatically will be really nice. 12:47.000 --> 12:53.000 And pretty recently as we were migrating our own repositories. 12:53.000 --> 13:03.000 We discovered that adding the ability to archive some tags that you use to tag an issue or poor request. 13:03.000 --> 13:07.000 Meaning that you will the tag will look be useful anymore. 13:07.000 --> 13:13.000 But it will still be visible to users is a feature that we would really like to have. 13:13.000 --> 13:23.000 So this is the set of new new features that we will focus on upcoming weeks or months to and try to contribute an upstream. 13:23.000 --> 13:27.000 Again, if the features are not acceptable for upstream. 13:27.000 --> 13:31.000 We can carry them on and they can live in our fork. 13:31.000 --> 13:35.000 And eventually maybe sometime get merged or not. 13:35.000 --> 13:39.000 So why are we doing this? 13:39.000 --> 13:41.000 What is the end goal? 13:41.000 --> 13:45.000 So the end goal is to being able to modernize the workflow a little bit. 13:45.000 --> 13:52.000 Having integrated it into the new services that further infrastructure and this project has at their disposal. 13:52.000 --> 13:57.000 And that's like the packet service which is therefore to run automatically CI on their packages. 13:57.000 --> 14:01.000 Or conflicts as I mentioned as a build service that will allow you to build containers. 14:01.000 --> 14:04.000 Eventually maybe one time RPMs. 14:04.000 --> 14:06.000 But currently it can be build containers. 14:06.000 --> 14:18.000 So having the ability to build containers directly from further approach is one of the big things that we are focusing on. 14:18.000 --> 14:26.000 So as I said, we are not really yet ready to provide distribution content from for Joe. 14:26.000 --> 14:30.000 So this is basically kind of roadmap where we are heading. 14:30.000 --> 14:34.000 As I said, we already have a staging instance for this get. 14:34.000 --> 14:40.000 We would like to stabilize the configuration, have it already by the end of March. 14:40.000 --> 14:44.000 Implanant the group and individual package repository access. 14:44.000 --> 14:46.000 And may this year. 14:46.000 --> 14:56.000 And ideally during summer and autonomous start to working on upgrading our tooling in federal project. 14:56.000 --> 14:58.000 We actually can move somewhere. 14:58.000 --> 15:05.000 And hopefully if everything goes as expected after the release of federal 45. 15:05.000 --> 15:10.000 If we get our changes approved, we will submit changes to federal to do that. 15:10.000 --> 15:17.000 We would like to start using the new for the for this for distributing the services and building the packages in federal. 15:17.000 --> 15:22.000 And I would like to thank. 15:22.000 --> 15:28.000 But they would like to thank the opportunity and thank the forger upstream which folks are very friendly. 15:28.000 --> 15:32.000 And there is so many contributors they didn't fit in the screenshot. 15:32.000 --> 15:34.000 And sorry. 15:34.000 --> 15:36.000 I, you are on the there. 15:36.000 --> 15:42.000 And I would like to thank Dean that is implementing the forge in federal. 15:42.000 --> 15:45.000 Except for me, I don't think anybody made it. 15:45.000 --> 15:48.000 For them this year, but hopefully in there. 15:48.000 --> 15:55.000 Well, we will meet here and be able to discuss future of the forge for federal. 15:55.000 --> 16:01.000 So with that, I would like to ask you if you have any questions. 16:01.000 --> 16:05.000 Can you come and you probably must be there. 16:05.000 --> 16:10.000 We use the way we use the way you are. 16:10.000 --> 16:21.000 And the question the question was if we had some to link to use or some doing it and mess or we are using the UI to import our repositories. 16:21.000 --> 16:28.000 So basically we use the UI, but we scripted away on the UI to be automated. 16:28.000 --> 16:33.000 So when you want to import tens of thousands of packages, you don't have to. 16:33.000 --> 16:37.000 Click on the UI 10,000 times. 16:37.000 --> 16:38.000 Yeah. 16:38.000 --> 16:43.000 Your slides show all the massive use of creation as a thing. 16:43.000 --> 16:44.000 Wouldn't be upstream. 16:44.000 --> 16:46.000 What do you think the door doing? 16:46.000 --> 16:47.000 Whatever. 16:47.000 --> 16:49.000 Can you continue? 16:49.000 --> 16:55.000 You listed all the massive use of creation as an example of a patch that couldn't be upstream. 16:55.000 --> 16:56.000 Isn't that why? 16:56.000 --> 16:58.000 Automatically. 16:58.000 --> 16:59.000 Did they mention that? 17:00.000 --> 17:14.000 I don't think so. 17:14.000 --> 17:25.000 This is why we have a diagram for the box to use each other. 17:25.000 --> 17:32.000 Now I'm lost because that is the slide that was made by Akash Deep not me. 17:32.000 --> 17:38.000 I understand the process and I know how it worked, but I don't know why he chose automated user creation. 17:38.000 --> 17:47.000 Because that is not really the thing that we do because for Joe. 17:47.000 --> 17:54.000 The question was why did we put the automated user creation as a feature that cannot be upstream? 17:54.000 --> 18:02.000 But I think that's not really a good example because for Joe's supports or IDC. 18:02.000 --> 18:05.000 So the account system is plugged into for Joe. 18:05.000 --> 18:11.000 So the moment you first try to log in, you will be redirected to our provider. 18:11.000 --> 18:13.000 And then your user is created in the forge. 18:13.000 --> 18:18.000 And you have a local user which is mapped to the account system. 18:18.000 --> 18:30.000 And then do you foresee a kind of needs in the Fedora project to do more project management type of tasks in the day or night, 18:30.000 --> 18:34.000 epic features and users from each this time? 18:34.000 --> 18:42.000 So the question is if we see the need for Fedora to do more project management style things in the forge. 18:42.000 --> 18:45.000 And the answer is yes. 18:46.000 --> 18:53.000 We are not necessarily looking for something robust, but something a little bit more than we have now. 18:53.000 --> 18:58.000 As I mentioned, the automatic issue transitions, for example, if you have the combat board. 18:58.000 --> 19:09.000 Things like this, like the small quality of life features, more than some huge, because we are looking at the simplicity of the project management. 19:09.000 --> 19:12.000 Because it isn't for Joe, it's something that we want to have. 19:12.000 --> 19:22.000 Like the simple clean user interface to work with, not some bloated things like giraffe or something similar. 19:22.000 --> 19:24.000 I think there was a question. 19:24.000 --> 19:32.000 Do you think that eventually a red cartoon of switch to this one? 19:32.000 --> 19:38.000 So the question is if I can foresee if Fedora will eventually switch? 19:38.000 --> 19:52.000 No, but I don't see red head switching to for Joe at any time soon, but I have had a chance to talk to some package maintainers and red head. 19:52.000 --> 19:59.000 And they are not necessarily completely happy with the current solution, but it's the solution that the company is using. 19:59.000 --> 20:08.000 So this is that it may be if you are able to prove in Fedora that our solutions is better and easier for developers. 20:08.000 --> 20:11.000 Maybe, but most probably not. 20:11.000 --> 20:20.000 Do you see a world in which for Joe is that actually tracking replaces Paxela for Fedora? 20:20.000 --> 20:33.000 Yes, so question is if for Joe issue tracking will replace Baxela for Fedora and the answer is yes, but we are missing the crucial features. 20:33.000 --> 20:37.000 One of the features that you are missing which is very important is the private issues. 20:37.000 --> 20:48.000 Because when you get a bug report on your package and the bug report should be confidential, we can't just make everything public now. 20:48.000 --> 21:00.000 So eventually at the end of this year I hope we will have feature party that will allow us to provide developers with similar feature set. 21:00.000 --> 21:08.000 Not the same feature set because we don't want to implement Baxela in for Joe, but yes. 21:08.000 --> 21:11.000 The question should be sure that I'm correct. 21:11.000 --> 21:17.000 You mentioned that the project should be migrated before, let's say the target is blocked so June. 21:17.000 --> 21:22.000 Does it mean that there would be a sunset and the commission will have to be loaded and it's just that. 21:22.000 --> 21:23.000 Very good question. 21:23.000 --> 21:26.000 So yes and no. 21:26.000 --> 21:34.000 We would like to, if it wasn't me, I would like to the commission, but you're tomorrow, because there are issues with the code base, we are not able to fix. 21:34.000 --> 21:41.000 But we are not going to the commission, but you're in a way that we will turn it off. 21:41.000 --> 21:43.000 There's not what's going to happen. 21:43.000 --> 21:49.000 We'll keep bugger running in redone mode for some time. 21:49.000 --> 21:55.000 I'm not sure for how long and when we decide this is the end of the bugger. 21:55.000 --> 22:01.000 What we will do is to generate static HTML pages from all the content that's currently on bugger. 22:01.000 --> 22:05.000 And those static HTML pages will live there. 22:05.000 --> 22:13.000 So people will be able to see what was there previously. 22:13.000 --> 22:26.000 Do we have any questions about the local patches that you have for that different from the upstream? 22:27.000 --> 22:33.000 Will be they just different too much in time, so let's say in one. 22:33.000 --> 22:37.000 We hope you won't be able to apply them without rewriting. 22:37.000 --> 22:45.000 So the question is about the divergence and about our patch set that we are carrying over. 22:45.000 --> 22:54.000 So as I mentioned, I was really surprised that for you merged our impugger importer, because that's not necessarily a feature good for everybody. 22:54.000 --> 23:04.000 And it seems from the attitude that for Joe upstream has that if we are able to provide quality features that we discuss with them first, 23:04.000 --> 23:10.000 and we agree on the implementation that maybe we got them merged. 23:10.000 --> 23:13.000 Currently, we are not carrying over that many patches. 23:13.000 --> 23:19.000 It's mostly about some minor UI fixes that we not necessarily want to push upstream. 23:19.000 --> 23:23.000 It's about the teaming of the fortunately, how it looks like. 23:23.000 --> 23:37.000 And things like that, we are trying to carry over minimal patch set with minimal functionality and minimal code base. 23:37.000 --> 23:44.000 But what other options did you explore and why did you choose? 23:45.000 --> 23:49.000 So the question was what other options we explored and why for Joe? 23:49.000 --> 23:54.000 As I said, we explored mainly GitLab. 23:54.000 --> 24:00.000 And we tried to look into different projects. 24:00.000 --> 24:03.000 We also looked at Gitty, for example. 24:03.000 --> 24:20.000 But philosophically, we as a federal project are closer to for Joe as we are to, I would say, almost any other Bitforce project that is around. 24:20.000 --> 24:23.000 Time's up, so thank you very much. 24:33.000 --> 24:35.000 Thank you very much.