WEBVTT

00:00.000 --> 00:07.000
Okay, good morning everyone.

00:07.000 --> 00:09.000
Hope you're all feeling clear headed.

00:09.000 --> 00:12.000
This is my first talk publicly, so be kind.

00:12.000 --> 00:13.000
Thank you.

00:13.000 --> 00:16.000
APPLAUSE

00:16.000 --> 00:20.000
So we're here for a flat car and Gen 2 sitting in a tree.

00:20.000 --> 00:23.000
It's about a collaboration of distributions.

00:23.000 --> 00:26.000
And my button doesn't work, typical.

00:27.000 --> 00:31.000
So yeah, I'm James McRow, better known around here as Chewy.

00:31.000 --> 00:37.000
I'm a flat car maintainer by day, working at Microsoft,

00:37.000 --> 00:42.000
just under a year now, and I have to say being paid to do this is awesome.

00:42.000 --> 00:45.000
So I've not been in the job that long,

00:45.000 --> 00:51.000
but I wasn't starving from scratch because I'm also a Gen 2 developer by night.

00:52.000 --> 00:58.000
Officially since 2015, but I've been contributing since 2004,

00:58.000 --> 01:01.000
and you might think, well, why did I wait so long?

01:01.000 --> 01:06.000
And I know I've just started to talk, but I just wanted to make a really important point here,

01:06.000 --> 01:11.000
that I was really worried that I wouldn't be able to contribute enough,

01:11.000 --> 01:13.000
and I'd get kicked out.

01:13.000 --> 01:16.000
And I hear this all the time, even just this weekend,

01:16.000 --> 01:21.000
and I want to tell you, even if you're just maintaining one package,

01:21.000 --> 01:24.000
and committing once a month, that's great.

01:24.000 --> 01:25.000
That's fantastic.

01:25.000 --> 01:29.000
I'd rather have hundreds of people doing that than like 10 devs

01:29.000 --> 01:32.000
breaking their backs because, you know, burnouts real.

01:32.000 --> 01:37.000
So yeah, I've come down from Edinburgh in Scotland.

01:37.000 --> 01:40.000
If you think I don't sound Scottish, you'd be right,

01:40.000 --> 01:42.000
but I have lived there most of my life.

01:43.000 --> 01:49.000
And I'm known to do silly things like run Gen 2 on the Pomodoro Amiga from 1993.

01:49.000 --> 01:53.000
You might have seen that yesterday, I hope to have it running again later today,

01:53.000 --> 01:55.000
if you want to come by.

01:57.000 --> 02:00.000
So, I didn't want to assume that you knew both distros,

02:00.000 --> 02:03.000
or even one distro, so let's do a quick intro.

02:03.000 --> 02:07.000
On the left, we got Gen 2, created around 2000.

02:07.000 --> 02:11.000
There's some developers who do this as their day job,

02:11.000 --> 02:16.000
as a community distro, and legal aspects are handled by the Gen 2 foundation.

02:16.000 --> 02:18.000
It's a source-based distro.

02:18.000 --> 02:22.000
There aren't many of those around still, but even on that.

02:22.000 --> 02:27.000
And it's not derived from any other distros, it's original.

02:27.000 --> 02:30.000
It's general purpose and extremely flexible,

02:30.000 --> 02:35.000
so you can use it on massive servers or tiny little arm boards

02:35.000 --> 02:40.000
that I have on my arm board at home in the corner of the living room.

02:40.000 --> 02:45.000
And it's sort of famous flash infamous for its compile flags,

02:45.000 --> 02:47.000
being able to optimize the hardware,

02:47.000 --> 02:50.000
but I think the real power lies in its use flags,

02:50.000 --> 02:53.000
which allow you to customize it saying,

02:53.000 --> 02:57.000
I want this feature or that feature or not this library or whatever,

02:57.000 --> 03:00.000
and yeah, it's very powerful.

03:00.000 --> 03:03.000
And sports many architectures.

03:03.000 --> 03:07.000
It was the last distro to sport Intel Lightning.

03:08.000 --> 03:11.000
So on the right, we have flat car container Linux,

03:11.000 --> 03:18.000
which was forked from the original Core OS container Linux in 2018.

03:18.000 --> 03:21.000
It's also community maintained.

03:21.000 --> 03:25.000
Our legal aspects are handled by the cloud native computing foundation.

03:25.000 --> 03:30.000
Now, me being a Microsoft engineer, you might think, hey, isn't flat car,

03:30.000 --> 03:32.000
maybe a Microsoft product.

03:32.000 --> 03:35.000
Not at all, it's a community project,

03:35.000 --> 03:38.000
maintained out in the open, non-git hub.

03:38.000 --> 03:42.000
You can come and talk to us on matrix where we discuss everything out in the open.

03:42.000 --> 03:45.000
And you're very welcome, so please do.

03:45.000 --> 03:52.000
And we do have other contributors who do this as day-to-day job outside Microsoft as well.

03:52.000 --> 03:54.000
It's in the mutable distro.

03:54.000 --> 03:56.000
It's a hot topic these days.

03:56.000 --> 03:58.000
What does that actually mean?

03:58.000 --> 04:01.000
There's no package manager.

04:01.000 --> 04:05.000
This size read only and upgrades are atomic.

04:05.000 --> 04:08.000
It makes it quite good security wise.

04:08.000 --> 04:13.000
It's derived from Gen2, so it's not obviously not got the package manager on it,

04:13.000 --> 04:16.000
but it's built from Gen2.

04:16.000 --> 04:21.000
But its history comes from Google's Chromium OS and the old container Linux,

04:21.000 --> 04:26.000
but we've cut from that now and it's just directly derived from Gen2.

04:26.000 --> 04:32.000
And yet, minimal just has what you need for running container workloads.

04:32.000 --> 04:38.000
So you can have it with Docker or Kubernetes, pardon man, container D, you get to choose.

04:38.000 --> 04:41.000
But it still feels like a familiar Linux distro.

04:41.000 --> 04:47.000
We haven't cut it down so much that there's still SSH in the still batch and what have you.

04:47.000 --> 04:53.000
And it supports MD64, ARM64, and unofficially risked five.

04:53.000 --> 04:58.000
You might have seen the talk about that yesterday from Adrian Fladoo.

04:58.000 --> 05:04.000
On VMs and bare metal, so we still consider those just as important.

05:04.000 --> 05:10.000
And runs on, I think, 20, at least 20 cloud environments now, not just as you.

05:10.000 --> 05:15.000
So moving on, what do we mean by sitting in a tree?

05:15.000 --> 05:18.000
Well, maybe this kind of tree.

05:18.000 --> 05:22.000
This is the family tree of Gen2 forks and derivatives.

05:22.000 --> 05:25.000
We're in there somewhere if you squint really closely.

05:25.000 --> 05:27.000
Or maybe this kind of tree.

05:27.000 --> 05:29.000
So, this works.

05:37.000 --> 05:39.000
I can deal with that.

05:39.000 --> 05:48.000
Yes, so this is a black car's main code repository.

05:48.000 --> 05:52.000
And might look familiar if you're a Gen2 user.

05:52.000 --> 05:55.000
It includes a copy of the subset of Gen3 packages.

05:55.000 --> 06:00.000
Not all of them because it was faster to just have the subset.

06:00.000 --> 06:04.000
And having it as a git sub module would be annoying.

06:04.000 --> 06:08.000
This way we can keep it aligned with flat car releases.

06:08.000 --> 06:11.000
It just makes a whole thing easier.

06:11.000 --> 06:16.000
Now, obviously, with this many shared packages, there's a good level of shared interest.

06:16.000 --> 06:20.000
But we also use other Gen2 tools directly like Catalyst,

06:20.000 --> 06:22.000
which we used to build our SDK.

06:22.000 --> 06:24.000
Not flat car itself, but the SDK.

06:24.000 --> 06:29.000
We use cross-step for cross-compiling and package check for QA.

06:29.000 --> 06:33.000
So how's this relationship evolved over times?

06:33.000 --> 06:40.000
So since Chromium OS, I'm the first maintainer to wear both hats.

06:40.000 --> 06:43.000
I'm hoping I'm not going to be the last time I need to sweet talk.

06:43.000 --> 06:45.000
I want to do people in my team, maybe.

06:45.000 --> 06:49.000
But considering that container Linux started in 2013,

06:49.000 --> 06:53.000
it's a shame it took so long to bridge that gap.

06:53.000 --> 06:58.000
And stemming from that in the seven years of container Linux is existence.

06:58.000 --> 07:02.000
There are only seven commits to Gen2's main package repo.

07:02.000 --> 07:04.000
That's not great.

07:04.000 --> 07:09.000
For about the same time period in flat car, we had 130 commits.

07:09.000 --> 07:11.000
At least, it's hard to count.

07:11.000 --> 07:13.000
I commit as my Gen2 address.

07:13.000 --> 07:18.000
Sometimes it's hard to tell whether a commit is work or just in my free time.

07:18.000 --> 07:20.000
But there's a overlap there.

07:20.000 --> 07:23.000
If you're thinking, oh, isn't this all just you?

07:23.000 --> 07:26.000
No. I can't take all the credit, even without me.

07:26.000 --> 07:29.000
There was 47 commits from the rest of the team.

07:29.000 --> 07:36.000
So they were already leaning quite heavily towards upstream.

07:36.000 --> 07:42.000
We fork a number of packages in flat car simply because the file system lay out slightly different

07:42.000 --> 07:44.000
with USR being read only.

07:44.000 --> 07:49.000
But we've recently found new ways to deal with that without having to fork the whole package.

07:49.000 --> 07:53.000
So that number is now reducing and bringing us closer to upstream.

07:53.000 --> 07:58.000
And in general, we just see a lining closer to Gen2 as a good thing.

07:58.000 --> 08:02.000
Whereas container Linux arguably drifted away from Gen2 over time.

08:02.000 --> 08:04.000
We're going in the opposite direction.

08:04.000 --> 08:09.000
Then we just believe that more commonality is good for both of us.

08:09.000 --> 08:13.000
So, there's Gen2 help flat car.

08:13.000 --> 08:16.000
Not by asking what flat car needs.

08:16.000 --> 08:18.000
We didn't have one Gen2 to ask that.

08:18.000 --> 08:21.000
It didn't come to anything yet, but we'll see.

08:21.000 --> 08:25.000
But that's okay because that isn't what flat car needs.

08:26.000 --> 08:31.000
So, flat car chose Gen2 because it's already a great fit.

08:31.000 --> 08:33.000
It's very flexible, as I said.

08:33.000 --> 08:39.000
And it's fantastic for cross compiling, which is very important for us as you'll see in a minute.

08:39.000 --> 08:42.000
And you might think, well, we didn't choose Gen2.

08:42.000 --> 08:47.000
It was contained in Linux and Kremlin OS before us.

08:47.000 --> 08:54.000
But I think if we were making this decision today, I think we've still choose Gen2.

08:54.000 --> 09:01.000
And at no point, if we said, oh, maybe we should try something else.

09:01.000 --> 09:05.000
No, we're very much still on board with Gen2.

09:05.000 --> 09:09.000
Gen2 helps by continuing to be well maintained.

09:09.000 --> 09:14.000
Keeps the packages updated with a strong adherence to QA.

09:14.000 --> 09:20.000
And I think the QA is really important when Gen2 is that flexible because you can't just assume

09:20.000 --> 09:22.000
it works on my machine.

09:22.000 --> 09:26.000
There's so many literally so many variables in play.

09:26.000 --> 09:31.000
They apply security fixes very promptly, and that's a key focus for flat car.

09:31.000 --> 09:35.000
And they handle our contributions, confidently.

09:35.000 --> 09:38.000
You know, don't leave them dangling for ages.

09:38.000 --> 09:42.000
And there's no issues with these being paid contributions.

09:42.000 --> 09:45.000
No suggestions of conflict of interest.

09:45.000 --> 09:49.000
They just recognize that we want to make Gen2 better.

09:50.000 --> 09:53.000
So, how does flat car help Gen2?

09:53.000 --> 09:56.000
Well, cross-compiling is one way.

09:56.000 --> 10:02.000
So we cross-compile everything, even for AMD64, because we believe it gives a cleaner build.

10:02.000 --> 10:06.000
By no accident, we're pulling in build components.

10:06.000 --> 10:11.000
And it just so happens that I've been Gen2's cross-compile guys since about 2006.

10:11.000 --> 10:15.000
And being paid to spend more time on those issues is great.

10:16.000 --> 10:25.000
So, flat cars help by fixing cross-compile issues in many packages from, say, SSSD to open ice,

10:25.000 --> 10:28.000
and e-classes from go to rust.

10:28.000 --> 10:31.000
And the great thing about e-classes is once you fix the issues there,

10:31.000 --> 10:34.000
it's pretty much fixes them for all those packages.

10:34.000 --> 10:40.000
And we're enhancing the cross-dev tool chain utility for cross-compiling.

10:40.000 --> 10:49.000
And, yeah, you'll be able to pick a different portage profile rather than the default embedded one,

10:49.000 --> 10:51.000
which isn't very helpful.

10:51.000 --> 10:55.000
And we're moving the need for extra flags when building against alternative statistics,

10:55.000 --> 10:59.000
which means that you can build against a different directory,

10:59.000 --> 11:01.000
other than the one that the tool chain's hosted in.

11:01.000 --> 11:07.000
So, say you mount your root file system from your Raspberry Pi or something over NFS,

11:07.000 --> 11:10.000
and you can just directly cross-compile against that,

11:10.000 --> 11:14.000
and much quicker than building on the Pi, for instance.

11:14.000 --> 11:18.000
And we're making it possible to cross-compile pre-fix systems.

11:18.000 --> 11:23.000
If you're not familiar with Gen2 pre-fix, this allows you to have a sort of

11:23.000 --> 11:28.000
Gen2 system in an arbitrary directory, not slash USR.

11:28.000 --> 11:36.000
And flat car wants to combine this with system D6 to bolt on additional packages that you might need on to flat car.

11:36.000 --> 11:42.000
So, we've also helped with the package manager.

11:42.000 --> 11:45.000
We're Gen2's flexibility.

11:45.000 --> 11:48.000
You can often end up in possible situations,

11:48.000 --> 11:50.000
and portage tries to figure them out for you,

11:50.000 --> 11:54.000
but sometimes use interventions required, for example,

11:54.000 --> 11:57.000
with circular dependencies.

11:57.000 --> 12:01.000
You need to disable the feature, but first time you build it,

12:01.000 --> 12:04.000
and then enable it later, and that's a pain for us,

12:04.000 --> 12:06.000
because it affects our bootstrapping process,

12:06.000 --> 12:10.000
and we have to shepherd it through with scripting,

12:10.000 --> 12:13.000
and we're thinking, well, it tells you what to do.

12:13.000 --> 12:15.000
Why doesn't it just do that for you?

12:15.000 --> 12:17.000
Turns out that's quite a hard problem.

12:17.000 --> 12:20.000
It's a 12,000-line Python file involved.

12:20.000 --> 12:22.000
I've said a bit for a few days,

12:22.000 --> 12:25.000
and it worked in some simple cases,

12:25.000 --> 12:27.000
but it needs a little bit more work.

12:27.000 --> 12:29.000
What's this space?

12:30.000 --> 12:34.000
We've added some new packages, so some of us use Gen2

12:34.000 --> 12:36.000
to do our work, like me.

12:36.000 --> 12:39.000
It didn't have to, but it helps with the upstreaming.

12:39.000 --> 12:41.000
Many more Microsoft engineers use Linux,

12:41.000 --> 12:42.000
believe it or not.

12:42.000 --> 12:45.000
Maybe Gen2, but I have heard of one of them.

12:45.000 --> 12:47.000
But even if they don't, I just wanted to set an example

12:47.000 --> 12:49.000
to the rest of Microsoft to say, look,

12:49.000 --> 12:52.000
if you're installing stuff on your distro

12:52.000 --> 12:55.000
or packaging stuff in a private repo,

12:55.000 --> 12:56.000
don't do that.

12:56.000 --> 12:58.000
You know, make it available to everyone.

12:58.000 --> 13:02.000
And that car's SDK is basically a Gen2 image as well,

13:02.000 --> 13:06.000
so any tools we need there might also be useful for Gen2.

13:06.000 --> 13:09.000
So we've therefore added a few packages,

13:09.000 --> 13:13.000
in tune, which we use to access our corporate resources.

13:13.000 --> 13:15.000
That's not just an internal Microsoft thing.

13:15.000 --> 13:18.000
Your company might use in tune as well.

13:18.000 --> 13:21.000
Similarly, as your VPN client, not just internal.

13:21.000 --> 13:24.000
If you use Azure, you may well need that.

13:25.000 --> 13:27.000
Azure VM Utils, as the name applies,

13:27.000 --> 13:30.000
is used for running VMs in Azure,

13:30.000 --> 13:32.000
helps with identifying the disks.

13:32.000 --> 13:36.000
Gen2 was actually the first distro to have that package.

13:36.000 --> 13:39.000
Git credential manager is an improvement

13:39.000 --> 13:41.000
over using traditional SSH keys,

13:41.000 --> 13:44.000
because it has like multi-factor authentication.

13:44.000 --> 13:46.000
Doesn't just support Git Hub.

13:46.000 --> 13:49.000
You can use it with BitBucket and GitLab as well.

13:49.000 --> 13:52.000
And the Azure SDK for C++,

13:52.000 --> 13:57.000
which we use for our signing tool that we have for secure boot,

13:57.000 --> 14:00.000
but could be useful for all sorts of things.

14:00.000 --> 14:02.000
Yeah, and we're keeping these up to date.

14:02.000 --> 14:05.000
We didn't just throw them over the wall.

14:05.000 --> 14:09.000
So yeah, and we're also helping with VM firmware.

14:09.000 --> 14:12.000
So you may know of EDK2,

14:12.000 --> 14:16.000
some open source implementation of the UFI standard.

14:16.000 --> 14:18.000
It might not realize that you use this,

14:18.000 --> 14:21.000
but it's commonly used for booting VMs of QMU.

14:21.000 --> 14:23.000
And we do this all the time.

14:23.000 --> 14:26.000
So if we need to test something out locally,

14:26.000 --> 14:28.000
or in our CI,

14:28.000 --> 14:33.000
we run extensive tests that spins up QMU all the time.

14:33.000 --> 14:36.000
And it was quite unloved in Gen2 to be honest.

14:36.000 --> 14:39.000
It was quite a few years behind in terms of the package version,

14:39.000 --> 14:41.000
and there were loads of open bugs.

14:41.000 --> 14:44.000
So we bumped the package,

14:44.000 --> 14:47.000
not just once, but twice, and we'll keep that going.

14:47.000 --> 14:51.000
It makes a pure boot work out of the box.

14:51.000 --> 14:55.000
It used to involve a very convoluted bootstraping process

14:55.000 --> 14:57.000
that didn't even work anymore, I think.

14:57.000 --> 15:01.000
And we added support for other architectures,

15:01.000 --> 15:03.000
so before it only supported AMD64,

15:03.000 --> 15:06.000
now we've added ARM64 and Risk5,

15:06.000 --> 15:10.000
and I've made the in people quite extensible in that regard.

15:10.000 --> 15:12.000
So we're hoping to add more.

15:12.000 --> 15:15.000
I think the next one will be the lung architecture.

15:15.000 --> 15:19.000
And we noticed that other distros are providing

15:19.000 --> 15:23.000
the images in different sizes for when those update these days.

15:23.000 --> 15:27.000
You need four megabyte images to the traditional two.

15:27.000 --> 15:31.000
And they were providing them in QK2 format,

15:31.000 --> 15:33.000
as well, which is more space efficient.

15:33.000 --> 15:34.000
So we did that too.

15:34.000 --> 15:37.000
And note that upstream's build system doesn't do any of that for you.

15:37.000 --> 15:39.000
So this all had to be done in the e-build.

15:39.000 --> 15:40.000
It was quite a lot of work.

15:40.000 --> 15:44.000
But in doing that, we resolved most of the open bugs.

15:45.000 --> 15:49.000
So how might Blackfire help Gen2 in the future?

15:49.000 --> 15:54.000
So as I said, we don't just post stuff over the wall.

15:54.000 --> 15:59.000
We recognize we've got a responsibility to maintain these things going forward,

15:59.000 --> 16:01.000
so we'll see more of the above.

16:01.000 --> 16:05.000
And we were discussing just this weekend,

16:05.000 --> 16:09.000
possibly assisting with key wording and stabilisation.

16:09.000 --> 16:13.000
So we have architectures, teams in Gen2,

16:13.000 --> 16:18.000
whose job it is just to check that everything works on different architectures

16:18.000 --> 16:21.000
as they get updated.

16:21.000 --> 16:23.000
Without extensive CI tests,

16:23.000 --> 16:26.000
we could semi-automate a lot of that and just say,

16:26.000 --> 16:30.000
hey, we've tested it on 164 and risk-vive and it's all fine.

16:30.000 --> 16:34.000
So yeah, there's some enthusiasm there.

16:35.000 --> 16:38.000
Blackfire has to pass the output of ported,

16:38.000 --> 16:41.000
choosing some really ugly regular expressions.

16:41.000 --> 16:43.000
And we really don't want to do that.

16:43.000 --> 16:45.000
And there was actually a Google summer of code project

16:45.000 --> 16:48.000
to add machine readable output like Jason,

16:48.000 --> 16:50.000
so we're quite happy to pick that up.

16:50.000 --> 16:52.000
So that's on the radar too.

16:52.000 --> 16:57.000
And if Blackfire starts using these,

16:57.000 --> 17:00.000
start using Gen3 prefixes I mentioned before,

17:00.000 --> 17:02.000
then potentially there are a lot more packages

17:02.000 --> 17:04.000
that aren't traditionally used with prefixes

17:04.000 --> 17:06.000
will come under our radar.

17:06.000 --> 17:09.000
So they may be issues lurking now

17:09.000 --> 17:13.000
that will be ready in willing to place.

17:13.000 --> 17:17.000
So some final thoughts.

17:17.000 --> 17:22.000
So derivative distributions typically target a niche,

17:22.000 --> 17:25.000
but this shouldn't be seen as stealing

17:25.000 --> 17:28.000
users away from the parent distro.

17:28.000 --> 17:33.000
The mutual benefit of working together is worth far more.

17:33.000 --> 17:34.000
And I'd say more than that,

17:34.000 --> 17:37.000
just every distro is your friend, right?

17:37.000 --> 17:39.000
You know, we all work with the same upstreams,

17:39.000 --> 17:43.000
so we can all benefit from working together.

17:43.000 --> 17:47.000
And maintaining a distro can be your day job.

17:47.000 --> 17:50.000
I've heard it said it fuzzed M that,

17:50.000 --> 17:53.000
you know, anyone could get a job in open source,

17:53.000 --> 17:54.000
and I thought, well, yeah,

17:54.000 --> 17:57.000
but I'm going to get paid for Gen2, right?

17:57.000 --> 17:58.000
Part two niche.

17:58.000 --> 17:59.000
Here we are.

17:59.000 --> 18:01.000
And although, of course,

18:01.000 --> 18:03.000
primarily maintaining flatcard,

18:03.000 --> 18:07.000
there are other people who do Gen2 as their day job.

18:07.000 --> 18:13.000
And they could sometimes be friction between paid developers

18:13.000 --> 18:17.000
and unpaid developers and due to differing requirements,

18:17.000 --> 18:20.000
time scales, expectations, and we've seen here,

18:20.000 --> 18:23.000
and elsewhere that with the right mindset,

18:23.000 --> 18:26.000
the relationship can prove very fruitful.

18:27.000 --> 18:29.000
Thank you for listening.

18:29.000 --> 18:31.000
Thank you.

18:31.000 --> 18:36.000
Thank you.

18:36.000 --> 18:38.000
Thank you.

18:38.000 --> 18:39.000
Oh.

18:39.000 --> 18:40.000
Don't wait, we got something.

18:40.000 --> 18:41.000
No.

18:41.000 --> 18:42.000
Okay.

18:42.000 --> 18:44.000
You have to go back.

18:48.000 --> 18:50.000
I was watching your cues, I'm not sped

18:50.000 --> 18:52.000
for that a bit quickly.

18:52.000 --> 18:55.000
Oh yeah, I looked like I forgot how to do that.

18:56.000 --> 18:57.000
Yeah.

18:57.000 --> 18:59.000
More stuff.

19:03.000 --> 19:04.000
Okay.

19:04.000 --> 19:05.000
Thank you.