WEBVTT 00:00.000 --> 00:07.000 All right, can you hear me? 00:07.000 --> 00:08.000 Yes. 00:08.000 --> 00:09.000 Excellent. 00:09.000 --> 00:10.000 Okay. 00:10.000 --> 00:11.000 Yeah. 00:11.000 --> 00:15.000 So we want to give a presentation about our project that we have. 00:15.000 --> 00:17.000 I've been working on for a while now. 00:17.000 --> 00:20.000 And we love to have some more input from others. 00:20.000 --> 00:22.000 And hopefully I have some good discussion about it. 00:22.000 --> 00:28.000 Which is about building a multi-product stack based on open source components. 00:29.000 --> 00:32.000 Just very briefly about ourselves, my name is Goodadationer. 00:32.000 --> 00:34.000 I work for IBM. 00:34.000 --> 00:37.000 I'm also part of the summer team since a long time. 00:37.000 --> 00:41.000 And yeah, I've been dealing with all kind of aspects with some of the past. 00:41.000 --> 00:45.000 Mostly with active directory integration, these sorts of things. 00:45.000 --> 00:50.000 But just recently also now with building an SMB product based on class of file systems. 00:50.000 --> 00:53.000 And in the beginning we were doing this with cluster. 00:53.000 --> 00:55.000 We also have a loop played in a port control. 00:55.000 --> 01:01.000 And now we are building an SMB layer on top of CFFS, which just got G8. 01:01.000 --> 01:07.000 And I manage a team of engineers looking into these integration aspects. 01:07.000 --> 01:10.000 Yeah, for IBM. 01:10.000 --> 01:11.000 Yeah. 01:11.000 --> 01:12.000 Yeah. 01:12.000 --> 01:14.000 I'm basically working on that. 01:14.000 --> 01:19.000 So as I mentioned, I'm also as somebody member in upstream. 01:19.000 --> 01:22.000 And focus on the VFS layer of things. 01:22.000 --> 01:27.000 Basically the integration to the distributed file systems. 01:27.000 --> 01:28.000 Okay. 01:28.000 --> 01:30.000 So what is mighty protocol? 01:30.000 --> 01:33.000 What do we really understand with this term? 01:33.000 --> 01:36.000 Even the term itself, sometimes it's written with a white space. 01:36.000 --> 01:38.000 Sometimes we're not, I just have my own convention. 01:38.000 --> 01:42.000 So it's all about a concurrent access to storage, 01:42.000 --> 01:44.000 or the standardized network access protocols. 01:44.000 --> 01:46.000 So you have one storage solution. 01:46.000 --> 01:49.000 And you have a well established list of network access protocols, 01:49.000 --> 01:50.000 which you all know. 01:50.000 --> 01:54.000 And you want to create an architecture where it can basically 01:54.000 --> 01:58.000 access the same storage of a multiple protocols at the same time. 01:58.000 --> 02:00.000 Which is of course challenging. 02:00.000 --> 02:02.000 And we're going to see. 02:02.000 --> 02:06.000 So for now, we will only looking at NFS and SMB. 02:06.000 --> 02:10.000 Of course, there has been, when we start the discussions around that. 02:10.000 --> 02:15.000 A lot of interest also in adding S3, right, which is kind of that the new 02:16.000 --> 02:20.000 of famous access protocol used by so many cloud components. 02:20.000 --> 02:24.000 A bit of something that we do not get considered for our purposes. 02:24.000 --> 02:27.000 So we only focus really on NFS and for SMB. 02:27.000 --> 02:32.000 I try to paint a very poorly looking image that kind of visualizes 02:32.000 --> 02:33.000 what we have to mind. 02:33.000 --> 02:36.000 So we have some entity of storage. 02:36.000 --> 02:38.000 It could be a share, it could be a directory. 02:38.000 --> 02:41.000 It could be a storage system of some sort. 02:41.000 --> 02:44.000 But at least some small granular entity of storage. 02:44.000 --> 02:48.000 And then you have this multiple protocol heads sitting on top of that. 02:48.000 --> 02:50.000 Accessing the same storage state at the same time. 02:50.000 --> 02:53.000 And then you have a myriad of possible client combinations. 02:53.000 --> 02:58.000 I just put Linux and Windows clients here as the front end consumers. 02:58.000 --> 03:03.000 Yeah, we were going to detail the details of all these architecture. 03:03.000 --> 03:05.000 So what is really the use case? 03:05.000 --> 03:07.000 Why are we actually really doing it? 03:07.000 --> 03:11.000 Why are we looking into this kind of infrastructure? 03:11.000 --> 03:19.000 So it's really about supporting just the whole suite of classic office use cases. 03:19.000 --> 03:22.000 Like like work like Leap office and so forth. 03:22.000 --> 03:27.000 This is one of the messages that we, I mean, I've got from our product managers. 03:27.000 --> 03:31.000 Even if you then consider word as just a simple use case. 03:31.000 --> 03:37.000 If you're familiar with the pay load that really the word application puts on top of a network 03:37.000 --> 03:38.000 system. 03:38.000 --> 03:44.000 It actually explores immediately all sorts of very complex aspects of these protocols. 03:44.000 --> 03:50.000 So it's everything but for sure not an easy use case, but nonetheless this is what people 03:50.000 --> 03:54.000 who were starting this discussion ahead in mind. 03:54.000 --> 03:57.000 Of course, we want to also allow the whole suite of developer tools. 03:57.000 --> 03:59.000 Make it Python from pilots. 03:59.000 --> 04:00.000 You name it. 04:00.000 --> 04:03.000 These need to be supported by the architecture. 04:03.000 --> 04:07.000 Also even web service that may be even simple databases. 04:07.000 --> 04:11.000 In particular databases are going to be very challenging as we're going to hear. 04:11.000 --> 04:13.000 But it's on the list as well. 04:13.000 --> 04:20.000 So if we start to allow supporting these use cases, we certainly really need to have a lot of focus on 04:20.000 --> 04:22.000 data consistency, right? 04:22.000 --> 04:26.000 Whatever we kind of allow between these protocols, we need to absolutely make sure that we don't 04:26.000 --> 04:30.000 put the storage and the central location at any risk. 04:30.000 --> 04:34.000 And so we need to look very deeply at all aspects of file locking. 04:34.000 --> 04:38.000 We need to also make sure that we have a consistent understanding of how to deal with 04:38.000 --> 04:39.000 caching. 04:39.000 --> 04:41.000 Be it the caching that the file system does. 04:41.000 --> 04:45.000 Be it the caching algorithms, which are exposed over these network protocols. 04:45.000 --> 04:50.000 And of course, all things related to authorization data must be very consistent. 04:50.000 --> 04:51.000 Acles in particular. 04:51.000 --> 04:56.000 At no means we may we may ever slip in. 04:56.000 --> 04:59.000 Yeah, access permissions in these kind of scenarios. 04:59.000 --> 05:03.000 Metadata, of course, needs to be dealt with as well. 05:03.000 --> 05:06.000 So has actually this been done before. 05:06.000 --> 05:11.000 And yes, of course, there are many proprietary solutions that provides this functionality 05:11.000 --> 05:13.000 to some sort, of course, always. 05:13.000 --> 05:18.000 In the fields, there's net up mix mode, vast, wick, nasuni, dellmc. 05:18.000 --> 05:23.000 There's an ongoing list, there are many new products, 05:23.000 --> 05:25.000 who just look at this as well. 05:25.000 --> 05:29.000 Tuxera isn't considering doing this, I think, with a new NFS server. 05:29.000 --> 05:34.000 But these are all fully proprietary solutions, meaning they have their own file system, 05:34.000 --> 05:36.000 which they have under full control. 05:36.000 --> 05:40.000 And they also implement their own set of protocols service on top of that. 05:40.000 --> 05:45.000 But there are also examples of, at least a partly open-based solution. 05:45.000 --> 05:48.000 Cetera is one that I started looking into. 05:48.000 --> 05:51.000 I don't have many details on it right now. 05:51.000 --> 05:55.000 But there is the currently named IBM Storage scale product, 05:55.000 --> 05:58.000 which I've already had a different name over the past. 05:58.000 --> 06:04.000 It's also an older product for IBM that uses the open source, 06:04.000 --> 06:08.000 components, server, and NFS Ganesha for the protocols, 06:08.000 --> 06:10.000 but provides its own proprietary file system, 06:10.000 --> 06:16.000 called GPS, which is a very mature and very well-performing file system. 06:16.000 --> 06:20.000 And their architecture, and we're going to touch a little bit on that as well, 06:20.000 --> 06:26.000 is all based on the kernel-based GPS file system that they mount. 06:26.000 --> 06:30.000 So they mount really a file system, as if it would be a local file system. 06:30.000 --> 06:33.000 And then they have these protocols just operating on the local files, 06:33.000 --> 06:35.000 as if they were really local. 06:35.000 --> 06:41.000 And this is only possible to, while leveraging a couple of integration pieces, 06:41.000 --> 06:48.000 that in particular, sound provides, to offload some of the complexity to the kernel itself. 06:48.000 --> 06:52.000 The three examples I gave here are kernel share modes, kernel uploads, 06:52.000 --> 06:58.000 and post-exlocking, so aspects for locking mostly, and also for caching. 06:58.000 --> 07:04.000 But if you use this infrastructure, and whatever let the kernel, 07:04.000 --> 07:09.000 the Linux kernel take the important decisions about access control, 07:09.000 --> 07:16.000 and the locking, in that example, then you will quickly identify a couple of limitations, 07:16.000 --> 07:19.000 these kernel APIs, they have been designed at some time. 07:19.000 --> 07:26.000 So they kind of didn't catch up with the ongoing extensions of many of these algorithms, 07:26.000 --> 07:29.000 like for kernel-leases, for example. 07:29.000 --> 07:32.000 So they didn't really change a lot, they are hard to implement, 07:32.000 --> 07:37.000 and then sometimes they are not implemented in a way that we can easily consume them, 07:37.000 --> 07:41.000 or they are even not always following the specification fully. 07:41.000 --> 07:46.000 So it's a very inflexible way to kind of do any changes at the kernel level. 07:46.000 --> 07:50.000 This will either have an impact on what we are planning to do. 07:50.000 --> 07:56.000 So there are examples, but is there also a standard, is there kind of any attempt being made ever 07:56.000 --> 08:01.000 to specify this behavior or this architecture in some sort? 08:01.000 --> 08:05.000 The network protocols per se, as you know, they are well standardized, 08:05.000 --> 08:11.000 and we are defined and we are documented for the NFS specifications. 08:11.000 --> 08:18.000 There are various RFCs, there are industry-wide entities that really just look into defining all aspects 08:18.000 --> 08:24.000 and all extensions and improvements for the protocols over a long process of iterations. 08:24.000 --> 08:29.000 And so there's a lot of specification going on there as well for SMB not so much, 08:29.000 --> 08:32.000 because it was mostly Microsoft owned protocol, 08:32.000 --> 08:39.000 and only with the core decision that has been made Microsoft was actually forced 08:39.000 --> 08:41.000 to finally document their protocols. 08:41.000 --> 08:49.000 But they did this in a really beautiful manner, not only specifying the properties 08:49.000 --> 08:52.000 that you will see on the network, but also document the client behavior. 08:52.000 --> 08:56.000 And that gives a person to implement the protocol a very good hint. 08:56.000 --> 08:58.000 What is really the expectation? 08:58.000 --> 09:01.000 The future ABC, is it optional? 09:01.000 --> 09:02.000 Is it really enforced? 09:02.000 --> 09:05.000 Will all Windows clients use it? 09:05.000 --> 09:07.000 And what happens if they are not? 09:07.000 --> 09:10.000 If these features are not supported and so forth, 09:10.000 --> 09:13.000 so it kind of not only specifies the protocol per se, 09:13.000 --> 09:16.000 but also the client and server expectations and the behavior. 09:16.000 --> 09:19.000 Very different from the NFS specifications. 09:19.000 --> 09:26.000 They leave a lot of room for interpretation and for the option character of implementing these features 09:26.000 --> 09:31.000 that makes it way more difficult to come up with the standard way of implementing a specific feature 09:31.000 --> 09:33.000 unlike SMB. 09:33.000 --> 09:41.000 But no one really made an attempt to kind of create an overlap and create a specification for all these overlaying aspects. 09:41.000 --> 09:48.000 And there are so many similarities in the protocols as you will see that this sounds to be very likely 09:48.000 --> 09:50.000 and easy thing to do, but it is not. 09:50.000 --> 09:54.000 There's a lot of detail which makes this very hard. 09:54.000 --> 10:00.000 So all these solutions that I've mentioned, they all came up with their own way of doing a multiple protocol implementation, right? 10:00.000 --> 10:04.000 There's no specification that you just follow and write down implement all things. 10:04.000 --> 10:12.000 It's really a lot about what can the infrastructure provide, what use cases do you want to support, what clients do you have even. 10:12.000 --> 10:20.000 But there is also the Microsoft Windows server and we as we both have a very strong sample focus obviously. 10:20.000 --> 10:24.000 So we always look okay how what does Windows do, how does Windows behave? 10:24.000 --> 10:34.000 And maybe this is a good opportunity to consider this to be a reference implementation and learn from that and see if things can be derived from that. 10:34.000 --> 10:40.000 They do support this concurrent sharing over SMB and NFS for many years now. 10:40.000 --> 10:49.000 But of course we want to be 100% open source based, so we will have to look also on what the servers can provide that we have. 10:49.000 --> 10:53.000 Yeah accessible. 10:53.000 --> 10:58.000 Along with all these discussions, I mean these are not new things. 10:58.000 --> 11:07.000 There is a for calendar, sitting behind over there, he made a wonderful paper, a great presentation at the storage developer conference two years ago, 11:07.000 --> 11:11.000 to really explain all the different concepts, right? 11:11.000 --> 11:15.000 So the concept of his session, for example, how does that map in these two worlds, 11:15.000 --> 11:21.000 apart from all the more obvious things like locking and directory names and these sorts of things? 11:21.000 --> 11:31.000 So that's really highly recommended start point to really start overseeing the complexity of really mapping these two worlds to one thing. 11:31.000 --> 11:37.000 Of course when we now want to implement something, we want to learn from that. 11:37.000 --> 11:43.000 We want to build up on top of that, we don't want to kind of bring in any new protocol or protocol extensions to some sort. 11:43.000 --> 11:51.000 We would ideally really just use the completely unmodified NFS and SMB clients out there, build in Windows, build in Linux or wherever, 11:51.000 --> 12:00.000 and let them operate to one server without really being aware that there's really a multiple infrastructure on the server side. 12:00.000 --> 12:06.000 So the server really needs to make sure that it understands all these problems that has a solution for all of them, 12:06.000 --> 12:12.000 or at least kind of has everything under control, so that there is no exposure of any kind of, 12:12.000 --> 12:17.000 yeah, undefined field, let's put it that way. 12:17.000 --> 12:29.000 So that's that we need to look at the components that we have at hand to really see what is available already, 12:29.000 --> 12:40.000 and see how these user space implementations of SMB and NFS can be modified appropriately if they don't already provide APIs that help us, 12:40.000 --> 12:43.000 you know, join in to implement it. 12:43.000 --> 12:48.000 And it is of course obvious that the file system which we are going to use really plays a key role, 12:48.000 --> 12:52.000 and as you know, we also part of the safe organization. 12:52.000 --> 12:57.000 So of course our file system of choice, our storage system is the safe infrastructure, 12:57.000 --> 13:05.000 where we have a lot of ways to bring in new changes and to plan out supporting new scenarios. 13:06.000 --> 13:11.000 There's also an important understanding that we are not planning to put a lot of authority on the kernel, 13:11.000 --> 13:18.000 but to really use the different API that we use in order to communicate with our storage system in user space. 13:18.000 --> 13:23.000 So we have an library which is easy to be extended, which is currently used by NFS connection, 13:23.000 --> 13:32.000 also by server as an open, going to more details into, and this is what we plan on building on. 13:32.000 --> 13:35.000 Okay, so I pass it up to you. 13:48.000 --> 13:52.000 All right, so let's look at some of the building blocks here, 13:52.000 --> 13:56.000 and as you can see, here is where you see, 13:56.000 --> 14:00.000 for the first time in this slide. 14:00.000 --> 14:04.000 As you all know, some of you might have already attended the earlier talk. 14:04.000 --> 14:08.000 It is an open source storage ecosystem which provides file, 14:08.000 --> 14:14.000 block, object, everything, and it is highly available and fault tolerant. 14:14.000 --> 14:20.000 We, as Gunther just mentioned, we make use, we do not rely on the kernel space. 14:20.000 --> 14:30.000 We rely mostly on the user space, because that's if most of the things majority of these logic happens in the user space. 14:30.000 --> 14:36.000 So we do have a library called LibsFFS with high level and lower level APIs. 14:36.000 --> 14:45.000 Right now, we make use of the lower level APIs to connect, connect with the underlying file system from a protocol site, 14:45.000 --> 14:53.000 like if it is from SMB or if it is from NFS, we make use of this LibsFFS library. 14:53.000 --> 15:00.000 And on the SMB site, obviously, some by search or choice here, 15:00.000 --> 15:06.000 and we do have a layer called, which we have a, we have a layer in Samba, 15:06.000 --> 15:12.000 where we have a module called, module dedicated for self. 15:12.000 --> 15:18.000 You'll see the keywords have underscore new, which is a new module, but not very new. 15:18.000 --> 15:21.000 It is at least one or two years old. 15:21.000 --> 15:26.000 So that's the module that we use to integrate with the underlying self-file system. 15:26.000 --> 15:30.000 It is capable of using this lower level APIs from LibsFFS. 15:30.000 --> 15:37.000 It can do asynchronous IO calls, and of course, the FS script support, 15:37.000 --> 15:41.000 which was discussed in the earlier talk by Vengi. 15:41.000 --> 15:49.000 There is a kernel integration kernel client also from SFS site, but we are not using it. 15:49.000 --> 15:54.000 We have purely going through the LibsFFS user space. 15:54.000 --> 16:04.000 And from Samba site, there are some kernel level parameters that we could configure within Samba, 16:05.000 --> 16:08.000 for oblogs, shy modes, and locking. 16:08.000 --> 16:17.000 But at the same time, there are some limitations, like there are a particular set of oblogs that we have to disable internally 16:17.000 --> 16:21.000 to make some part of it clear. 16:21.000 --> 16:29.000 So the kernel level approach is kind of limited, and coming to the NFS world, 16:29.000 --> 16:34.000 so the open source approach is via the NFS Ganesha. 16:34.000 --> 16:41.000 In a similar way, where we have a VFS layer in Samba, we have a file system abstraction layer in NFS Ganesha, 16:41.000 --> 16:46.000 which allows us to talk to underlying file systems. 16:46.000 --> 16:54.000 And we do have a self-file module in NFS Ganesha, which handles this integration. 16:54.000 --> 17:02.000 The common approach here is that both these integrations, like whether it is from Samba or from NFS Ganesha, 17:02.000 --> 17:14.000 both of these make use of LibsFFS, it goes through the lower level API calls from the C native API library. 17:14.000 --> 17:20.000 A little bit about how the self storage tag looks like in the current upstream world. 17:20.000 --> 17:32.000 We did have some protocol enhancements, like we did not have a native deployment scenario for Sambi with SF. 17:32.000 --> 17:40.000 But recently with the technical release, we had this integrated service, so we can manage both NFS and Sambi in a similar way, 17:40.000 --> 17:45.000 from the same orchestration component of SF, which is a fedium. 17:45.000 --> 17:51.000 And we can do monitoring IO retlimiting and the first script as I mentioned earlier. 17:51.000 --> 18:03.000 There is also dashboard support for both NFS and Sambi, which you can basically do make use of to do the monitoring and IO retlimiting live stats. 18:03.000 --> 18:10.000 And from Samba's idea, obviously, for active directory integration, we need windbind. 18:10.000 --> 18:19.000 And we also do, we have integrated this aspect also, the active directory integration also with this storage tag. 18:19.000 --> 18:25.000 Yeah, and the common IO path here is LibsFFS, which is the library that we make use of. 18:25.000 --> 18:28.000 There is no kernel client involvement here. 18:28.000 --> 18:35.000 Now, I will quickly mention some of the technical differences. 18:35.000 --> 18:44.000 In some of the key aspects, when we consider the multiple protocol approach to such file systems. 18:44.000 --> 18:49.000 First and foremost, the echoes, we know how complex the echoes can be. 18:49.000 --> 19:04.000 But when we compare, for it is mostly or similar to some extent for Sambi and NFS, in terms that the permissions that we can do, 19:04.000 --> 19:14.000 we can configure the LODI aspects of the echoes. Those are all kind of similar concepts. 19:14.000 --> 19:18.000 But there are a subtle differences in both. 19:18.000 --> 19:23.000 In Sambi world, it is called a security descriptors. 19:23.000 --> 19:33.000 And in security descriptors, we can have different kind ofacle and risk like we can have a discretionary access control as cycles and stuff like that. 19:33.000 --> 19:37.000 It is not just the normal file directory echoes. 19:37.000 --> 19:46.000 And in Sambi, we can store these echoes as such in the file system using X Actors. 19:46.000 --> 19:57.000 And at the same time, and at the same time, Sambi can map these echoes into the posics world, which is obviously a lossy mapping. 19:57.000 --> 20:11.000 So right now, in the storage stack, in upstream, I think we have the touring of these securities descriptors as it is, using a different VFS module in Sambi calledaclexactor. 20:11.000 --> 20:20.000 And with NFS Genesh, I also, I think, have a silent integration for NFS Genesh users, posic cycles. 20:20.000 --> 20:37.000 So I don't think there is same app. Of course, yeah, I have linked draft the regards to how the mapping from NFS VFORacle, how low-ZGs and related details you can find it in that link. 20:37.000 --> 20:45.000 But NFS Genesh can do NFS VFORacles. So that's already that. 20:45.000 --> 20:55.000 When it comes to leaps of FFS, which is the common IOPA, yeah, leaps of FFS only provides posic cycles, right now. 20:55.000 --> 21:10.000 Yeah, that's it with echoes. And regarding the exclusive locking, there is a concept of share modes in Sambi, and a similar concept called share reservations in NFS. 21:10.000 --> 21:28.000 You can see a few of the flags that we use in both of these protocols like, there are some common things like read, write, but there is a, but in Sambi, we see a file share delete, which is not present in NFS Gen, NFS protocol. 21:28.000 --> 21:35.000 So here also there is a difference in how this export of all exclusive locking is considered. 21:35.000 --> 21:53.000 When it comes to leaps of FFS, we don't have anything right now. So for a multi protocol approach, we need to have some kind of interface from leaps of FFS to accept and accept all these semantics. 21:53.000 --> 22:05.000 And by trying to locking, in posics world, the locking is always adversary, but in Sambi world, it is considered to be a mandatory one. 22:05.000 --> 22:22.000 So if an application, which is opened and processed accessed by an Sambi client, now it is supposed to be done in a mandatory locking behavior that any other client is expected to check whether this is in use. 22:22.000 --> 22:36.000 And then respond accordingly. But in posics world, there is no such an invitation. Like, even if there is an lock held by other process, you can just simply access the file. 22:36.000 --> 22:46.000 Unless there are some file mode level, file mode bits are set and some kernel level involvement is, otherwise it is all adversary locks. 22:46.000 --> 22:58.000 And since FFS compliance to a posics file system, I see a typo that we can do only adversary locks right now. 22:58.000 --> 23:26.000 All right. And locking is one thing. The other thing is also caching. Of course, they are very similar concepts in the FFS and Sambi when it comes to caching, Sambi leases and NFS delegations have very similar functionality, even some similarity in the calls. 23:26.000 --> 23:38.000 But they also differences that need to be taken into account. But foremost, just by kind of creating this mapping, this is just one aspect, but we also need to kind of find a solution. 23:38.000 --> 23:47.000 How do we actually deal with conflicts? So what happens if an Sambi client has a lease and an NFS client wants to have a delegation. 23:47.000 --> 23:55.000 And there is no kind of indication. What is the server behavior? How does it look? How does it need to react to these kind of conflicts? How does it resolve them? 23:55.000 --> 24:08.000 These are all things that we as an implementation then need to define. And we need to really be very careful in particular these caching aspects. They have callback, callback spilt in. So they are calledbacks to the client. 24:08.000 --> 24:21.000 For least, there is a lease break request, lease break reply and so forth. And they are even carries specific error messages and these error messages then triggers some specific expectations and behavior on the clients. 24:21.000 --> 24:37.000 So it's a very sensitive area. And as said, there is just not one solution to kind of implement these things, but it's always the need for having a lot of testing and to really watch a monitor and understand how the clients react to specific reactions. 24:37.000 --> 24:53.000 So this is going to be a very difficult and tedious area, which we will then need to look at next, maybe not for the first prototype that we are planning to create. 24:53.000 --> 25:07.000 I right now, which is used by NFS Ganesha, to implement NFS delegations. That may be can serve as a vehicle to implement a mapping and a proper client server behavior that we have in mind. 25:07.000 --> 25:20.000 Then of course, there are big differences in how high availability failovers are dealt with with NFS. You have a handling with grace periods and S&D. You have a much more complex way of doing it. 25:20.000 --> 25:32.000 To be has some built-in ways to provide that, and there's even in the S&D 3 protocol and use service invented by Microsoft that will allow transparent failover, which is now also fully implemented in server by the way. 25:32.000 --> 25:36.000 So there's a lot gap on that side that needs to find an answer. 25:36.000 --> 25:47.000 Obviously, you have different file handle types in the S&D world, a durable resilient and persistent file handles, which don't have a similar concept on the NFS side, so that's also something to be taken care of. 25:47.000 --> 25:56.000 And then of course, a whole meal read of small details like file system attributes, case sensitive file names, and so forth permissions. 25:56.000 --> 26:05.000 So there's a lot of lists that we could easily extend to to a longer call, but there's not only the technical challenges. 26:05.000 --> 26:17.000 There are also organizational challenges, right? To quote, forker again, he wants not maybe in this exact terms, but since he said no individual can oversee the design and resolve the implementation complexity alone. 26:17.000 --> 26:36.000 So it can't just be done by one talented engineer or one talented team. It really requires a team effort of all these components to really have the same goal in mind and to work towards the same project and then also open up the server implementations so that it can be fully and rightfully implemented. 26:36.000 --> 26:50.000 And yeah, nonetheless, as often just by looking at the problem space and painting it in a bigger and bigger picture, one could easily give up and never really start, but we actually want to start and we actually did start. 26:50.000 --> 27:07.000 So the first thing that we do is to really invest into test cases and try to find simply use cases, try to find test cases that we have reproducible way to first really explore the the currently existing problems. 27:07.000 --> 27:18.000 And so then really scale down what we actually all need to do on the implementation perspective to overcome these things and to really design the solution that we have in mind. 27:18.000 --> 27:26.000 So one engineer on the sev team has looked into creating a testing framework based on Python with a defined list of use cases. 27:26.000 --> 27:40.000 So that we can explore the the actual behavior first of all to really see if looking at actual as an isolated component is maybe a good first start point to really see what all needs to be done and what is already there. 27:40.000 --> 27:53.000 And to build up and test infrastructure which is ideally then publicly available allowing us to to verify and to to to see if we're all doing the right thing in the turn of coming changes. 27:53.000 --> 28:00.000 One could also think to leverage the existing sea libraries which are present and can be used for that. 28:00.000 --> 28:10.000 We could also build up and invest more using these libraries and we need to of course design the implementations that we have in mind. 28:10.000 --> 28:20.000 But also not really kind of trying to create the super multi protocol specification and really then end up with a monster that no one will ever be able to implement. 28:20.000 --> 28:32.000 So we really want to make this very use case driven and to always motivated by what we can actually really use in the products and what are really possible use cases for this approach. 28:32.000 --> 28:39.000 So it has to be all following a certain stage implementation as a set we look at the occult first. 28:39.000 --> 28:48.000 The mandatory file locking next which was kind of a good way to probably look out the other protocol entirely so that we don't need to resolve complex overlaps. 28:48.000 --> 28:56.000 But we can just kind of if someone keeps a mandatory lock on a file then the other protocol won't be able to access it in the meantime. 28:56.000 --> 29:02.000 It's a very simplistic and simple thing to do but maybe this could be one of the first things to accomplish. 29:02.000 --> 29:10.000 Currently we have a set we have a earmarking feature that basically this allows you to deploy the both services on the same sub volume. 29:10.000 --> 29:15.000 Which is kind of our way to prevent basically running into multi protocol scenarios right now. 29:15.000 --> 29:24.000 And then of course we need to look at the much more complex word of creating a ping of pipe range locking and so forth. 29:24.000 --> 29:31.000 Again this is can then be endlessly continued and of course we want to and love to involve the community. 29:31.000 --> 29:38.000 And just because we have chosen NFS and NFS Ganesha and Samba as use based choices. 29:38.000 --> 29:42.000 This I think without really any big alternatives in the open source base. 29:42.000 --> 29:52.000 But there could be also some interest maybe from other classed file systems to to to help building up such an infrastructure and discuss with us like maybe from cluster maybe from cluster. 29:52.000 --> 30:07.000 I don't know but at least right now we are focusing all our all our things on on surface because there we can really easily bring in or easier bring in changes for the required interfaces like lip surface but. 30:07.000 --> 30:13.000 Yeah so we would love to hear more input from the community and from from others outside. 30:13.000 --> 30:25.000 And I think yeah I gave a list of some some very interesting talks the one talk that I mentioned from forker about creating this NFS and S&P common infrastructure highly recommended but there others as well. 30:25.000 --> 30:36.000 There was a good talk from Tom tell P it's last year some exp conference comparing S&P 3 and NFS 4 also in light of the history also very very good talk. 30:37.000 --> 30:45.000 There's actually one feature that we gave a presentation last year together with my colleague Patrick about case incentive. 30:45.000 --> 30:57.000 File name support in surface which could be considered as maybe one first baby step towards market protocol support because it kind of harmonizes the the excess over multiple different layers. 30:57.000 --> 31:02.000 And yeah that is all that we have thank you very much. 31:02.000 --> 31:08.000 Thank you. 31:08.000 --> 31:11.000 Sorry done. 31:11.000 --> 31:14.000 I have a question so catch you.