WEBVTT 00:00.000 --> 00:12.480 I'm back 20 years now, I think 10 years ago, for a long time, I've already, so what 00:12.480 --> 00:17.560 it's for those of you who don't know about it, it's a web engine that provides a I level 00:17.560 --> 00:23.040 API that allows native applications to embed the web in native main manner, and if 00:23.040 --> 00:27.880 that you can build web browsers, of course, but any kind of application where you want 00:27.880 --> 00:33.240 to load a web site on, and what things we do quite a bit is a set-up of UI as well, 00:33.240 --> 00:38.800 and use readers, you can need not many new things, and you have various layers, and 00:38.800 --> 00:45.880 at the platform level, we have ports, so we're like companies build the specific integration 00:45.880 --> 00:51.480 with their platform, so they can ship webkit in their port, in their hardware, so we have 00:51.480 --> 00:57.480 Apple projects, the Sony Pistition, so as a port, and then on the Linux, the stuff we have 00:57.480 --> 01:03.520 a GTK port, and then for embedded platforms we have, I think, on WP, and then there are other 01:03.520 --> 01:10.720 things for Windows and JavaScript only, if you only need JavaScript. 01:10.720 --> 01:16.400 So I want to address it here in the room, if you think about it, many browsers nowadays are 01:16.480 --> 01:23.400 most, even the ones that based on Chrome use the web site, so it's good, I think, to wonder 01:23.400 --> 01:30.960 how sustainable that is, is it really a situation we want to stay in, and yeah, something 01:30.960 --> 01:33.960 to think about. 01:33.960 --> 01:39.760 Of course, you might not know it, but you can already use the webbath to see in WP and 01:39.760 --> 01:45.280 what GTK, it's just not enabled by default, you have to be let yourself, and there are various 01:45.280 --> 01:52.960 reasons why we don't do enable it by default, licensing reasons with borrowing this cell, 01:52.960 --> 01:59.280 which cannot really be shipped for GP applications, the foot pit in tabozy is quite 01:59.280 --> 02:04.760 high, that's not really a big issue, but it's one, and then as we use GTK, we have to 02:04.760 --> 02:10.720 integrate with the code as an encoders, with its own integration for the code as, but for 02:10.720 --> 02:12.200 encoders, it's not supported. 02:12.200 --> 02:19.200 Right now, we come to the other way, I said, I did encoding with the webbathc, and so as I said, 02:19.200 --> 02:23.360 we have already used a multimedia framework, the webbathc itself is kind of a multimedia 02:23.360 --> 02:28.520 framework already, so it's kind of duplicating one, where we have one already, so I think 02:28.520 --> 02:34.320 it makes sense to keep the one we have already, and so yeah, let's talk about the streamer 02:34.320 --> 02:39.120 quick introduction, and the multimedia framework, actually it's more like a data processing 02:39.120 --> 02:43.640 framework, where you build graphs, pipelines, we call them, while data flows, and you 02:43.640 --> 02:48.400 have filter elements, and you can like do processing with that, it's quite cool, and it's 02:48.400 --> 02:55.600 deployed in many different kinds of situations, this space station is one, there's a 02:55.600 --> 03:01.480 robot there, using the streamer, it was also used to, in the legal project, to detect 03:01.480 --> 03:08.640 graphic signal waves, like to do the data processing of that information, and yeah, many, 03:08.640 --> 03:12.720 many plugins that are available for many platforms, and check out tomorrow there's a 03:12.720 --> 03:18.800 talk about it in the state of the union, in the open media development, so a quick example 03:18.800 --> 03:25.120 is a digital pipeline on the left, you have the file source, which we read a file, and 03:25.120 --> 03:31.360 then you can like DMACC it, if it's a case of ODG file, and then the code and render with 03:31.360 --> 03:38.240 the things on the right side, so data flows from the left to the right, and then just 03:38.240 --> 03:44.720 see what's the two things, it's the elements, like the ones I showed before, and it's also 03:44.720 --> 03:50.240 library, so the element can be added in pipelines, and then you can have, you can, you have 03:50.240 --> 03:56.600 signals and properties to use, to interact with a robot C, so for instance to do the 03:56.600 --> 04:03.720 offerings or stuff, to add ice candidates and so on, and then there's a library, or 04:03.720 --> 04:09.960 LiBGST WebRTC, which defines classes and objects API that you need in your application 04:09.960 --> 04:17.440 to interact with, with robot C, and then for the transport, we use a library called 04:17.440 --> 04:26.440 LiBNICE, that's also in charge of doing the connection establishments, so this 04:26.440 --> 04:32.440 constatus of the integration of DST WebRTC in what kids, so we have many pipelines, one of 04:32.440 --> 04:38.880 them is for capture devices, and another one is for the streaming, the encoding and streaming, 04:38.880 --> 04:44.880 and also the coding, if you have incoming tracks, and also, we demonstrated that we can 04:44.880 --> 04:52.160 do zero-copy encoding from directly from a webcam to RTP payloader, which is quite cool, 04:52.160 --> 04:57.800 and then when you receive streams, they are like, to a different pipeline, we have for 04:57.800 --> 05:02.080 playback for the video element, for instance, so we have at least three different pipelines 05:02.080 --> 05:12.160 for single-packed connection, so for string capture, we have one pipeline, as I said, and 05:12.160 --> 05:16.560 then you have to handle the permissions, for string capture, you have also to handle permissions, 05:16.560 --> 05:20.680 because most of the time you're on the Linux desktop, you have to interact with things 05:20.680 --> 05:27.080 we call the desktop portal, so we have one permission to work it, and one at the desktop 05:27.080 --> 05:32.480 level, and eventually we plan to recover it on that, and then we interact with pipelines 05:32.480 --> 05:38.000 really on Linux, and it's quite nice because we get DMBs, and we can like pass that directly 05:38.000 --> 05:45.360 to the encoder, the media streamer link, we have a custom element for that call, it's 05:45.360 --> 05:50.920 a source element, so the thing you put on the last side of a pipeline, and it can be used 05:50.920 --> 05:57.280 to interact with capture pipelines, the canvas video elements, and also incoming streams, 05:57.280 --> 06:02.880 like when you want to play an incoming stream, incoming video or use stream, and then if 06:02.880 --> 06:09.960 you think of a media stream, like it has one of multiple tracks, and at the GSM level, 06:09.960 --> 06:15.920 those correspond to the paths, the things we use to connect elements together, and that's 06:15.920 --> 06:23.880 also our width pass data between elements, so yeah, but really simple example is stream 06:23.880 --> 06:27.680 simplified, because the pipeline can be really big, I wouldn't fit on one slide, so I 06:27.680 --> 06:33.440 should simplify a few things, on the last side I have a source element, which is doing 06:33.440 --> 06:39.520 capture on the capture device, then you can encode and then RTP payload, and then you pass 06:39.520 --> 06:46.040 that to our WebATCB, which does a lot of things, and it's sending data over the network, 06:46.040 --> 06:51.200 and then if you receive an incoming track, you have a source pad that was created by WebATCB, 06:51.200 --> 06:58.960 and then we can be notified of that pad and connect to a media player, so yeah, the 06:58.960 --> 07:05.400 per connection WebATCB was designed to follow the spec, so if you know WebATCB, you can 07:05.440 --> 07:11.560 quite easily use WebATCB already, you have signals corresponding to all the different steps 07:11.560 --> 07:18.840 of the spec, and yeah, it was quite natural to interact into WebKit, we have some issues, 07:18.840 --> 07:27.840 but in general it's quite easy to have a basic prototype, so it's just a precise interface 07:27.840 --> 07:35.880 very, very, quite well, if WebKit's promised asynchronous API, so for instance, on the 07:35.880 --> 07:46.280 JavaScript side, if you want to connect to a video element, you would say videorelement.source, 07:46.280 --> 07:53.400 and then in WebKit what it does is that you would create a media player, and then use the 07:53.400 --> 07:57.680 media stream source element that I introduced a couple of slides before, and connect 07:57.680 --> 08:01.920 that into the pipeline, and the data would flow naturally to the decodes, and to the 08:01.920 --> 08:10.440 sync. Then more things like statistics, data, and all this is already supported by 08:10.440 --> 08:16.320 a GST WebATC, but the statistics, since WebATCB normally knows about the streaming, and 08:16.320 --> 08:22.080 doesn't know about the encoder or decoder itself, we have to augment the statistics 08:22.080 --> 08:30.040 with information we have in our pipeline, so there's a few a bit more work to do on that, 08:30.040 --> 08:34.160 and then the data challenge was so quite easy to interact, because the APIs like one-to-one 08:34.160 --> 08:41.480 mapping, and then it was really nice to interact with. There are more things, those two things 08:41.480 --> 08:46.920 are not implemented by GST WebATC, we have to do it in WebKit, we play strike, so we're 08:46.920 --> 08:53.120 like swish between one media stream track to another, that's basically like just using 08:53.120 --> 09:00.080 the pipeline manipulation, basically, and then we have to add support for all legacy offer 09:00.080 --> 09:04.140 options, because we had some application that needed it, I will talk about that in a few 09:04.140 --> 09:06.140 slides. 09:06.140 --> 09:14.360 So yeah, the use case is obvious ones, video conferencing, we are a little bit there yet, 09:14.360 --> 09:18.520 we have some basic code types for our couple platforms, but they are not fully functional 09:18.520 --> 09:19.520 yet. 09:19.520 --> 09:26.800 So we did some work on GST on LifeKit, yeah, still work to do there in that, in the 09:26.800 --> 09:27.800 phones. 09:27.800 --> 09:33.120 But I want to talk about a couple more things, so CloudGaining is one really interesting 09:33.120 --> 09:39.160 use case, we've been working on, we got Amazon Luna to work on setto boxes, it's related 09:39.160 --> 09:46.160 with our web decoders, and you can also, if you use Amazon Luna, you can stream to Twitch, 09:46.160 --> 09:51.320 your work can, it's not really supported yet by our reports, but eventually we will get 09:51.320 --> 09:52.320 that working. 09:52.320 --> 09:57.600 I had a demo, but maybe I don't think I will have time, maybe I will add in a real 09:57.600 --> 09:59.880 short. 09:59.880 --> 10:06.040 Another use case we worked on is pixel streaming, we had a contract with a company called 10:06.040 --> 10:11.680 Zoo, they have an open source code application called Kitikad, and we made it working 10:11.680 --> 10:17.720 in Linux with our Kitikad, we fixed a copy of things in work kit for that, and it was 10:17.720 --> 10:24.960 really interesting, I also demo, but maybe later, it was quite interesting because I used 10:24.960 --> 10:29.960 data channel for the pointer events, for the integration with the model, and I thought 10:29.960 --> 10:35.560 it was an interesting use case for web artists. 10:35.560 --> 10:40.660 I'm going work with quite a few things, we want to focus a lot on DevTools, because as you 10:40.660 --> 10:46.600 might know, in work kit and in Safari, the DevTools are not as good as in Chrome, so people 10:46.600 --> 10:49.960 have a bit of time that, maybe you know, this is a feature you have been expecting on 10:49.960 --> 10:55.520 the web art catalogs, it's really, I don't think it's many people know about it, but 10:55.520 --> 11:03.440 it's there, it's really basic, so we want to build up on that, we have made a back-end, 11:03.440 --> 11:10.880 we call it DevTools back-end, which like when you enable it, it will emit a JSON stream, 11:10.880 --> 11:18.400 you can do live-grafing or post-motaman analysis of your web art selection with that, for 11:18.400 --> 11:23.400 now it's enabled with an environment viable, and I have built a basic, really basic 11:23.400 --> 11:31.680 content for that, you might recognize the graph from Chrome, so I reuse that to make 11:31.680 --> 11:36.760 the graphs, so that's currently like a third-party project, they don't really would like 11:36.760 --> 11:45.640 to have it integrated in WebKit itself using a custom UI protocol, or maybe in Inspector. 11:45.640 --> 11:51.480 Now things we work on currently, network collision simulations, so we want to improve our 11:51.480 --> 11:59.480 packet of story, like fake RTX and these things, so being able to simulate conditions 11:59.480 --> 12:05.000 is really nice, and we added that support recently, we have the basic infrastructure for 12:05.000 --> 12:12.520 similar cast, as we see, like VP8, Templars, Calabity Works, but it's not working yet for VP9 12:12.520 --> 12:19.720 on V1, so some work we've been needed in GSMXF for that, after encoding we made some early 12:19.720 --> 12:25.160 experiments already on Raspberry Pi and on desktop, and we've got it working, but it's not 12:25.160 --> 12:31.320 free of string yet, and then there's other things such as entering encryption, yeah, the story 12:31.320 --> 12:43.480 there, we should implement that because you can encrypt in various ways, yeah, we need to work 12:43.480 --> 12:51.240 out on the implementation details, we're getting that. We want to improve, so I understand 12:51.240 --> 12:56.920 boxing story, because WebKit is splitting various processes for secretary reasons, right now we have 12:56.920 --> 13:00.520 everything working in the web process, which is not really a good for security, like the 13:00.520 --> 13:06.840 catcher device success is done for the web process, yeah, eventually we want to improve on that, 13:06.840 --> 13:12.840 and then the platform we will have to continue with that, but on the desktop, we have to use the 13:12.920 --> 13:20.280 we could use a portal called the camera desktop, camera portal sorry, yeah, and again we got 13:20.280 --> 13:27.240 really guarding sandboxing, right now we also have to like disable the sandbox in the web process 13:27.240 --> 13:34.280 to be able to set up a peer connection, because we don't have the UDP, the RTP traffic rooted to the 13:34.280 --> 13:42.920 network process, so yeah, it's still some work we plan to do with a new ice implementation called 13:42.920 --> 13:49.800 device, which is written in Rust, and it's quite cool projects, and eventually we plan to 13:49.800 --> 13:57.960 integrate the new project into a kit as a ice transport backend, yeah, that's it, that's it, that's 13:58.760 --> 14:00.760 thank you, thank you. 14:18.920 --> 14:20.600 sorry 14:20.600 --> 14:34.320 So the question is about RTCP feedback, yeah, if we support that or not, yeah, I think we, yeah, it's 14:34.320 --> 14:42.240 supported in about Sibinia, yeah, the RTP stack in just to me is quite complete, it's really awesome, 14:42.240 --> 14:50.240 there's many things there supported and it works quite well as well as so, yeah, yes. 14:50.240 --> 15:00.360 I do, I do explore perhaps using the higher level elements with RTCS or Sibinia from the 15:00.360 --> 15:04.800 rocks plug-ins, because that came after you started the project, yeah.