WEBVTT 00:00.000 --> 00:14.960 Okay, so good morning, my name is Pavel Pisa, and I have there a talk about my experience 00:14.960 --> 00:26.960 with real time and control from the Nux environment, so I will start with some introduction, 00:26.960 --> 00:34.120 and then I will follow about the history of fully-preemptive patches, and the events 00:34.120 --> 00:44.320 of prevails attempts to implement some real-time capabilities for the Linux operating systems, 00:44.320 --> 00:52.280 and then I will follow to control our network to the communication which is used in most 00:52.360 --> 00:55.680 of the cars and even industry and so on. 00:55.680 --> 01:05.720 To introduce myself, I have first met with the Unix operating system in 1988, the development 01:05.720 --> 01:15.760 workshop of the Academy of Sciences, where was a special system for the Mikrocontroller development. 01:15.760 --> 01:27.480 It was based on 68 cases, and has one half of the megabyte of RAM, and it has something 01:27.480 --> 01:35.320 like five megahart megabytes of hard drive, but I fall in the law, which is this system and 01:35.320 --> 01:44.760 with the Unix and the concept, so then when I have started the development of labor-interference 01:44.840 --> 01:55.960 and hardware for the project of my father, and then even for our own company, then I have 01:55.960 --> 02:02.520 interest in the Unix system as well, and one day my schoolmates at the university came with 02:02.520 --> 02:09.680 the set of the discets, and it was the first version of the Linux system, and we have tried 02:09.720 --> 02:18.200 to use it even for some development and increment way switch to the Unix environment, and from 02:18.200 --> 02:27.040 1996 we have done all the development of the embedded systems and so on in the Linux environment. 02:27.040 --> 02:35.160 I have started the university in 1990, there I have with my colleague, Peter Porazell, 02:35.200 --> 02:44.240 and we have started to do even hardware development and so on, and okay in 1993 I have met with 02:44.240 --> 02:51.920 the Unix and because I have interest to control of instruments through the Unix environment, 02:51.920 --> 03:02.600 I have started to look for the possibilities to use it for RS485 communication, which we use in our set-ups, 03:02.600 --> 03:10.680 and I have found the control area network and tried to implement some drivers for it, and so on, and 03:10.680 --> 03:18.280 usually even that our customer swans and the staff on Windows, we have first developed our drivers on the Linux. 03:18.280 --> 03:27.880 So this is on the right, our first communication card, which was for our open micron protocol, 03:27.960 --> 03:34.840 which is open source project from the days, and still it is available for Linux Windows, 03:34.840 --> 03:42.120 systemless environment, unfortunately, that driver is in such a day that it has never propagated 03:42.120 --> 03:49.160 to the Linux mainline, but we use it, and when we have built that communication interface, 03:49.240 --> 03:58.840 we have added on the card, even the control area network, controller, and I have started to work 03:58.840 --> 04:10.120 on the Kenbass support in end of 1990s, and then we have also our project at the university, 04:10.120 --> 04:21.800 and we have implemented the Ken drivers and can open a subsystem for Linux, which is been 04:21.800 --> 04:31.080 in use till 2020 and so on, by more companies, but in parallel, there appeared a socket 04:31.080 --> 04:38.200 Ken solution, which won the community world, so I have helped even to that project. 04:38.360 --> 04:46.920 Even that our project has not continued on the for the Linux, still our drivers are used in 04:46.920 --> 04:55.800 another real-time system, for example, one week ago, we have mainline our Ken 04:55.800 --> 05:02.920 and Ken every subsystem to the RTMS, so it still is alive, how many of you 05:03.800 --> 05:15.480 have some idea what it means to work with real-time system, and what is important for real-time 05:15.480 --> 05:24.920 system is not that throughput, but it is important to have some guarantees that you fit 05:24.920 --> 05:29.880 into the time, which is given for a given computation, so for example, if you are controlling 05:30.280 --> 05:37.720 the steering of the car, then if your computation is exactly precise, how to 05:38.600 --> 05:46.440 turn through the crossing the best way, which minimum effort and minimum effort is and so on, 05:46.440 --> 05:53.640 but the computation takes longer than is the time to react, and you hit the pedestrian or you hit 05:53.720 --> 06:00.280 the building, then it is great that you have very precise algorithm based on many layers of 06:00.280 --> 06:09.400 neural networks and so on, but it is of no sense, and it is saying for the communication, because 06:09.400 --> 06:16.440 if you find that there is a pedestrian in the street and your communication latency to the 06:16.440 --> 06:24.120 actuators is too long, then again you hit him or her and it is a problem, so for hard real-time 06:24.120 --> 06:35.560 systems, you have this fixed time, and for hard real-time, you basically should guarantee that 06:35.560 --> 06:43.720 it never happens, but it is too high guarantee because you never can declare that your system 06:43.720 --> 06:52.280 cannot be misbehaved, but there are such attempts and you need to follow some rules according 06:52.280 --> 07:01.240 to the criticality of the system, that the probability of the failure is once, 1,000 years, or 10,000 07:01.240 --> 07:07.720 years, or something like that, if it is, for example, a blow-up of the nuclear power plant, then 07:07.800 --> 07:14.680 basically you should work with those probabilities, on the other hand, if you take care about real-time 07:14.680 --> 07:22.760 for some multimeter stuff and video and so on, then it is called a soft real-time, and in this case, 07:22.760 --> 07:32.040 if you lose the frame from time to time, then basically it is making a quality of service 07:32.120 --> 07:37.560 or little wars, maybe somebody will not want to pay the full price for this, but it is not 07:37.560 --> 07:45.640 so big disaster, so we devise those two categories, those hard real-time system are used for the 07:45.640 --> 07:52.440 control systems, for the avionic, automotive, industrial, robotic and so on, so it is one of 07:52.520 --> 08:03.560 very interesting areas of real-time for multimedia and so on, and if we start with the attempts to work 08:03.560 --> 08:13.800 with real-time on the Linux system, then there has been more attempts to run real-time system 08:13.880 --> 08:21.960 in parallel to the Linux kernel, in some smaller real-time executives, which can provide 08:23.160 --> 08:30.040 remaining time to the full-featured Linux system, because usually, have relatively simple stuff which 08:30.040 --> 08:38.440 runs in some close-loop controls, which you need to run at that, with that guarantees, timing 08:38.520 --> 08:46.200 guarantees, but then you want to have the comfort on full-featured system for the graphic and 08:46.200 --> 08:52.600 for the communication and so on, so this is reason why, for example, even on Windows, there are 08:53.800 --> 09:03.400 soft PLCs, which run in the house, inter-applier of the Windows system and use the rest of the time 09:04.360 --> 09:13.480 to run the actual full-featured Windows, so there has been some similar attempts for the Linux, 09:14.520 --> 09:22.680 and there was start of the community project and effort to gather together to start to 09:23.400 --> 09:31.880 communicate and change the information, and any territory of this was Peter Wenzobler from 09:33.400 --> 09:44.040 the Center of the Transfer MicroTechnics. He was developing a micro-PS electric testing 09:44.040 --> 09:53.400 trick for the micro-multures to measure the torque, which those motors could provide, okay, this is 09:54.360 --> 10:02.040 this setup, and he has been used to use the commercial stuff at the university, it's map-lapse 10:02.120 --> 10:10.120 morning again, so on, but when he switched to the industry, he has not access to those extreme 10:10.120 --> 10:21.240 expensive environments, so he wanted to use Linux, and he now, Nicholas McGuy, who was expert in this 10:21.240 --> 10:30.280 area and worked it on that dual kernel real-time Linux system, and they decided it would be useful 10:30.360 --> 10:36.920 to gather together and to exchange the knowledge, and it was the first real-time Linux workshop, 10:37.720 --> 10:51.560 even which has been prepared this way, it was in 1999, and the other people who has participated 10:51.560 --> 10:59.320 was Victoria Reiken, who was the author of that approach of dual kernel, where we have the real-time 10:59.320 --> 11:04.920 smaller real-time executive for the real-time part, which is running in the remaining time, 11:04.920 --> 11:16.760 non-real-time Linux system, and Linux system can upload modules with real-time task to the real-time 11:16.840 --> 11:24.040 domain of the system, there has been problem with the patents, Victoria Reiken has blocked 11:24.040 --> 11:32.680 the others to do the fully open source solution, there has been alternative RTI, but due to the 11:32.680 --> 11:41.960 patents, they have to invite some bypass of the patent, so they have implemented areas as the 11:41.960 --> 11:49.800 supplier for propagating the interrupts between multiple systems, but it was the dual kernel approach, 11:50.200 --> 11:56.040 and there is interesting start of the attempt to make even the standard Linux 11:57.640 --> 12:08.200 capable to run a real-time task, and I would revine back to 2006, here, where there was 8 real-time 12:08.200 --> 12:16.920 Linux workshop in Ranju, University in China, and there was quite interesting debate between 12:17.960 --> 12:26.520 people proposing a different approach, how to run real-time application on the Linux, and 12:27.560 --> 12:34.680 this is the first time I have met with Thomas Blakesoner from Linux, and he proposed that 12:35.640 --> 12:43.640 based on the experiment of the Kansas University court system, it is possible to slowly change 12:43.640 --> 12:51.480 the main line in which the kernel would be real-time, so it was one approach, another approach 12:51.480 --> 13:01.640 which has been proposed and defended at that time by Niklas Megar from OpenTech, was that real-time 13:01.640 --> 13:09.160 Linux, there has even attempt to implement a GPL version of that commercial version of the 13:09.160 --> 13:16.680 real-time inux, by the way, the real-time inux has been later bought by Windriver company, 13:16.680 --> 13:27.240 so it is part of the software stack, so it was dual kernel approach, another option was that 13:27.320 --> 13:37.640 areas which has special subsystem for the interrupts and RTI and Xenomai, which was proposed by 13:37.640 --> 13:46.360 a group of universities from Switzerland and Italy and so on, Roberto Bacher has defended 13:46.440 --> 13:55.080 the network shop and the last one, no, not last one, second to last one, was his mother report 13:55.080 --> 14:06.840 from University of Polytechnica Revolentia, who proposed a extractum, which was high-provisors, 14:06.840 --> 14:13.640 which can run in one domain non-real-time Linux and in another domain, some smaller real-time 14:13.960 --> 14:21.240 time-posts, alternative, and the last one was Professor Hermann Hartick from the Technical University 14:21.240 --> 14:31.960 of Resden, who has suggested to use L4 fiasco, micro-chernel, and again run in one domain, full-feature 14:31.960 --> 14:39.960 Linux and in another domain, two run real-time tasks, on some small-posts, in some small-posts, 14:39.960 --> 14:47.080 environment, because usually for the actual feedback control, you do not need so much resources, 14:47.080 --> 14:53.400 usually, today you want to have something like artificial intelligence and so on, so it gets 14:53.400 --> 15:00.120 complex, but for the standard close-to-controller of some service and so on, it is simple. 15:00.120 --> 15:08.440 Okay, but we get back to that mainline Linux approach proposed as a fully preemptive kernel, 15:09.320 --> 15:17.080 solution, and what is important to understand, the real-time is not as fast as possible, 15:17.080 --> 15:25.560 so if we modify Linux kernel such way that it will be even a little slower with a little 15:25.640 --> 15:37.880 slower throughput, but it can give or provide some guarantees that it fits into the 15:39.960 --> 15:49.000 required time, for example, timing of some feedback and so on, something like when 15:49.000 --> 16:02.680 he sends a one, it is good, and okay, that attempt started as that court system, and the main idea 16:03.560 --> 16:12.280 to change the mainline kernel to the real-time operating system was that it has became already 16:12.280 --> 16:21.320 capable to run in parallel the task, and even in kernel staff on multiple processors, 16:21.320 --> 16:27.720 because Linux kernel has started to support symmetric multiprocessing in the version 16:27.720 --> 16:35.080 to zero, it works such way that there was a big kernel lock, which allows only one processor 16:35.800 --> 16:42.920 to switch to the kernel mode and other has to wait, then there was ability that when the 16:42.920 --> 16:51.320 kernel task was in the waiting for the IO, it can switch to another task, which can go into the 16:51.320 --> 17:01.960 kernel, and then slowly the locking granularity has tuned to locking only the critical sections, 17:02.040 --> 17:08.200 but still the critical sections in the kernel has not been printable, they have been 17:08.760 --> 17:19.800 has been protected by the spin locks, but that fully-printed kernel switch those spin lock 17:19.800 --> 17:27.480 apart, so critical sections to be printable with the trick that they basically provide 17:27.960 --> 17:37.480 to the model of the Linux kernel as if you have infinite number of the processors, so they 17:37.480 --> 17:43.640 abuse the ability of symmetric multiprocessing of the kernel, that they virtualized processors 17:43.640 --> 17:50.280 yet again, and basically provide for each task separate processor, okay, I am over simplifying 17:50.280 --> 17:59.320 it, but it was the main idea behind this, and this allows to slowly and gradually change the 17:59.320 --> 18:11.000 Linux kernel to have the fully-printed variant, and even that in something like 2004, there was 18:11.000 --> 18:18.600 a version of the Linux kernel, which can work with this fully-printed approach, 18:19.560 --> 18:28.920 the way to get it into the main line to about 20 years, because it has changed so many 18:28.920 --> 18:35.880 subsystems in the Linux kernel, that it was necessary, basically to gradually change one sink after 18:35.880 --> 18:42.440 another, and the Linux storehouse has not so big interest in the real time, because he has main 18:42.440 --> 18:51.880 interest in the throughput, so each gradual change has to be even a vein for the use of the main 18:51.880 --> 18:57.560 line in oops without real time, so first, purchase which get into the main line was some lock 18:57.560 --> 19:05.400 checking, which finds that even the main line kernel has bugs, which can be exposed only, for example, 19:05.400 --> 19:12.120 or three processor systems, which such system has never been built, but at that time, but that 19:12.120 --> 19:18.520 locking test from Thomas Blakeson and others has proved that there are those problems in 19:18.520 --> 19:27.400 us, start to understand that it is useful, even for main line, then real time, mutaxis, 19:27.400 --> 19:32.680 which ability of priority inheritance has been designed by Thomas Blakeson, but 19:33.240 --> 19:39.240 in us storehouse said, okay, it is so complex that I will never get it into the main line, 19:39.240 --> 19:45.480 but when Steve Rostett has written documentation for this and described how it works, 19:45.480 --> 19:53.800 then a Linux storehouse accepts this solution, and gradually we get to the situation that we get 19:54.120 --> 20:04.280 patches into the state that it can get into the main line, last blocking problem was 20:05.320 --> 20:11.560 the print key, so basically the first line of the code which Linux storehouse has written, 20:11.560 --> 20:18.440 because when you want to log in from multiple threads, without blocking it is quite complex. 20:18.440 --> 20:25.880 Okay, now I will go to very quickly through the way how to write real time tasks, so you need to 20:25.880 --> 20:34.120 log the task in the memory, you need to select the proper scheduler, there is some way how to 20:34.200 --> 20:47.800 start the task with correct scheduler, there is how to write the sampling frequency or some 20:48.920 --> 20:58.840 loop which can run at the specified time, and you should have some handling if your task is 20:59.800 --> 21:06.440 interrupted, this is a simple hardware for Raspberry Pi, which you can build with very 21:07.160 --> 21:15.480 rich three or four components, which allows to run the full servo control of the DC motors, 21:15.480 --> 21:23.560 this is simple loop which you can build to control the servo, and okay, if you want you can use 21:23.560 --> 21:29.240 even some sync more complex, so you can build real time tasks from MATLAB simooning, we have 21:29.240 --> 21:36.440 even our oven target for the Linux which I think is better than the MATLAB works, and you can 21:36.440 --> 21:43.160 control for example para kinematic, and on the same system you can run the visualization and 21:43.160 --> 21:48.280 simooning and so on, and the real time tasks which are working at something like 1 kilohertz, 21:48.280 --> 21:55.560 or we have even proof on x86 something working even at static kilohertz with no missing sample, 21:56.120 --> 22:05.240 this is FPGA approach to control a DC motor and because the MATLAB simooning is quite expensive, 22:06.440 --> 22:11.560 there is the attempt to build something open source of the Python, this is attempt of the 22:11.720 --> 22:19.160 Roberto Bachelor from Switzerland, he is picing correct system, so you can draw the control 22:19.880 --> 22:26.440 diagram in the editor and build the code and so on, but if you build the real time stuff you need 22:27.000 --> 22:34.360 latency testing and to be sure that your system behaves correctly, so you can use a secret test, 22:34.680 --> 22:41.640 it is a area where open source automation development is testing a lot of systems in the 22:43.000 --> 22:48.840 quality assurance farm for real time, and you can see that fully primitive kernel, 22:49.560 --> 22:59.480 you can get on the x86 to something like 80 microseconds, on the embedded target it is little worse, 23:00.360 --> 23:09.000 you see that they are running those long-term tests for each day in the year for many hours testing 23:09.000 --> 23:20.680 the system that there is no, no misbehaved, no sample above where the latency is longer than 23:21.480 --> 23:26.920 100 for example microsecond, and if they find that there is a problem then they can offer the 23:26.920 --> 23:36.760 service to solve this, and there is the famous real time event where the finally last year in the 23:36.760 --> 23:46.360 September the new store was accepted the packages for the fully primitive kernel for x86, 23:47.240 --> 23:54.120 risk five and 64 bit arm to the mainline, so it gradually gets to the mainline and now you have 23:54.120 --> 24:06.920 it as part of the mainline, it has been released into dot 12, into dot 13, we have given 24:07.480 --> 24:15.400 the easy prevention option to because fully primitive kernel has drawback in the throughput for 24:16.600 --> 24:24.120 non real time task, because they are switch it too quickly and too much switching has troubles with 24:24.120 --> 24:31.800 the throughput, so this is the easy prevention, we can look at zinc architecture, arm architecture 24:32.520 --> 24:39.880 we have latency profiles, we have the long-term profiles, it is run on OOS serial and we are 24:39.880 --> 24:48.680 doing a can and can have the testing at our system where we check if the gateway or some system 24:49.560 --> 24:56.600 in the can network gets a message how fast it can provide the output on the second network, 24:56.600 --> 25:04.440 so we measure this latency to propagate the message through the system and this is our set up 25:04.440 --> 25:13.320 which is running each day, those tests and okay the result are quite bad, because okay because 25:14.760 --> 25:24.760 the even that the system has very good responsibility for task, cyclic test and everything is okay 25:24.840 --> 25:32.600 when you want to run again, it is today based on the socket can and goes through the networking 25:32.600 --> 25:40.600 clear and we are fighting with the latency here and we are attempting, okay excuse me, 25:41.880 --> 25:45.720 if you're on the other side of that tape you're not visible on the camera, 25:45.800 --> 26:01.000 oh okay so it is mistake but I have feedback if I'm on the other side, okay so those are the 26:01.000 --> 26:08.120 latency profiles for the can communication and I'm now communicating in Sebastian and Mark 26:08.120 --> 26:15.560 ain't ever done that to resolve some such stuff, so it is what we have done, so I will quickly finish 26:15.560 --> 26:24.040 you can find some educational, most educational stuff for Raspberry Pi and some other systems, 26:24.040 --> 26:34.760 how to test a real-time canell and controls DC motors and permanent magnets in chronos motors 26:34.840 --> 26:40.920 and do different kinds of server controls and so on, you can find our articles and 26:43.160 --> 26:51.640 and there is the famous, you know, our publication from Thomas Glicksner about the main lining 26:51.640 --> 27:04.040 of 40 apprentice pitches and there are pointers to the, to the Linux foundation real-time project 27:04.040 --> 27:12.200 we get open source automation development labs and so on, for our stuff which we do with our 27:12.200 --> 27:20.760 students, many TZs for Linux and RTMS new ticks, we have the Picing Corarras alternative for the 27:20.760 --> 27:27.800 Matlab Simo link which is already used by some universities over the world, so it is probably all 27:27.800 --> 27:33.960 because I do not have too much time but if you want to take some reflets they are here so you can 27:33.960 --> 27:38.040 look at our project in the details, okay?