WEBVTT 00:00.000 --> 00:14.640 So, let's start it, so we'll have Augustine talking to us about accelerating 00:14.640 --> 00:19.640 robotic development for simulation, and I am personally very interested about that, so 00:19.640 --> 00:20.640 that's going to be awesome. 00:20.720 --> 00:21.840 No pressure, never. 00:21.840 --> 00:32.960 Hello everyone, well, thank you very much, just edit a bit, I'm working at 00:32.960 --> 00:37.920 correct human consulting films, focus on robotics, and other topics based in Argentina, 00:37.920 --> 00:42.480 but we also have a pretty much distributed team, and this is super helpful for the topic of this 00:42.480 --> 00:43.480 talk. 00:43.480 --> 00:50.080 Some of the projects that we have are nano and educational robots that we are going to use 00:50.160 --> 00:55.200 as a target robot for the top, Beluga, and Monte Carlo localization package, check it out, 00:55.200 --> 00:57.360 super cool. 00:57.360 --> 01:03.360 This is my colleague, Hiku and make it, I'm helping him with the talk, he's the simulation 01:03.360 --> 01:10.320 lead, I take him in, this is me, if you don't recognize me, here I am, you can talk to me 01:10.320 --> 01:16.560 through social media or after this talk, and this is our journey today. 01:16.560 --> 01:24.720 So, who has ever used the simulation engine for robotics, is your hands, awesome, 01:24.720 --> 01:28.560 who has ever used a robot without the simulation engine? 01:28.560 --> 01:35.680 Nice, okay, so let's talk a little bit about robots and simulators, this is a super 01:35.680 --> 01:44.000 simplified scheme of what a robot is, right, so we have a hardware platform, software platform, 01:44.000 --> 01:48.080 if we think of the hardware platform, we have sensor actuators and a compute engine, 01:48.880 --> 01:53.760 we think of a robot suffers structure, we have an operating system driver, 01:53.760 --> 01:59.360 application and system libraries, and then we have Taiga, and the robot is interacting with an 01:59.360 --> 02:06.880 environment, the physical world, surrounding it, people, things, this is a super cool example, 02:06.880 --> 02:12.000 here I am, I'm part of an environment, but this robot may also be interacting with a 02:12.080 --> 02:19.680 combo, sorry, of more compute, services, other systems running a like a robot, so we need to account 02:19.680 --> 02:26.880 for that one building robot. Let's talk about the simulator, what's that? Well, a simulator is an 02:26.880 --> 02:34.160 application that runs on a scenario, right, and tries to follow a given model, and a scenario 02:34.160 --> 02:41.600 is a representation that they both do at a time, and it has a lot of systems on it, and the key thing 02:41.600 --> 02:48.720 is that it's not static, right, it evolves, what is the model, a model is an error representation 02:48.720 --> 02:56.880 of the rules that governs a process, so if we think of robotic simulators, it's an application 02:56.880 --> 03:02.080 which allows to model the scenarios with robotic systems, it seems to be pretty straightforward, 03:02.080 --> 03:07.520 so far, right, but when we think of an architecture, this is a super simplified one, and it 03:07.520 --> 03:14.560 becomes to add more and more complexity, why? Because we need logging, every simulator has logging, 03:15.280 --> 03:22.720 we need a physics engine to evolve the models, and we also need rendering, so, closing the 03:22.720 --> 03:27.680 limit, so we need a rendering engine to get the visuals to process the sensors that need to capture 03:27.680 --> 03:34.480 environment, that's super important. All these three things alone, they don't do anything, so we need 03:34.480 --> 03:38.800 to put everything that glutes, glutes, all the systems together, and that's the causing, 03:38.800 --> 03:45.040 that's a main responsibility. But then we need to interface with outside world of the simulator, 03:45.040 --> 03:50.240 those are interface APIs, there may be many like the communication system that interacts with 03:50.240 --> 03:56.800 the external world, or direct clients that get into the simulator itself, and they want to talk 03:56.800 --> 04:02.400 with the robotic software stuck directly, or any third-party external services, even to control 04:02.480 --> 04:10.000 the simulator. Maybe we want to step the simulator just once, at the time, or let it flow. 04:12.800 --> 04:18.320 With this, we get the basics of the simulator, and then we may get a target for an application, 04:18.320 --> 04:28.240 or get a target for a testing environment. So, let's recap on the robot diagram and think of 04:28.320 --> 04:33.840 what are we going to replace here with the simulator? Probably we want to replace a hardware 04:33.840 --> 04:39.280 and environment. And this is how we achieve a robotic simulator, right? We get the simulator 04:39.280 --> 04:47.680 in the middle and interacts with the onboard software platform. So, this huge blue block here thinks, 04:47.680 --> 04:53.200 but there's a real hardware under the hood, but actually it's similar, it's similar, it's 04:53.200 --> 05:01.200 running on a computer platform. It can directly use or access the external of board services, 05:01.200 --> 05:06.240 and your board service is also can interact directly to the simulator. So, far so good, 05:08.080 --> 05:15.920 excellent. So, this is an A-R for robotic arm that we build in the company. 05:17.120 --> 05:22.640 It's open hardware, open source, thermal, open source controllers, and right now we are using 05:22.640 --> 05:27.520 move it, with a motion, which is a motion planner through service, in the middle, and we have a simulation 05:27.520 --> 05:33.040 in Gazevo with it, right? What can we do with the simulator? It's not because of Gazevo. 05:35.600 --> 05:41.040 But we can do a lot of things. We can model the actual physical thing that we have, we can test it, 05:41.040 --> 05:46.080 we can validate it, learn and tune parameters, let's say that we want to go faster to achieve the 05:46.080 --> 05:52.720 role. We need to modify the controllers and we can learn that in simulation. We can generate data, 05:52.720 --> 05:59.120 so as to later on get closer to the real thing. We can train, 06:01.040 --> 06:07.840 valid data and train an A-R, or M-O-D controller, and we can even solve, right, because we may not 06:07.840 --> 06:12.320 have the final configuration of your robotic system that is going to be deployed to your customers, 06:12.320 --> 06:17.840 but maybe we can use this in order to let your customers know how it would look like, 06:17.840 --> 06:25.760 how it would proceed on the real scenario. So, this comes to a bit problem, 06:25.760 --> 06:32.880 like, how do I find the right tool for the right job? There are many, many possibilities in the market. 06:34.480 --> 06:39.760 So, what we want to present today here, it's like a set of questions that would guide 06:40.480 --> 06:45.600 hopefully objectively towards that conclusion. So, let's start with a rename. 06:46.960 --> 06:52.160 You have to identify use cases. So, it's not the same if you are going to use it on a recurrent 06:52.160 --> 06:56.800 manner for testing purposes or for validation purposes. It might look like the same, 06:56.800 --> 07:05.200 actually it's not. Imagine that you have a team of 50 developers with five products on the market, 07:05.200 --> 07:12.240 and you want to have that on a CI, and you have, I don't know, 50 PRs per way, 07:13.200 --> 07:21.680 HPR gets five commits until it gets merged. So, you have like 250 commits, and you want to run 07:21.680 --> 07:27.440 a simulation, test suite for that. You need to think of a compute power for that. That's a lot of 07:27.440 --> 07:34.240 cost that you need to put into your infrastructure team, so as to get it. Fast, efficient, 07:34.400 --> 07:39.280 and achieve the goal. But if you want to just validate things before release, you've made a 07:39.280 --> 07:44.960 wrong of QA. Probably it's not going to be that exacerbated the number of times that you're 07:44.960 --> 07:53.200 going to run. Is it going to be a development tool that do your workforce need actually 07:53.200 --> 07:59.440 an actual simulator because they don't have access to automation software example? Well, that's 07:59.520 --> 08:07.040 super important to know. The kind of requirements that the development tool has is completely different 08:07.920 --> 08:13.120 from the one that you have in CI, right? Hardware in the loop. Do you know what hardware in the 08:13.120 --> 08:20.960 loop is? Is it fun? Okay, pretty much. Everyone, so for those that don't know what hardware in the 08:20.960 --> 08:28.320 loop is. Imagine that you have this Raspberry Pi, and you connect it to a laptop, you run the simulation 08:28.320 --> 08:33.680 here, but the full software stack is in the Raspberry Pi, so you can run simulations against it, 08:33.680 --> 08:39.200 and you don't have to have this all-interview full of people to actually check that it can avoid 08:39.200 --> 08:48.320 a crash with them. Ground, aerial, vehicle, there are many robotic systems are super 08:48.320 --> 08:53.520 indigenous. Identifying your use case is super important. If you have robotic arms, the type of 08:54.480 --> 08:59.920 simulation that you need is completely different to the ones that you have on mobile robotics. For 08:59.920 --> 09:07.600 example, there are other physical features that you need to account for. When talking all the scenarios 09:07.600 --> 09:13.760 and thinking of the scenes, you have indoor and outdoor or mixed environments, which will affect 09:13.760 --> 09:19.760 differently the sensors and actuators there, you may have one of multiple robots, and that's a 09:19.760 --> 09:25.680 completely different thing. Imagine that you want to make tests on a bit of 50 robots. You don't 09:25.680 --> 09:34.400 care how much one will perhaps is turning on each prototype. Maybe you want to see if the system 09:34.400 --> 09:40.400 as a whole is behaving as you expected, right? So it's a completely different thing. Do you have 09:40.400 --> 09:45.440 automatisms? Do you know the different videos between an automatism and an autonomous system? 09:45.520 --> 09:52.080 Raise your hands. Okay, go into that. An automatism will decay. The deterministic elite 09:52.080 --> 09:58.000 an autonomous system may be exposed to new situations and behave differently, or it's not such 09:58.720 --> 10:06.320 deterministic system. So in case of an automatism, you may lower a bit, there are on the simulation 10:06.320 --> 10:12.880 engine required for that because the number of constraints to the system is much bigger. You know 10:12.880 --> 10:20.160 how to control the variables, so you may not need as many requirements on the simulation engine. 10:22.240 --> 10:26.160 Do a scenarios last long. You have simulations that are running for hours, 10:26.800 --> 10:32.800 pace, or is it just a few seconds that you need to actually validate that your system is behaving as 10:32.800 --> 10:39.040 expected. The unique determinism that goes back to the previous question, 10:39.920 --> 10:45.200 sing creation tools. This is a super interesting topic. So imagine the robot, the aroma that we have 10:45.200 --> 10:52.160 there, or the vacuum cleaner, you know, any specific robot, is it required to actually vacuum clean 10:52.720 --> 10:58.640 every single living room on planet Earth? So you need to actually account for different geometries. 10:58.640 --> 11:05.200 If you have a carpet, if you have a wooden floor, that requires some kind of random 11:05.200 --> 11:11.120 generation of homes to actually validate that your system can behave properly on all those different 11:11.120 --> 11:18.880 scenarios. And did you doubt that it unlikely? Maybe you have a super nice coach that you want to 11:18.880 --> 11:25.600 actually detect, or you have, I don't know, a set of spare cables on the floor, and you don't want 11:25.600 --> 11:32.960 your vacuum cleaner to go over it. You need to detect that, and that's certainly not a tool, 11:32.960 --> 11:39.680 or sorry, a task for a roboticist, maybe it's something that you need to get away to a graphic designer. 11:40.960 --> 11:46.720 And the list keeps growing to be honest, this is a never-ending list. So if we get into the physical 11:46.720 --> 11:52.960 requirements, it's kinematic simulation enough for your use case. Are you going to be able to speed up 11:53.520 --> 11:58.240 your simulation by doing that, or do you actually care about dynamics of the system? Sometimes you 11:58.320 --> 12:04.320 need to deal with complex dynamic systems, and there's more pressure on the physics engine. 12:05.840 --> 12:12.480 Since there are many models for different types, maybe some are simplified, maybe you need to 12:12.480 --> 12:17.360 get more resolution and imagine that you have a rotary lighter, and you care about how the point 12:17.360 --> 12:24.800 cloud that you generate moves with your robots, so actually it's not static. That's something 12:25.040 --> 12:31.840 really hard to get, and you need to get a super custom model for it. It's CPU power enough, 12:31.840 --> 12:37.360 maybe you have to delegate that to the GPU, and that's not simple. More on that later on, probably. 12:38.800 --> 12:43.760 Rendering requirements, do you need photo-rallism? Maybe you don't care about photo-rallism? 12:43.760 --> 12:52.480 Why are we using a super expensive simulator for that? Then, mind-tenance model. So when getting 12:52.480 --> 12:59.920 up to this point, you think, okay, I have two or three options out there, but I'm going to rely 12:59.920 --> 13:05.760 on the community to get upgrades or fixes on my system. It's my system right now. I don't know. 13:05.760 --> 13:11.200 I'm going to 18, and my simulator needs to run and I want to 24, who's going to do the 13:11.200 --> 13:18.320 party with it. That's a lot of burden on the development teams to take care of, and you have to 13:18.320 --> 13:28.240 think ahead of that. Otherwise, it's going to be a problem. So what did we do? We got a lot of 13:28.240 --> 13:34.720 these analysis many times, right? Because on every new custom run, every new project, we have 13:34.720 --> 13:39.680 two answer all these questions, and doing that for all the option available out there, what's 13:39.680 --> 13:47.520 a really complicated task. So we kind of model, and you know, which has this structure, as I 13:47.520 --> 13:54.640 mentioned before, and it's composed with a 2D LiDAR, actually the camera, IAMU, actuators, 13:54.640 --> 14:02.800 which are models, motors, wheeling coders, frames, battery, etc. This is pretty much what every 14:02.800 --> 14:08.800 more robot out there has, probably you may have multiple cameras on it, or maybe you have a 3D 14:08.800 --> 14:16.480 LiDAR, maybe you have multiple LiAMU, maybe you lack of wheeling coders or it happens. 14:18.160 --> 14:23.280 This is a Diff drive robot. You may have an acarmonic configuration, you may have a tricycle, 14:25.280 --> 14:32.160 a non-nidrive, but still it's one configuration that helps you to benchmark, but actually to evaluate 14:32.160 --> 14:41.920 different simulators. Super simple, everyone understands this, and then we model all those things 14:42.880 --> 14:50.080 for all of these simulators. So we got a COW without so-vide, Mujogo, and Flatland. There are many 14:50.080 --> 14:58.720 others, probably not suitable for this kind of conference, there may not be open source, so we have 14:58.720 --> 15:04.560 leading those outside of discussion, but it's been super useful to actually understand what 15:04.560 --> 15:12.000 can we do with each one. I'm not going to get into the details of everyone, but all these things are 15:12.800 --> 15:18.960 in public repositories, I have only videos, I want to emphasize some strengths of each of those, 15:18.960 --> 15:28.560 and how we use them. You can scan now, you will be directed to a link free, a good manual 15:28.560 --> 15:36.000 way, if you're organization, you will find all the Andino underscored something there, and you will 15:36.000 --> 15:44.960 find other projects. It's important that you also start them. I'm looking at you, so. All right, 15:44.960 --> 15:54.960 so, if you download, let's start with Casivo. 16:14.960 --> 16:39.760 There we are. How it's very just in case. So we have over there, Casivo for this simulation, 16:39.760 --> 16:48.320 it's in our HQ office, and it's running to land on it, so it will be mapping the entire office. 16:49.680 --> 16:55.840 This simulation is running four times, if I remember correctly, and real time, it's a super small 16:55.840 --> 17:04.720 robot like this, and it's mapping a big area. Casivo is a general purpose simulation, 17:05.680 --> 17:13.040 simulator, sorry. Right now, it's using Dart as it's physics engine and ORE2, which comes from 17:13.840 --> 17:20.160 the gaming, perfect thing here. The gaming industry, as a rendering engine, the cool thing 17:20.160 --> 17:25.760 out is that Casivo has a super modular structure, so you need to change for example the physical engine, 17:25.760 --> 17:31.840 there's another one that is much faster right now that you can use. It's not fully implementing 17:31.920 --> 17:35.760 in terms of all the features that you might get, but you might speed up simulations that don't 17:35.760 --> 17:45.680 require such a detail on the physics engine. It might be the full choice, and not a bad one, 17:45.680 --> 17:52.400 per se, when you are starting a simulation project, but sometimes in terms of rendering, it falls short 17:52.400 --> 18:00.400 to be honest. We may need other things. Cas, a lot of examples, a lot of sensor support, so if 18:00.400 --> 18:04.720 much in that you want to have a segmentation model, a ground truth, a segmentation model, you can just 18:04.720 --> 18:09.360 use the one that is out there, you don't have to implement something at all to it, and that happens 18:09.360 --> 18:18.000 with many other things. It works with a aerial robotic model, as well as with robotic arms and 18:18.080 --> 18:31.120 wippers, so it's a great choice. Let's see if we can use another one. So the other one that I wanted 18:31.120 --> 18:42.000 to present is wewots, let's see if it loads, otherwise I'm going to switch here, now it's not working. 18:42.000 --> 19:07.120 So, wewots. Now I just mine now. There we are. So, wewots is solved for a general purpose simulation 19:07.360 --> 19:16.240 engine. It has a monolithic structure with respect to a sieve, it's not full-blown of components, 19:16.240 --> 19:21.360 so it might be better if you are looking for something that is lighter. It has a lot of 19:22.400 --> 19:28.560 planes as well, and support for different types of robots. It's been used also in the AV industry, 19:28.560 --> 19:33.440 and it's a great tool. In this case, we are only celebrating the robot. So on the right side, 19:33.440 --> 19:37.920 we can see all the information that we are extracting from the simulation engine, and we also have 19:37.920 --> 19:46.160 on the left side, the camera feed. It's relying on ODE for the physics engine, and open GL for the 19:46.160 --> 19:53.440 brandening. It's great. It has a lot of bindings, so you don't have to be proficient in c++, 19:53.440 --> 19:59.680 you can use other languages to use it, come into the AV and so on. It's simple, a great choice as 19:59.680 --> 20:09.840 well. So now, I'm going to switch to this one and speak briefly because we have an R section 20:09.840 --> 20:21.840 with Jan. How about this is ODE, short video, great graphics. We can get everything will learn more 20:21.920 --> 20:30.560 about it later on in the next stop. This is in Java. I don't know if you know it, but this is great 20:30.560 --> 20:37.280 for gathering, like very fine rain, physical data, like you can get a lot of different models, 20:37.280 --> 20:43.760 integrators, physics engines, and so on. We are using it within our R and D team, so as to tune 20:43.760 --> 20:51.120 the process of getting a great model, great physical model of a robot, and then try to aid 20:51.120 --> 20:57.840 the development with that, get some structure on how to do it. It's out there as well, 20:58.640 --> 21:05.440 different use case, not great graphics, certainly. And this is flat land. I really like this video. 21:05.440 --> 21:13.200 So in this one, we get a lot of robots moving around. This is using a 2D physical engine. 21:14.160 --> 21:20.080 We are running about 20 robots in here. Of course, we are not getting the camera, 21:20.240 --> 21:28.320 rendering for them, but we can achieve RTFs real time factors up to 20 even more with hundreds 21:28.320 --> 21:41.120 of agents. So this is great to look at other use cases. So with that, this is the A4. 21:42.080 --> 21:46.800 I think I finished. Thank you very much. I'm not sure if you can. 21:54.160 --> 21:55.360 Question. 22:02.640 --> 22:04.640 Hello. Thank you very nice for speech. 22:05.600 --> 22:11.520 I wonder, how do you handle misalignments between the simulation and the real world? 22:11.520 --> 22:15.280 In general. What are your techniques? Misalignments between? 22:15.280 --> 22:22.000 Between simulation and real world. Great question. So if it depends. 22:25.920 --> 22:32.560 So the question was, how do we handle misalignments between the simulation and the real robot? 22:33.120 --> 22:39.120 It depends on what you want to actually do with it. Sometimes you don't care about 22:39.120 --> 22:46.960 the misalignments and it's more expensive to actually try to develop something that matches real life. 22:48.400 --> 22:55.120 So we first analyze what we want to do. That's the first thing. Sometimes you don't care. 22:55.120 --> 23:04.160 When you care, you go back to the list of questions and we see a limitation on the simulation 23:04.160 --> 23:12.880 or the physics extension. Maybe the problem is that the simulator is suffering you and not what 23:12.880 --> 23:20.240 the physics extension is actually offering. So you need to go over that gap. Maybe you are not doing 23:20.240 --> 23:26.400 a proper model, offering like let's say you have got zero, right? In a zero you have a 23:26.400 --> 23:33.040 description model, which is SDS, a specific format and you cannot account for all that. 23:35.520 --> 23:39.600 So in those cases what you end up doing is changing the simulator to cover that case. 23:41.040 --> 23:48.320 We have done regressions. So what do you want to do?