WEBVTT 00:00.000 --> 00:13.000 And I'm giving this presentation with Eric, and we're going to talk about Puma, which is our tool to automate 00:13.000 --> 00:17.800 abactions in Android. 00:17.800 --> 00:23.000 We work at the Netherlands Forensic Institute, located in the Netherlands in the Hague, and we're part 00:23.000 --> 00:30.200 of the Ministry of Justice, and within the Institute we conduct forensic research in various 00:30.200 --> 00:31.200 disciplines. 00:31.200 --> 00:38.400 So, for instance DNA research, all the motive, and also digital forensics, and we're in the 00:38.400 --> 00:46.360 landscape department, and we write parsers for file formats. 00:46.360 --> 00:52.600 And in our work, we came across the challenge of generating reference data for our parsers 00:52.600 --> 01:00.080 to test them, and manually it's a big challenge, and we're moving towards automating 01:00.080 --> 01:01.080 it. 01:01.080 --> 01:03.080 So, what is Hanskun? 01:03.080 --> 01:07.600 It's good to note that Hanskun is not open source. 01:07.600 --> 01:18.440 Yes, Hanskun is our digital forensic tool to investigate digital traces, and Hanskun 01:18.440 --> 01:24.560 can process multiple devices at the same time, multiple types of devices, such as phones, 01:24.560 --> 01:26.760 and laptops. 01:26.760 --> 01:33.080 And Hanskun parsers, the data from these devices, and also the output of other forensic 01:33.080 --> 01:36.920 tools, like cellar bright. 01:36.920 --> 01:47.400 Hanskun parsers this, the data and converts it to traces like contacts and locations, messages, 01:47.400 --> 01:53.560 and presented for instance to the users in a searchable way. 01:53.560 --> 02:00.120 Hanskun's used nationally and internationally, and it's used in big cases, so we support a 02:00.120 --> 02:10.640 lot of formats, and we need to make sure to keep up-to-date with updates of those formats, 02:10.640 --> 02:16.520 and we need to validate if Hanskun parsers them correctly. 02:16.520 --> 02:21.960 So that's why we make reference data. 02:21.960 --> 02:27.240 We need to verify if our parsers work as we expect, so we can both generate a data ourselves 02:27.240 --> 02:32.480 or acquire it from the internet or from other people, and we actually prefer to generate 02:32.480 --> 02:33.480 it ourselves. 02:33.480 --> 02:39.960 The reason for this is the ground truth, we want to know exactly what is in it, because 02:39.960 --> 02:48.520 only then we know what should come out of it, we want it to be complete, so if we generate 02:48.520 --> 02:59.160 reference data for applications, we want all features to be represented in it, so like 02:59.160 --> 03:08.120 what's up, we want to send a message of photo, etc, and we need to comply with GDPR, we want 03:08.200 --> 03:11.960 to make sure there's no personal information in there. 03:11.960 --> 03:19.080 Why wouldn't we generate it ourselves in practice, we do a combination of the two, because 03:19.080 --> 03:24.440 we cannot foresee all edge cases, and real-life data, we always come across some situations 03:24.440 --> 03:30.640 we did not think of ourselves, it's not real-life, so what's that data base of a victim 03:30.720 --> 03:37.360 spends over like 15 years, and it does not really compare with the database we made in one day, 03:39.120 --> 03:44.480 and it's very laborious, it's not scalable, so it takes as much time to do it a second time, 03:44.480 --> 03:51.040 it's the first time, and app updates are a real problem, we're to illustrate this, we have 03:51.120 --> 03:57.120 our fictional owl app, let's say version one comes out, we support it in Honske, 03:59.040 --> 04:04.000 and then an update comes out, version two, and it breaks Honske, the database scheme, 04:04.000 --> 04:08.560 and I'd be updated, and Honske does not expect this, so what happens then, 04:09.680 --> 04:14.480 well, nothing, because we don't know about it, at least not until some time, 04:15.040 --> 04:22.880 then there's a crime, the police sees the device of the victim, version two is on it, 04:23.600 --> 04:28.880 that version three comes out, version four comes out, and it's still broken, and then 04:31.040 --> 04:36.960 it's noticed by an investigation start, and it's noticed, and the back is felt, and of course we 04:37.040 --> 04:45.840 will try to fix it as soon as possible, so we do this for version four, because that's the 04:45.840 --> 04:51.360 version we can get from the place to all right, we cannot really go back and test all versions, 04:52.800 --> 04:58.000 so we still don't know if version two and three are supported, because there might have been 04:58.000 --> 05:05.040 another change in the meantime, and then version five comes out, and we might support it, 05:05.040 --> 05:13.040 but we don't know, applications are very volatile, because for what's up, we know from our own data 05:13.040 --> 05:20.320 that two versions a week come out on average, and we can just not keep up with that, 05:21.440 --> 05:25.760 and the old version might not be available anymore, and even if it's available, if we save the 05:25.760 --> 05:34.400 APK, the server might force us to update to the newest version, so I think I've made my case that we 05:34.480 --> 05:42.320 really need to automate this, so that's what we did with Puma, this is open source, we found out 05:42.320 --> 05:46.080 we talked to people, we find out that a lot of people in the forensic stone main, 05:48.400 --> 05:55.360 yeah, run into this problem, we also by version of APK on it, and Puma aims to 05:55.680 --> 06:05.120 do to automate writing these scenarios before we did it by hand, with all scenarios who we knew exactly 06:05.120 --> 06:11.680 when we sent a message, now we can just do it automatically, it's written in Python, so it's easy 06:11.680 --> 06:19.920 to integrate, and it's built on top of APK, APK is a testing framework, it's used by the developers, 06:20.320 --> 06:27.440 and we use it to automate the actions, so this is basically how it works, Puma talks to the 06:27.440 --> 06:34.240 APK server, APK installs an application on the device, which hooks into the accessibility framework, 06:34.240 --> 06:48.560 and that actually runs the actions, so bit more on APK, it works with production code, which is really nice, 06:49.920 --> 06:59.280 you don't need root access, and it works both on Android and iOS, and with APK you can see everything 06:59.280 --> 07:06.800 that's on the screen, you can interact with elements, so tap an app to open it or send the send button 07:06.800 --> 07:12.240 to send a message and perform actions like tilting the screen and spoofing a location, 07:13.200 --> 07:19.840 so if we have APK and why would we need to, why would we need Puma, this is APK code 07:20.800 --> 07:26.560 to just send one telegram message, so it's several steps, we need to open the app, 07:26.560 --> 07:32.400 then open the chat with Alice, insert the message into the text box, which we first need to click 07:32.400 --> 07:36.640 and then click the send button, and this is fine for just sending one message, but if you have 07:36.720 --> 07:43.680 all scenario, it's too for both, and it's not really readable, so if Puma, you might actually 07:43.680 --> 07:51.120 wraps around this, and it's just one line, it's one initialization and one line, so you can just say 07:51.120 --> 08:01.840 Alice send message to Bob, so the features Puma now only works on Android, but we aim to also 08:01.840 --> 08:08.480 support Alice, you can use multiple applications so you can install an applications and messages 08:08.480 --> 08:19.120 with WhatsApp telegram, also drive routes with Google Maps, we have ground truth locking, 08:20.480 --> 08:24.640 generic for all applications with locks when an action is performed, 08:25.600 --> 08:31.920 location spoofing, we will see more of that in a minute, generic pop-up handling, so system pop-ups 08:31.920 --> 08:37.920 like you need to give this app permissions, we can just, and let all them medically, so it doesn't 08:37.920 --> 08:45.200 break or work flow, and we have some spend some time into making the UI navigation robust, and we'll 08:45.200 --> 08:54.240 also tell more about that. So now that we know what Puma can do, we will actually 08:54.240 --> 09:01.840 like to show you a bit what it looks like, so this is a full scenario, and it contains of 09:01.840 --> 09:07.200 it uses multiple phones, and also the phone can use multiple applications at once, so at the top, 09:07.200 --> 09:16.160 we actually, in the initialized telegram, and we actually will first send some messages, 09:16.160 --> 09:21.200 then just start around, send some more messages, but well, why tell you when you can just show 09:22.160 --> 09:27.520 you, so these were just two physical devices which were laying on our desk, I promised we did not 09:27.520 --> 09:35.840 touch them, and we just set them off, just if you start using it, just lay it on your desk, 09:35.840 --> 09:40.800 it's very fun to look at it. We start off with just sending some messages, and as you can see, 09:40.800 --> 09:45.840 we're also not typing the actual messages, we're just sending the text directly into the text box, 09:46.720 --> 09:53.280 and so so messages pop us going to work, he landed just on Skipple, I don't know why you would go 09:53.280 --> 10:00.080 directly to work after that, but let's go for it. So we start around from the location, he is at, 10:00.080 --> 10:04.880 just me, we were just at the NFI, we were not on Skipple, at the at the time, and we'll start the 10:04.880 --> 10:11.520 route, and of course, when you're driving, you should always adhere to the speed limit, so 10:11.520 --> 10:18.480 drive 50 kilometers an hour is fast enough, but Bob deployed something to production just before 10:18.480 --> 10:24.960 he left, and Charlie noticed something, the servers are a bit on fire, so there might be a 10:24.960 --> 10:29.760 problem in the office, well, what do you do? Well, you don't tell somebody, you just take a picture, 10:30.720 --> 10:34.960 that's the work, the way the world works nowadays, so when we send the picture, and Bob 10:34.960 --> 10:42.400 receives it, he takes, oh no, I made a mistake, maybe we should start driving a little bit faster, 10:42.400 --> 10:48.240 so he starts driving about 160 kilometers an hour to just get back to the office a little bit faster, 10:48.240 --> 10:53.120 and if you don't believe me that we were not actually driving this fast, you cannot drive this corner 10:53.120 --> 11:03.120 at 160 kilometers an hour, which also when you start to speed up, look when I was at the 11:03.120 --> 11:07.920 bit of a trouble because it tries to interpolate how fast you're actually going, if we were to stop 11:07.920 --> 11:13.520 the spoofing, immediately it will still keep driving for a kilometer or so, and suddenly 11:13.520 --> 11:19.920 jump back to the beginning, Google Maps is not that smart. Now we'd like to talk you through a 11:19.920 --> 11:26.080 couple of use cases, so you can probably think of many more use cases than we can, but we will talk 11:26.080 --> 11:32.480 through three use cases that we actually use, and the first one is to initialize a application from 11:33.120 --> 11:38.240 scratch, so when do we actually do this? Well first off, when we start researching a new application, 11:38.240 --> 11:42.800 so we don't know what the application does yet, so we just want to figure it out, and then we 11:42.800 --> 11:48.480 met all the features that we can think of, like setting a message, setting a picture, sharing a 11:48.480 --> 11:54.400 location, we implement the diploma, and then we can actually run a sketch to make sure that we 11:54.400 --> 11:58.800 have all the actions linked to the trace that come out of it, and the second one is we want to 11:58.800 --> 12:03.680 create a large data set, because repeating things is not something we would like to do, so just 12:03.680 --> 12:08.320 write it in one line, that says, I send a message, then send 500 messages, and you have a nice 12:08.320 --> 12:14.560 full database. If you would do this manually, well it takes a lot of time, trust me, we've done it in 12:14.560 --> 12:19.600 the past, and well you can also make some mistakes, and then you have to write down that you made 12:19.600 --> 12:25.600 a mistake, and then you just look stupid. With Bouma, it's very simple, you just run the script, 12:25.920 --> 12:31.280 and then supervise, but one way, with the script, my question with no where it happens, and you 12:31.280 --> 12:36.320 also meant to make sure it does every step that you set, it should do, but also, as again, it's just 12:36.320 --> 12:41.680 very fun to watch a phone, do things when you're not actually touching it. Of course, always some 12:41.680 --> 12:47.440 caveats, well it does take time to implement something in Bouma, unfortunately, every development takes 12:47.440 --> 12:52.480 time, UI change can break Bouma, where your heavily reliance on actually where the button might 12:52.480 --> 12:58.000 be, or what the button is called, so when something changes, we also might need to change something, 12:58.800 --> 13:04.160 and we don't support actually creating new accounts. Most of the time, which is create one 13:04.160 --> 13:11.760 account manually, and we use that account when running Bouma. Second use case, monitoring app updates, 13:13.200 --> 13:18.560 in this case, you already have a application that is populated with data, and then you can just 13:18.560 --> 13:24.640 update the application to see if there are changes in the database structure, for example. 13:25.520 --> 13:30.000 So we actually have a pipeline for this, which we start always up with, no, downladying the new update, 13:30.000 --> 13:33.600 you can just down it from the website in case of what's set, but we can also use Bouma to 13:33.600 --> 13:38.640 get it from the app store. We installed on the populated device, ready, and we just start the app. 13:38.640 --> 13:44.960 So we just still Bouma update the app, start it, done, then we extract the database, give it to 13:44.960 --> 13:49.840 Hansgen, just see what it's a bit out. We validate that the traces, but we found are actually still 13:49.840 --> 13:53.760 the traces that we expected to be in there, and then we can report the results back to ourselves. 13:53.760 --> 13:59.440 Hey, we still support this new version, everything is still okidoki. This is useful for applications 13:59.440 --> 14:03.360 where the database actually changes when you update the application, which we know is the case for 14:03.360 --> 14:10.160 what's set, where this is not the case, is for telegram, here you actually need to perform 14:10.160 --> 14:16.080 some UI actions before the new database structure is used. So the first step is still the same, 14:16.080 --> 14:22.320 you just update it on all the device, but now we actually need to execute some actions with Bouma, 14:22.320 --> 14:27.600 and also here, make sure that you log, which is actually happening, which is because you need this 14:27.600 --> 14:33.360 in a few minutes, and the same steps, you just extract the database again, feel it to Hansgen, 14:33.360 --> 14:37.280 validate the fundraising report result, but the validation here is a bit more difficult, 14:37.280 --> 14:42.480 because you just execute new actions, so there are new traces in your database, but Bouma can 14:42.480 --> 14:46.880 help you with that, because it knows what it did, so you can validate that it still did. What it did, 14:47.760 --> 14:51.520 and this is useful when the database structure actually changes when you use it, 14:53.040 --> 14:56.240 which brings us to the next point, there are some things we noticed when we started 14:56.240 --> 15:01.360 automating this pipeline, and the first one was, well, telegram likes to update this blocks, 15:01.440 --> 15:07.200 it just stores all the data in the database, but it used a tiny large object, 15:07.200 --> 15:12.320 and what we could see here is, well, there are two messages, which still look to be in the same 15:12.320 --> 15:17.360 spot, but the magic of the block has changed, which might not be a problem with every block 15:17.360 --> 15:21.200 was a message, but some blocks might be a friend request, and if you start parsing a friend request 15:21.200 --> 15:27.440 as a message, things might break. The next thing we noticed is that the WhatsApp scheme has 15:27.440 --> 15:33.440 actually quite stable, so in the three years we are monitoring it, there were two changes, 15:33.440 --> 15:36.960 only one was very minors, some apps and timestamps have changed, and the second one was 15:36.960 --> 15:41.440 actually a major scheme update were also the message stable was changed, well, what we also noticed 15:41.440 --> 15:46.960 there was that was a staged rollout, so when the new version came out, there were new tables 15:46.960 --> 15:51.120 in there, but when you send a message, it was still in the old table, but when you created the new 15:51.120 --> 15:55.520 account, the new tables would be used, which makes it a bit more difficult, but at some point 15:55.520 --> 16:00.000 the data was, what's my credit to get a table, but it was not finished by your version, 16:00.000 --> 16:05.600 it actually was linked to the account of the user, in conclusion, and I think I need to speed 16:05.600 --> 16:10.640 a bit, so you all know this, well, what is this? Well, it's just very useful for us, yes, I see, 16:10.640 --> 16:16.000 five minutes, as I said, need to need to speed a bit, so the initial app just saves a lot of time, 16:16.000 --> 16:21.600 we don't like to manually enter text messages, there are computers for that, but there's also 16:21.600 --> 16:26.240 means that we know when something happened, we know within 24 hours if we still support the new 16:26.240 --> 16:30.960 app version not, and we have, well, these are a nice reporting way of doing it, it's just a read me, 16:30.960 --> 16:36.640 if it works, it works, and just a fun thing for us, we actually now support telegram faster than 16:36.640 --> 16:43.440 celebrate us, this is a high potential, not just for us as a fully color person, but also for 16:43.440 --> 16:48.240 researchers, educators, and we hope also for some hobbyists, maybe you can automate your grocery 16:48.240 --> 16:52.560 shopping with this. There are, of course, the caveat, well, it has more time, 16:52.560 --> 16:58.240 it's not as if the government is looking in your phone, but if you want a complete forensic 16:58.240 --> 17:02.240 image, you will actually find that Puma has some traces on there, because it does use any 17:02.240 --> 17:08.240 of the commands to actually connect to these applications, and be mindful of the stage well 17:08.240 --> 17:14.160 at now to the fun part. How does it actually work? Well, a Puma makes use of a state machine, 17:14.160 --> 17:19.680 just to make sure that we know where we are in the application at all times, so we have split 17:19.680 --> 17:26.880 every application into three parts, there are states, transitions, and actions. So every screen 17:26.880 --> 17:31.200 that you can see is most likely a state, so in this case you have the conversation state, 17:31.200 --> 17:36.080 and in this case this is the initial state. The universal state is just the first screen that 17:36.080 --> 17:42.400 that opens when you open the application for the first time. Then you can use transition by pressing 17:42.400 --> 17:45.680 the new message button, for example, when you go to another screen, well, that's just not a state, 17:45.680 --> 17:48.800 and you can have a transition to the new message state, and you can have a transition back to 17:48.800 --> 17:54.720 the new message state, but if you actually create a new jet, well, then you go to a jet state. 17:56.080 --> 17:59.040 And from that jet state, you can off, of course, all the transition to other states, 17:59.040 --> 18:02.960 like you can call somebody, you can have some jet settings, you can send media or send media 18:02.960 --> 18:08.960 from a gallery, and Puma can just transition between all these states automatically. So each 18:09.040 --> 18:14.160 action that you have, so what you actually saw in the script is linked to a specific state. 18:14.160 --> 18:19.120 So if you are in the conversation state, you can just say, I want to send a message from my gallery 18:19.120 --> 18:25.600 to Bob, and we'll automatically transition through these states, and send the media from there. 18:26.480 --> 18:30.560 If something goes wrong during those transitions, it actually checks, 18:30.560 --> 18:36.000 am I actually in the state? I'm now expected to be in, if not, it will try to detect which state 18:36.080 --> 18:40.160 it's actually in and continue from there. And if it cannot find out in which state it was, 18:40.160 --> 18:44.640 it will just reset the application, then we know we are in the initial state, and it will try again. 18:44.640 --> 18:48.480 This will, it will only do that once, just to prevent some infinite loops. 18:49.840 --> 18:53.920 But the nice part is, well, when you open the gallery, there might be some 18:54.800 --> 18:58.080 permission, Bob perhaps, it can handle those automatically as well. 18:59.600 --> 19:05.760 Some applications spoofing always nice. So you can just set your location at a specific point, 19:05.840 --> 19:09.120 which is very useful if you don't want your friends to know you're at. Your home, if you ask, 19:09.120 --> 19:14.160 oh, okay, loud no, I'm in the middle of the event Ocean. Very nice to prevent some of social 19:14.160 --> 19:21.040 in the interaction, but you can also navigate as we just saw. So you can either provide a GPX 19:21.040 --> 19:26.000 yourself or you can just call open street maps, which is we've actually built this in. You can just 19:26.000 --> 19:31.440 say, I want to drive from Skipple to identify if we will request a route from the internet and 19:31.440 --> 19:36.480 will just start updating automatically. You can also just adjust the speed, but also just give 19:36.480 --> 19:40.480 it some variance to give it a bit more realism. You say, I want to play 50 kilometers an hour, 19:40.480 --> 19:44.480 I want the deviants of 500 hours an hour, and it will just fluctuate a bit between that. 19:45.600 --> 19:51.120 Well, now for the slide, you never saw coming, please contribute. Welcome to open source. 19:53.120 --> 19:57.600 The first of all, if you use it and you notice that anything isn't working, just please report a bug. 19:57.600 --> 20:03.280 It helps us a lot. We can keep up the date with it. If you want anything just at a future request 20:03.280 --> 20:07.520 or if you're a bit more technically involved, please add something yourself. It can be found 20:07.520 --> 20:12.320 in the difference in suitors, Puma. And if you want a bit more technical in the information, 20:12.320 --> 20:17.520 we have also written that paper about it is on the slide here below, and the QR card will take you 20:17.520 --> 20:20.240 to the GitHub repository. Thank you very much. 20:20.240 --> 20:29.600 What minute? We will be here to hold it.