WEBVTT 00:00.000 --> 00:16.880 I want to start with a quick, about me jumping to a high level overview, jumping to a demo 00:16.880 --> 00:21.320 of the project, do a high level overview of how it works together and I'll wrap up with some 00:21.320 --> 00:23.280 quick lessons learned. 00:23.280 --> 00:28.400 As you might have noticed from the title slide, I'm an engineer at Apple and platform architecture. 00:28.400 --> 00:32.420 I've worked on a pretty wide variety of projects, but some of you might know me from 00:32.420 --> 00:37.600 our SIFPLATIVE examples of what I post I publish about a year ago, SIFPLATIVE examples 00:37.600 --> 00:43.280 at WWVC session from the previous summer that I worked on with Cuba and SIFPLATIVE examples 00:43.280 --> 00:47.000 of repo I help maintain. 00:47.000 --> 00:51.600 SIFPLATIVE examples is a collection of small demo projects to get you started on your favorite 00:51.600 --> 00:52.600 microcontroller. 00:52.600 --> 00:58.360 This is a range of devices, but the primary goal of this repo is to get you into Swift 00:58.360 --> 01:01.640 and you can take it from there. 01:01.640 --> 01:07.400 This would be things like linking an LED or talking over the most complex cases, controlling 01:07.400 --> 01:12.000 a new pixel LED strip, and so this doesn't really constitute a real product, and so that's 01:12.000 --> 01:13.560 what I'm here to announce. 01:13.560 --> 01:18.680 I'm announcing a new addition to SIFPLATIVE examples, which is a full product you could build 01:18.680 --> 01:20.240 and embed the Swift. 01:20.240 --> 01:24.240 Without further ado, let's take a look. 01:24.240 --> 01:26.680 So this is Harmony. 01:26.680 --> 01:30.600 It's a Bluetooth speaker and Fairfluent Music Visualizer. 01:30.600 --> 01:33.360 I'm aloft in white, there are stereo speakers. 01:33.360 --> 01:37.840 On the front there are some human input controls, like playpods, buttons, and a volume knob. 01:37.840 --> 01:39.880 And dead center, there is a Fairfluent. 01:39.880 --> 01:43.760 So Fairfluent is a iron liquid that reacts to magnets. 01:43.760 --> 01:48.120 So by behind the Fairfluent is a Ultra Magnet. 01:48.120 --> 01:55.680 So I'm going to connect to Harmony and play some music. 01:55.680 --> 02:03.320 This is Bluetooth digital volume, flight sensor headphone, or on the knob. 02:03.320 --> 02:18.080 And as the beets, you can set a Fairfluent react. 02:19.040 --> 02:20.520 How do I just go on? 02:20.520 --> 02:22.520 So I'll come off there. 02:27.520 --> 02:30.520 So let's take a look at how it works. 02:30.520 --> 02:38.000 I'm going to focus on the firmware of Harmony, and not the virtual and mechanical parts. 02:38.000 --> 02:40.400 At the bottom of Harmony is the PQ SDK. 02:40.400 --> 02:42.680 This provides a whole bunch of functionality. 02:42.680 --> 02:45.840 The one critical piece of the hardware API is it provides. 02:45.920 --> 02:48.000 However, API abstractions it provides. 02:48.000 --> 02:54.840 This allows us to control things like GPIO pins, I2S blocks, I2C blocks, and PIOs. 02:54.840 --> 02:59.360 On top of these abstractions, I built an audio driver that can speak I2S. 02:59.360 --> 03:03.600 And now we do control the audio pixel strip on the front and callbacks for the very 03:03.600 --> 03:06.560 human input controls. 03:06.560 --> 03:09.600 The PQ SDK also provides a Bluetooth stack. 03:09.600 --> 03:13.840 So on top of that, I implemented the hand-lace for the various Bluetooth packets and wrapped 03:13.840 --> 03:18.000 one of the audio decoters in a nice Swift API. 03:18.000 --> 03:22.600 Additionally, bridging these two halves together, I use the CMpsus DSP APIs to create 03:22.600 --> 03:24.800 an audio analyzer. 03:24.800 --> 03:26.600 All of this together forms the Harmony firmware. 03:30.320 --> 03:35.680 Now I want to dig in to how the audio data flows through Harmony as it's playing music. 03:37.040 --> 03:42.520 Now, it all starts when a controller, a phone, or a Mac, connects to Harmony over Bluetooth. 03:42.520 --> 03:48.120 Once the controller starts playing audio, Harmony receives encoded audio data which it stores 03:48.120 --> 03:50.440 in a ring buffer. 03:50.440 --> 03:55.040 Once that ring buffer reaches some threshold value, Harmony begins decoding the compressed 03:55.040 --> 03:59.640 data into uncompressed PCM audio. 03:59.640 --> 04:04.560 The uncompressed data is chunked into frames of 256 stereo samples and sends over to the audio 04:04.560 --> 04:07.480 analyzer to handle. 04:07.480 --> 04:12.120 The analyzer performs an FFT in real time to determine how much low frequency content is 04:12.120 --> 04:17.080 in that frame, and it based on some threshold value chooses whether or not to power 04:17.080 --> 04:20.080 the electromagnetic. 04:20.080 --> 04:27.480 The frame and this bit of data are then passed to the audio driver to control the hardware. 04:27.480 --> 04:34.200 The audio driver takes the raw audio data, sends it out as I2s to a digital analog converter 04:34.200 --> 04:38.200 who signals them amplified and played out the speakers. 04:38.200 --> 04:43.480 That single bit we had earlier goes through a different analog amplification circuit and 04:43.480 --> 04:46.040 controls the electromagnetic. 04:46.040 --> 04:49.920 There's a way more details about how this all works together. 04:49.920 --> 04:54.120 So if you're interested, please go out and have to answer any questions. 04:54.120 --> 05:02.280 I want to wrap up with some quick lessons learned. 05:02.280 --> 05:07.000 The first is to find great mentors while I was building this project, the first 50% of 05:07.000 --> 05:10.520 the work, I'm turning up to be working on firmware. 05:10.520 --> 05:17.200 The next 50% was the chemical design, and then there was another 50% that was a electrical 05:17.200 --> 05:21.960 design, and then at the end I didn't even know this was going to happen. 05:21.960 --> 05:25.360 There was another 50% that was chemistry. 05:25.360 --> 05:29.600 The point here is that I could not have done this alone, and I relied on the insight from 05:29.600 --> 05:35.040 great mentors to guide me to the process and help me uncover the bits I didn't know. 05:35.040 --> 05:39.400 So I'd like to do a shoutout right now to Kuba, Mejir, and Kevin for helping me put this 05:39.400 --> 05:42.400 together. 05:42.400 --> 05:43.400 All right. 05:43.400 --> 05:47.760 The next lesson learned was to start with something that works. 05:47.760 --> 05:51.600 When I started this project, I wanted to start with an empty Sophia file and be like, 05:51.600 --> 05:53.240 yeah, I made the thing from scratch. 05:53.240 --> 05:56.040 That was a terrible idea. 05:56.040 --> 06:00.040 I quickly realized the scope was very large, and I was solving problems I didn't really 06:00.040 --> 06:01.040 care about. 06:02.040 --> 06:06.720 So the actual solution I ended up following was finding some example C code of how to control 06:06.720 --> 06:12.680 the Raspberry Pi's Bluetooth stack, building on top of it, and molding it into the project 06:12.680 --> 06:13.680 you see today. 06:13.680 --> 06:19.880 The takeaway here is building from nothing, like going from zero to one is really difficult, 06:19.880 --> 06:23.280 and often has you focusing on problems that aren't important to you. 06:23.280 --> 06:26.520 Find a good starting point that allows you to solve the problems you care about, adding 06:26.520 --> 06:28.520 or the other things. 06:29.520 --> 06:32.520 All right, so I was a really quick overview of Harmony. 06:32.520 --> 06:39.520 All of the resources needed to build Harmony yourself are available either right now or very shortly 06:39.520 --> 06:41.520 on Swift Embedded Examples. 06:41.520 --> 06:46.720 This includes the embedded software, the build of materials, and shortly the electrical 06:46.720 --> 06:48.720 schematics, and 3D models. 06:48.720 --> 06:52.520 And please check out Swift Embedded Examples if you want to build one of your own. 06:52.520 --> 06:54.520 I'm excited to see what you guys do with it. 06:54.520 --> 06:55.520 Thank you. 06:58.520 --> 07:00.520 Thank you.