WEBVTT 00:00.000 --> 00:07.000 Thank you. 00:07.000 --> 00:13.000 Thank you. 00:13.000 --> 00:14.000 Thank you, folks. 00:14.000 --> 00:15.000 Good morning. 00:15.000 --> 00:18.000 Thank you for being here so early on Sunday. 00:18.000 --> 00:20.000 This is maybe it was hard for you. 00:20.000 --> 00:23.000 It was not that hard for me with this a bit. 00:23.000 --> 00:27.000 So here we're here to talk about open telemetry instrumentation. 00:27.000 --> 00:31.800 I was supposed to be giving this talk with my friend Israel, but he cannot come to the 00:31.800 --> 00:37.200 maybe it's because he went viral a couple of days ago when he got a job offer or from a recruiter 00:37.200 --> 00:42.360 asking for three years of experience in deep seek, which is pretty weird. 00:42.360 --> 00:45.600 So now he's famous and he cannot come. 00:45.600 --> 00:51.960 But yeah, and you are stuck with me and I'm also, I work now as a product leader with 00:51.960 --> 00:52.960 telemetrics. 00:52.960 --> 00:58.960 I guess you didn't know there is a cloud product and I also work at the university teaching physics 00:58.960 --> 01:06.440 to some vet students in Madrid in Spain and yeah, I have a past like everybody does. 01:06.440 --> 01:13.280 And I also love playing heavy metal and seemed racing so if you do these things and you 01:13.280 --> 01:17.200 want to talk about them later, there will be super cool as well. 01:17.200 --> 01:22.960 So we are here to talk about instrumentation and open telemetry. 01:22.960 --> 01:29.080 So you will be hearing this a lot, so I will try to make it super quick. 01:29.080 --> 01:37.800 So open telemetry is this project, the superheets and gives you APIs, SDKs and tools to understand 01:37.800 --> 01:42.640 or create observability and this means metric flux and traces. 01:42.640 --> 01:47.200 So things start to get a bit complicated and overwhelming because this means that these 01:47.200 --> 01:54.520 APIs as the case and tools need to produce or ingest and show metric flux and traces. 01:54.520 --> 01:59.200 And if you think about it, then you need to implement this in a lot of languages. 01:59.200 --> 02:04.720 So it's like, okay, now it's like this is super overwhelming. 02:04.720 --> 02:10.600 The good part is that open telemetry said, hey, or this super clever guy said, okay, let's 02:10.600 --> 02:12.120 establish a protocol. 02:12.120 --> 02:16.280 So we don't care about what happens in these tools and we have interoperability. 02:16.280 --> 02:21.840 So the protocol helps to abstract that and, okay, I don't care, with language, you 02:21.840 --> 02:26.600 are programming your tool or mine, they just talk it's other. 02:26.600 --> 02:29.720 And also it unifies in the protocol, metric flux and traces. 02:29.720 --> 02:35.520 So you don't need a separate protocol for each of them, which is really convenient. 02:35.520 --> 02:42.800 And it also provides instrumentation libraries and out instrumentation for the code. 02:42.800 --> 02:48.400 So you don't need to think about it or put a lot of different libraries when you go to 02:48.400 --> 02:49.400 the code. 02:49.400 --> 02:51.960 So what is this thing about instrumentation? 02:51.960 --> 02:55.200 Let's start with zero code because it's fancy. 02:55.200 --> 03:02.200 And so it's something that tells you effortless helps you to create metric flux and traces 03:02.200 --> 03:03.840 in your code. 03:03.840 --> 03:09.640 This looks super cool and it's great for your managers and because then you don't need 03:09.640 --> 03:17.160 to spend money doing this, but it has some inconveniences, as well as we will see. 03:17.160 --> 03:18.880 It's agent driven. 03:18.880 --> 03:21.620 This means that just imagine you are in Kubernetes, you would have an operator that 03:21.620 --> 03:28.640 will inject or activate some observability signals and in your code. 03:28.640 --> 03:29.640 And how is it done? 03:29.640 --> 03:34.040 And in three different mechanisms mostly, this monkey patching or byte code manipulations 03:34.040 --> 03:36.240 if we are talking about Java or a BFF, right? 03:36.240 --> 03:43.120 Like I'm calling the kernel, sorry, there is an event in the kernel and then I say, 03:43.120 --> 03:48.240 you want to send a message and then I produce this metric or this span. 03:48.240 --> 03:55.520 But there are two super interesting calls later to talk later today about these topics. 03:55.520 --> 03:58.040 So I will let them talk more about that. 03:58.040 --> 04:02.240 I will be focused more on manual instrumentation. 04:02.240 --> 04:07.400 However, if you are curious about how this monkey patching works, for example, you can 04:07.400 --> 04:11.600 look at Wikipedia, super simple, just imagine you are using this. 04:11.600 --> 04:19.120 This is an example before AI, just imagine you are using a Python library and you want to 04:19.120 --> 04:21.440 say, hey, what's the number pi, right? 04:21.440 --> 04:22.440 What's 3.14? 04:22.760 --> 04:28.920 But then what you can do is, okay, at runtime, I change the behavior of this library. 04:28.920 --> 04:32.920 So at runtime, I change the value and now it's like 3.2. 04:32.920 --> 04:38.720 The same as it happens with sensitivity, by the way, but then it doesn't change the library itself. 04:38.720 --> 04:43.920 So open telemetry or this mechanism helps you to, you have your library that is behaving 04:43.920 --> 04:47.040 in one way, but also you can at runtime change the behavior. 04:47.040 --> 04:50.360 So it starts sending metric locks and traces. 04:50.360 --> 04:52.240 So that's the principle. 04:52.240 --> 04:57.040 And you do it by configuring libraries and exporters. 04:57.040 --> 05:00.640 For example, I want to generate metrics and send them to this endpoint. 05:00.640 --> 05:01.640 I want to generate. 05:01.640 --> 05:05.720 So this is what you will be doing when you do serial code instrumentation. 05:05.720 --> 05:09.840 So there's nothing super fancy for the user. 05:09.840 --> 05:13.720 So it's pretty convenient and it's great for edges and libraries. 05:13.720 --> 05:18.960 What I mean by this is you have your code, you will be, for example, calling your database. 05:18.960 --> 05:23.440 And when, and you want to understand what's going on there. 05:23.440 --> 05:28.120 So the guys, super clever, of that database, they created some instrumentation for you. 05:28.120 --> 05:30.040 So you can activate or deactivate. 05:30.040 --> 05:36.240 So you can understand if you have issues in your code or in the database communication itself. 05:36.240 --> 05:40.160 So this is super good, but at the end of the day, not every language support that. 05:40.160 --> 05:43.760 I come from the C++ world and this doesn't work there. 05:43.840 --> 05:52.960 It's also really hard to control the data because you can be generating a lot of signaling. 05:52.960 --> 05:57.560 And if you don't treat it properly, then you can have a lot of increasing costs. 05:57.560 --> 06:00.880 And at the end of the day, we want to understand what's going on in our code. 06:00.880 --> 06:06.720 We want to understand if it's failing because of us and you need to do something. 06:06.720 --> 06:09.320 So let's talk about manual instrumentation. 06:09.320 --> 06:11.520 That's why we are here. 06:11.560 --> 06:19.920 So when you start to include your open telemetry libraries inside your code base, it's kind of hard. 06:19.920 --> 06:26.680 So actually, that there was this survey a couple of months ago, months ago, 06:26.680 --> 06:32.640 that people were asking the community, hey, we need more examples. 06:32.640 --> 06:37.680 We need the documentation to be more thorough. 06:37.680 --> 06:44.680 So actually, this is why this talk is because when I was using to instrument my code, 06:44.680 --> 06:46.680 I was using an open telemetry, I had some doubts. 06:46.680 --> 06:49.960 So the first thing you need to understand, so now this is what we're going to talk about, 06:49.960 --> 06:53.640 how is this structure, how you can do it. 06:53.640 --> 06:58.520 The cool thing as every other library, it provides an API and an SDK. 06:58.520 --> 07:02.920 So the cool part is that it's the couple. 07:02.920 --> 07:08.440 So you can abstract yourself from, hey, where I'm sending my data. 07:08.440 --> 07:09.680 I don't carry my code, right? 07:09.680 --> 07:14.520 When I am processing my request, I don't really care where I'm sending my metrics, 07:14.520 --> 07:17.600 where I just want to don't pollute my code. 07:17.600 --> 07:24.120 So this is one of the principles I have my API in which I say, hey, this is my plus one to this metric, 07:24.120 --> 07:27.280 but I don't care about the rest and then you have the SDK. 07:27.280 --> 07:30.840 So let's start discussing how you do this for metrics. 07:30.840 --> 07:34.120 And we will also do it for traces and logs. 07:34.120 --> 07:35.960 Formatrix, I say what's in. 07:35.960 --> 07:41.640 You normally have your function in which you want to add an error. 07:41.640 --> 07:46.200 So at the end of the day, what you want to do is say, okay, I have one more error, 07:46.200 --> 07:51.240 and then I don't care, I just throw it and someone else will pick it up. 07:51.240 --> 07:56.200 So you do this by calling your API and when you start really in the documentation, 07:56.200 --> 08:00.600 it can be, as every documentation, it can be boring. 08:00.600 --> 08:06.520 And that's why I'm trying to put here some fancy pictures, maybe it's more clear. 08:06.520 --> 08:10.600 So you have these concepts that are measurements, instruments, 08:10.600 --> 08:16.120 meter, meter provider, and you have, okay, but I just want to do like plus plus my metric, right? 08:16.120 --> 08:17.080 My measurement. 08:17.080 --> 08:21.400 And then you also have the SDK which provides you and reader and a supporter and what is that? 08:21.400 --> 08:24.200 So this is one I'm trying to explain now. 08:24.200 --> 08:27.640 So they tell me, okay, we have a meter provider. 08:27.640 --> 08:33.000 When I want to create a measurement, I need to call a meter provider, 08:33.000 --> 08:37.000 and a meter provider is this thing that will give me a meter. 08:37.000 --> 08:38.200 And what is a meter? 08:38.200 --> 08:40.360 So it's my interface, the meter provider. 08:40.360 --> 08:46.600 The meter is something that tells me which resource is generating the metrics. 08:46.600 --> 08:49.240 So it's bound, for example, to a library, right? 08:49.240 --> 08:53.400 In this case, if I'm using a MongoDB library, 08:53.400 --> 08:58.840 it will be a meter with this name, right? 08:58.840 --> 09:02.200 And then I can ask for this meter for an instrument. 09:02.200 --> 09:03.480 What is an instrument? 09:03.480 --> 09:07.560 In metrics, you can have different set of kinds of metrics of instruments, 09:07.560 --> 09:12.760 so a counter or a histogram, so that's a meter. 09:12.760 --> 09:16.520 So then I'm done, because now I have my counter, right? 09:16.520 --> 09:21.480 My instrument, and I can tell, okay, I have one more error, and I push it, right? 09:21.480 --> 09:26.120 So we are now done with our goal. 09:26.120 --> 09:29.320 We know how to say, okay, give me one more error, 09:29.320 --> 09:32.760 or I want to report that I had an error. 09:32.760 --> 09:37.720 Well, just we just need one more thing, which is, if we go just to real code, 09:37.720 --> 09:44.120 what you will be having is you create, well, this is a plus, of course, 09:44.120 --> 09:49.960 so lines can get long, but you create a set of labels, hopefully, 09:49.960 --> 09:53.720 you know, with no high cardinality, and you will say, okay, 09:53.720 --> 10:00.280 this ID, or this response code, and then I added to my labels to my metric. 10:00.280 --> 10:06.680 And I also, the number, I just say, okay, just have one more, but you can put them, right? 10:06.680 --> 10:08.200 So this is it, right? 10:08.200 --> 10:13.560 So now these concepts, coming again, I have a measurement, which is what I want to add, 10:13.560 --> 10:15.560 and I added to an instrument, which is my counter. 10:15.560 --> 10:19.160 And then the other things are for the API. 10:19.160 --> 10:22.600 So what's, okay, so that's cool, but what happens after that? 10:22.600 --> 10:26.600 Because I was wondering, like, okay, I just throw this metric and now what, right? 10:26.600 --> 10:30.920 Well, it gets a stored, you know, into an aggregation store, into a local 10:30.920 --> 10:35.400 a storage, and then the cool thing, and this is what I love about this project, 10:35.400 --> 10:40.680 is that it says, okay, now you define a reader, and an expert. 10:40.680 --> 10:45.240 So from this reader, you are able to say, okay, every second, every five seconds, 10:45.240 --> 10:50.920 every half a second, you say, now you get this data, you aggregated, and you send it out. 10:50.920 --> 10:55.080 Where, in this case, okay, to an open-tylometric collector, but wait for it, 10:55.080 --> 10:59.000 because you can put another reader and say, you know what, leave this metric here, 10:59.000 --> 11:02.360 because I promise you guys will come and take them. 11:02.360 --> 11:10.040 So now, I don't really care if I, in my code, if I decided that I would use a 11:10.040 --> 11:15.880 permission, sorry, if I would use open-tylometric forever, whatever, because now I'm done, 11:15.880 --> 11:20.680 and how you define this, you define this, for example, in a separate file, or part in your code, 11:20.680 --> 11:24.920 you will be doing a single tone, or whatever, and this is how you do it. 11:24.920 --> 11:30.920 So super quick, here, I'm showing you how you define a reader, you know, like, 11:30.920 --> 11:35.080 this, every second, you get the metrics and you take them out, 11:35.080 --> 11:40.840 and you will have some, define some parameters that you can do via configuration files. 11:40.840 --> 11:45.640 For example, hey, I have my open-tylometric collector, or my backend there, so do you define the 11:45.640 --> 11:50.360 exporter, so this is like configuration file, and yeah, a lot of code, and you know, to make things work, 11:50.360 --> 11:57.960 so it's a single tone, but that's it, right? So, and of course, I just, this one was an example 11:57.960 --> 12:03.240 for the open-tylometric exporter, but you will have the similar function for promises, 12:03.240 --> 12:10.520 if you are, if you want both in my example. So, this is it, right? I have a measurement, 12:10.520 --> 12:15.240 I will throw it to my instrument, and that's part of the API, and then there will be a store, 12:15.240 --> 12:20.680 and sense, whatever I want, period, I'm done. So now we know how to instrument our code 12:20.680 --> 12:27.080 with open-tylometric, in C++, well, no, it's not only C++, because the cool thing about open-tylometric 12:27.080 --> 12:31.720 is that it gives you this, for it's an specification for every language, so every language 12:31.720 --> 12:37.160 that implements open-tylometric libraries need to do this. Of course, you will have nuances, 12:37.160 --> 12:42.360 but that's it. So now you know how to instrument in every language in the world, it's pretty cool. 12:43.400 --> 12:49.800 Now, traces, we are done with methods, let's go with traces. Normally, 12:50.840 --> 12:56.200 what I see is that everybody thinks they know traces, but then they have some misconceptions, 12:56.200 --> 13:00.840 so if you allow me, I will do a super quick introduction to traces, because normally, 13:00.840 --> 13:06.760 what I see is people thinking, no, this is just a low level log debug, and then I'm done, 13:06.760 --> 13:11.720 well, no, this is a different thing at this in this notation. So, what is a trace? 13:12.520 --> 13:17.160 Where will we meet for a second? It's just some spans I put together under a trace ID. 13:18.520 --> 13:25.240 Nothing else, nothing fancy, spans under the same trace ID. And what is a span? 13:25.320 --> 13:30.040 It's a representation of an operation, and it has, because every operation happens in time, 13:30.040 --> 13:34.120 I have this time dimension, which is pretty cool, because it will help me in the end. So, 13:35.320 --> 13:42.280 and why are this important? Because when I have a distributed system, and a guy is doing some 13:42.280 --> 13:47.800 stuff in JavaScript, that goes to a go API, and then goes to my database, and then something fails, 13:48.600 --> 13:54.760 then I cannot save my day by doing QCTL logs, because then you are like, okay, what do I do with all of this, right? 13:54.840 --> 14:01.000 So, if you have a span, you can track, if you have a trace, you can track, you can trace your call, 14:01.640 --> 14:05.160 and with this span, it's like, okay, this service call, this function, that's a span, 14:05.160 --> 14:08.840 and then I have this other function, that's another span, and then I have all the conflicts. 14:08.840 --> 14:14.200 So, this is this distributed tracing, okay, and at the end of the day, as I was saying, 14:14.200 --> 14:17.240 you can, you can take it in a gunster, which is a normal visualization. 14:18.200 --> 14:23.800 Now, let's get to instrumenting fun. Now, we are more experts, so maybe we don't need to go deep 14:23.800 --> 14:29.880 dive into all of them, but this is even simpler, so I started a spoiler error, I started with the 14:29.880 --> 14:35.160 most difficult one. Now, tracing is even simpler, because there are no different instruments, 14:35.160 --> 14:40.280 I was saying, okay, you have metry counters, or histograms, there are different, here we have a span, 14:40.280 --> 14:44.600 there's no different sets of span, you have a span, and that's it. So, our API gives that 14:44.600 --> 14:51.720 a trace provider, a tracer, and the span, so I will be reporting a span, and they will go to the 14:51.720 --> 15:00.680 SDK to export it. So, it's the same kind of representation, and let's start from, from the end 15:00.680 --> 15:05.640 this time, this is just, okay, I will be defining my resource, this is an important concept in 15:05.640 --> 15:13.480 open telemetry, because I tell, hey, this instrumentation, or this signaling is coming from this 15:13.480 --> 15:18.360 resource, in this case my library, or my program, so I have that, so at the end, when I want to 15:18.360 --> 15:25.480 take the data, I can understand where this is coming from, so I put my resource, I also say my, 15:25.480 --> 15:29.480 where I want to explore my data, like, okay, I have an open telemetry collector run there, 15:29.480 --> 15:35.240 and some options like batching, for example, I have processing source, I can process my data, 15:35.240 --> 15:39.480 so I can say, okay, I can say, let's send it every, you know, every megabyte, 15:39.480 --> 15:44.360 I don't want to have a lot of signaling, right? Another thing I, so that's for the SDK, 15:44.360 --> 15:48.200 another thing I want to recommend mostly in C++, because how it works, 15:48.840 --> 15:54.840 is to create this kind of auxiliary function that will help you to create child response, 15:54.840 --> 16:01.720 because what we want to do is to propagate context, don't forget that, so I want to have a function 16:01.720 --> 16:07.720 in which I say, okay, create me a span out of this one, so it's just propagating the context, 16:07.720 --> 16:12.280 nothing fancy, but it will help you, because when you are in the real world, and you say, okay, 16:12.280 --> 16:17.400 I'm going to send an HTTP request, okay, this is our example, let's go with that. 16:17.480 --> 16:22.920 I want to send an HTTP request, so hey, let's start my span, and I will set some attributes, 16:24.120 --> 16:31.000 which attributes, how do you decide, how to call this attribute, how are they going to function in the end? 16:31.000 --> 16:39.240 Well, that's a great question, because the open telemetry project defines where they are called 16:39.240 --> 16:43.480 semantic conventions, and there is a semantic convention for a lot of things, 16:44.440 --> 16:50.600 you can say, I don't know, like for CICD or function as a service, or for example HTTP, 16:50.600 --> 16:57.880 which is our case, so by using the open telemetry library, you will have all these things 16:57.880 --> 17:04.120 already in your code, because you know, okay, I am sending a request that is going to this, 17:04.120 --> 17:09.080 you are, how do I put this in a span, so I can take it in the end, so it's so easy, okay, 17:09.080 --> 17:13.000 you just use this, you adjust it to the span, you just use the primitives from the library, 17:13.000 --> 17:19.480 and you are done, you don't need to overthink or redo, so it's pretty cool. After that, so I added, 17:20.360 --> 17:26.360 write my URL to my span, I send my request, and what I do now is say, okay, my counter, remember, 17:26.360 --> 17:31.480 we had our counter, okay, plus one, I sent a request, so that's another counter, it's not another, 17:31.480 --> 17:36.520 and I add an event, which is another very important concept for traces and spans, 17:37.400 --> 17:43.560 is not only about having this, this one called this one, no, you can add your logs, 17:43.560 --> 17:48.040 we all love to say, hey, I was here, hey, this fail, hey, I don't like this, if I am here, 17:48.040 --> 17:53.240 we are all screwed, right, this kind of things we want to add them, so these are events in this jargon, 17:54.760 --> 18:00.600 and yeah, so, okay, I send my, this is just continuing with your sample, I send my request, 18:00.600 --> 18:07.080 when I get my response, I will be adding another event, hey, I also, you know, in a super short demo, 18:07.960 --> 18:13.080 but then I can also validate my answer, and if it went back, I can also add another event, 18:13.080 --> 18:21.880 or set my span as this span was an error, so I can take in the end, the trace, just, and we are 18:21.880 --> 18:29.160 done, however, we just need one more thing, if you are wondering how to help this propagates, 18:30.360 --> 18:37.160 well, because you need to propagate the context, this service at the other side needs to 18:37.160 --> 18:42.440 understand your span and your library and things, so how does this travel, it travels, for example, 18:42.440 --> 18:49.640 in HTTP, it travels via headers, so, but then you need to invent them, no, this is standardized, 18:49.720 --> 18:55.240 and actually these are the headers, and you would just just do it your hint, so the library 18:55.240 --> 19:03.000 will handle this for you, in C++, if you want to do it yourself, you know, like, it's super big, 19:03.000 --> 19:09.560 and everything, but it's just a couple of functions that you will say, the concept is super simple, 19:09.560 --> 19:17.720 is hey, I have this request coming from HTTP, please inject them in this span, or I am sending 19:17.720 --> 19:24.280 a request, okay, let's take from my context, you know, the library, the primitive, and put them 19:24.280 --> 19:32.840 into HTTP headers, and I'm done, so that's the only part we were left, so yeah, we can do this super 19:32.840 --> 19:39.400 quick, just to show how, like, why is this important and what, how does it look like, I have here, 19:39.640 --> 19:52.040 I have here a client that is sending 100 requests per second to a dummy server, and so let's give it 19:52.040 --> 20:02.840 sometimes, so it generates data, and here we go, and let's refer us, and at the end of the day, 20:02.840 --> 20:08.680 what I'm going to look is to some spans, you know, like a trace, and I can take them, and why 20:08.760 --> 20:14.600 all of these that I was saying is important? Well, we have now, I know it's super small, but we have 20:14.600 --> 20:21.800 this now, this spans, because what I'm doing in my client is I send a get request, then I get the response, 20:21.800 --> 20:28.120 I send a put request, then I get the response, so I can take the behavior of this other service, 20:28.120 --> 20:33.880 the other side, so the cool thing is, because I was instrumental in my spans with this 20:34.840 --> 20:42.760 open telemetry standards and libraries, now I can inspect my span, and I can take here that, 20:42.760 --> 20:48.600 okay, this span corresponds to a post method, and it received a 200 code, so everything is okay, 20:48.600 --> 20:54.200 what's I do not need to worry about these things, it was in OTP format, it was a client, 20:54.200 --> 20:58.840 you remember I was saying this resource, okay, I said client, now I have here a client, I know 20:58.920 --> 21:06.360 this span is coming from a client, and also the URL I was calling was this one, and I can also have 21:06.360 --> 21:13.160 some information about the library that is generating this, so I can see that, and the other cool part 21:14.040 --> 21:21.960 is I have here my logs, right, so I was talking about metrics, log and traces, traces, 21:21.960 --> 21:28.280 when you start using them, and understanding it's power, maybe you start to put less logs 21:29.800 --> 21:35.160 to the pipeline, and in case you were wondering for metrics, yeah, now we have them, 21:36.120 --> 21:41.720 this is by the way I was just sending these metrics and and traces to an open telemetry collector, 21:41.720 --> 21:49.080 and of course, storing them in Jagger and here in a victory metrics instance, and here, 21:49.080 --> 21:52.840 you know, this is open telemetry, in case you are not used to open telemetry, I'm more to 21:52.840 --> 21:58.760 permit you, you can visualize it the same, so you don't need to for the user, at least this is 21:58.760 --> 22:08.040 not an issue, right, so you can easily take that, so that's it, for the demo, let me go super quick 22:08.040 --> 22:14.680 to logs and we are done, yeah, and in case the demo didn't work, I had those screenshots, logs, 22:15.640 --> 22:19.160 you all know logs, that's the first thing we do as programmers, hello world, that's a log, that's 22:19.240 --> 22:25.320 observability, right, we are all experts in observability and logging, so what I am proposing you here, 22:25.320 --> 22:31.000 right, is use, well, me, everyone is metrics, logs, and traces with open telemetry library, 22:31.800 --> 22:38.840 and that's it, why? After all, you will tell me, yeah, I like to just bring to the console, 22:38.840 --> 22:42.440 yeah, whatever, we all do it, actually if you go to the open telemetry library, they have in 22:42.440 --> 22:49.560 their examples, they are printing to the console as well, so why shouldn't I? Well, if you continue 22:49.560 --> 22:55.400 doing this, first of all, you have already the library, you are just need for metrics and traces, 22:55.400 --> 23:00.600 so why not for logs? You can add attributes, for example, in an open telemetry collector, if you 23:00.600 --> 23:05.080 use it to collect the logs, you can add the attributes which are standardized, like hey, this is coming 23:05.080 --> 23:11.000 from this container or this name space, and you can also filter and but, so you don't have a lot of 23:11.000 --> 23:16.280 signaling, you can transform and process the data, so it's pretty convenient, you can also help 23:16.280 --> 23:22.440 for correlation. For example, logs, if you add a trace ID, can you imagine how cool is that? Because 23:22.440 --> 23:28.200 now, if, because you like to look logs, okay, and you do your kip city a lot, if you see a 23:28.200 --> 23:33.640 spanity, now I can go together and take, okay, this is all the context, cool, right? So there are 23:33.640 --> 23:43.000 many reasons why you should be doing this. Also, yeah, we all love, for example, in C++, when you are 23:44.360 --> 23:50.040 instrumenting logs, then you start to reinvent the wheel, because you start, how do I activate or 23:50.040 --> 23:55.640 deactivate the login? How do I reduce the login level? How do I, and then you start, we all have 23:55.640 --> 24:00.200 that coworker or even inside ourselves that likes to reinvent the wheel and create another library 24:00.280 --> 24:03.880 for this and another, and things can get messy already, have it, why don't we don't use it? 24:04.920 --> 24:09.560 However, I know you guys will be still using it, I will still be doing that, and I will just, 24:09.560 --> 24:15.160 hey, I have an error, I just put it to the console and I'm done. Okay, if that's your case, it's okay, 24:15.160 --> 24:22.680 because if you, one means putting your errors to the console, in this that when Linux is a file, 24:22.680 --> 24:27.960 you are putting your logs into a file. So the open telemetic collector can read them, 24:29.480 --> 24:34.200 process them, and do all these things, and send them. Of course, you will lose some things, 24:34.200 --> 24:39.080 like correlation, you cannot correlate the idea if you didn't instrument it with a spam, 24:39.080 --> 24:47.320 but at least you get something out of it. So it's not that bad. And okay, you are experts now in 24:47.320 --> 24:55.320 this, you will be creating your resource, and adding it to an exporter, and the processor, 24:55.320 --> 25:01.480 and all of that, so you already know how this works, it's the same for all, for all, once, and 25:02.680 --> 25:08.440 and this is how it looks. So it's not that bad. I mean, it's super quick. You just say, okay, 25:08.440 --> 25:15.320 give me my locker, and I love my data. However, what I would recommend is if you are using traces, 25:16.200 --> 25:23.160 let's add your spine IDs, because now you can add the end correlate. So it's just a small thing. 25:25.560 --> 25:33.720 But, and this is the last part. Why, how does it look like, right? We know, hey, 25:33.720 --> 25:40.280 ever here, that's a log, this is a log in open telemetric, okay? It has those resources, 25:40.280 --> 25:45.640 it has maybe some time stamps, which is good because, at the end, you like the time stamps, 25:45.640 --> 25:50.840 you want to add them, right? It also has all this information about my library that I was showing, 25:50.840 --> 25:58.040 so this is all standardized. So my only message here is what's out, because these things can be 25:58.040 --> 26:05.080 expensive. We love to put a lot of logs there, but we will have, as I was saying at the beginning, 26:05.080 --> 26:11.160 there is another talk on how good our body's auto instrumentation, the same goes for manual instrumentation, 26:12.440 --> 26:16.200 normally, at least in my experience, one of the biggest issues we had in performance 26:17.160 --> 26:21.800 was because we were logging a lot, and we needed to produce them. And also with costs, right? 26:22.520 --> 26:28.440 Okay, so as a summary, if I want to give you some takeaways, or I would like you to take 26:28.440 --> 26:34.760 some things home, try to mix both worlds because automatic instrumentation is also nice, 26:34.760 --> 26:42.360 but now hopefully you know how to do manual one. Open telemetric will help you decoupling the SDK 26:42.360 --> 26:48.520 from the API, so you can abstract your business logic from where you are supporting data and 26:48.520 --> 26:53.640 change in the supporters, in this case, and semantic conventions, don't forget them, they are 26:53.640 --> 26:57.640 super important because they help us to correlate all the data and understand what the hell is going 26:58.360 --> 27:04.680 and okay, if you still want to, you know, like print, then you use a collector, like the 27:04.680 --> 27:09.640 open telemetric collector, it will help you to inject some data and enrich your data so you can 27:09.640 --> 27:14.840 have good observability in the end. So that's it, and now we are open for questions, thank you. 27:15.080 --> 27:19.080 Thank you. 27:23.080 --> 27:25.320 Can we have any questions? 27:29.160 --> 27:35.640 What's rough, last night, right? I'm out of question about using it in embedded settings, 27:35.640 --> 27:39.800 so I have an embedded Linux system that I want to apply open telemetric too. Sorry, I kind of 27:39.800 --> 27:47.800 hear that the mic is the mic, or maybe it's an embedded Linux system, I want to apply 27:47.800 --> 27:52.200 open telemetric too, but I'm worried about things like the aggregator, how long is the retainer, 27:52.200 --> 27:56.840 across reboots and stuff, have you tried using an outside of like a Kubernetes cluster? 27:56.840 --> 28:03.480 So you mean inside the code? I'm just talking like instead of on a cloud somewhere, like doing it 28:03.480 --> 28:07.800 on an embedded device with a battery and limited resources, you know how well it's a 28:07.800 --> 28:15.960 lot of people there, or? Yeah, so if I understand correctly, you want to use the open telemetric 28:15.960 --> 28:20.840 API and you are seeing that it's aggregating data and maybe it has some performance. 28:22.520 --> 28:28.120 Just imagine or in my experience, what I saw is that it's actually convenient to have that, 28:28.120 --> 28:33.560 because otherwise you will be sending all the signals one to one, and then you have like 28:33.640 --> 28:38.440 a lot of networking issues, because you are having all that bandwidth. Of course, you will need to 28:38.440 --> 28:43.880 find you in the end how this thing is, but I have an experience in issue with these parameters, 28:43.880 --> 28:48.200 it's just more like, okay, really, I just send one second of aggregation and I'm done. 28:48.200 --> 28:56.360 Of course, if your process goes down, then you may lose some data, but maybe when things explode, 28:56.440 --> 29:00.440 it's not easy to debug, because it's a very question, thanks. 29:02.440 --> 29:08.120 In the upcoming profiling signal, will there be also automatic and manual instrumentation? 29:09.800 --> 29:14.120 That's, so if profiling signals will have automatic instrumentation, 29:15.400 --> 29:20.360 I'm not an expert in that field, so I will not like to you, I believe, yes, because 29:21.000 --> 29:28.600 I believe it's done with EVP-Ed, but I will set up and maybe there are other talks later, 29:28.600 --> 29:30.360 so maybe they will have you with that one. 29:34.600 --> 29:35.000 Start it. 29:37.480 --> 29:38.520 And so, thank you. 29:38.520 --> 29:40.520 Thank you very much. 29:40.520 --> 29:49.640 So do we have the next speaker present?