WEBVTT 00:00.000 --> 00:07.000 Hello. 00:07.000 --> 00:08.000 Morning. 00:08.000 --> 00:09.000 Welcome. 00:09.000 --> 00:10.000 Stop moving tables. 00:10.000 --> 00:14.000 My name is Brian Bohrum. 00:14.000 --> 00:18.000 I'm a Prometheus maintainer. 00:18.000 --> 00:21.000 I've worked on the code for about seven years. 00:21.000 --> 00:24.000 My day job is at Grafana Labs. 00:24.000 --> 00:25.000 Hi. 00:25.000 --> 00:26.000 I'm young. 00:26.000 --> 00:29.000 I guess I can call myself a team member now too. 00:29.000 --> 00:34.000 The PR is still open, so go vote for it. 00:34.000 --> 00:35.000 Sorry. 00:35.000 --> 00:37.000 Do I need to repeat? 00:37.000 --> 00:39.000 All right. 00:39.000 --> 00:43.000 My day work I work for Red Hat on the OpenShift Monitoring 00:43.000 --> 00:48.000 and the rest of the time I try to contribute to Prometheus. 00:48.000 --> 00:50.000 All right. 00:50.000 --> 00:56.000 So who's here? Who's hard of Prometheus? 00:57.000 --> 00:59.000 Okay, that's good. 00:59.000 --> 01:01.000 Who uses Prometheus? 01:01.000 --> 01:03.000 Everybody nearly? 01:03.000 --> 01:06.000 Who uses Prometheus three? 01:06.000 --> 01:10.000 I don't like maybe five percent. 01:10.000 --> 01:12.000 Is it okay? 01:12.000 --> 01:16.000 Is it income down to report your bugs? 01:16.000 --> 01:21.000 Is anyone like totally new to Prometheus in the room? 01:21.000 --> 01:24.000 Nobody's opening up to that. 01:25.000 --> 01:29.000 So we don't need to go through the architecture. 01:29.000 --> 01:31.000 So that's good. 01:31.000 --> 01:34.000 That's going to save some time. 01:34.000 --> 01:36.000 I did want to put this. 01:36.000 --> 01:40.000 The QR code takes you to the documentary. 01:40.000 --> 01:45.000 Produced, which is really nicely like produced and edited 01:45.000 --> 01:48.000 and so on a really high quality video. 01:48.000 --> 01:52.000 And it has the history of Prometheus. 01:53.000 --> 01:56.000 So I recommend that if you've never seen it. 01:56.000 --> 02:00.000 Yeah, I mean, the point of the architecture. 02:00.000 --> 02:03.000 Prometheus is a one process solution. 02:03.000 --> 02:06.000 And there's a lot of other things out there. 02:06.000 --> 02:08.000 And someone was just saying a few minutes ago, 02:08.000 --> 02:11.000 you could add Kafka and stuff like that. 02:11.000 --> 02:14.000 So the big thing about Prometheus is you just run it. 02:14.000 --> 02:15.000 One thing it runs. 02:15.000 --> 02:17.000 It does your monitoring. 02:17.000 --> 02:19.000 And you're alerting. 02:19.000 --> 02:26.000 So that's the kind of the key part of the architecture to stress. 02:26.000 --> 02:27.000 Right. 02:27.000 --> 02:31.000 And so not everybody has to go out and watch that documentary. 02:31.000 --> 02:33.000 We have a little bit of history here. 02:33.000 --> 02:36.000 So Prometheus started that soundcloud. 02:36.000 --> 02:40.000 It's over 12 years ago now. 02:40.000 --> 02:44.000 As you can see, after the initial commit, 02:44.000 --> 02:46.000 there was around the 2.0 release. 02:46.000 --> 02:49.000 And when I say around, like I'm talking years, right? 02:49.000 --> 02:57.000 It joined the CNCF, but also graduated the CNCF as the second project ever in each categories. 02:57.000 --> 03:01.000 And after that, there was a lot of focus on stability. 03:01.000 --> 03:03.000 There's non-purpose support versions now. 03:03.000 --> 03:08.000 And of course, now we have with 3.0 since November last year. 03:08.000 --> 03:09.000 Yeah. 03:09.000 --> 03:12.000 I think that's it for a quick one. 03:13.000 --> 03:15.000 Right. You release. 03:15.000 --> 03:18.000 Obviously, there's a lot of stuff in there. 03:18.000 --> 03:20.000 You know, you have the cornerstones there. 03:20.000 --> 03:24.000 Obviously, we won't ever be able to talk about all of those in a single talk. 03:24.000 --> 03:26.000 Like, some of these are a single talk. 03:26.000 --> 03:29.000 In fact, there are talks out there already. 03:29.000 --> 03:36.000 This just takes you to a YouTube query that has the current committed three talks. 03:36.000 --> 03:40.000 This will update over time, of course, which is nice. 03:41.000 --> 03:48.000 So today, we'll pick out a few of these and we actually look at some of what we consider breaking changes. 03:48.000 --> 03:55.000 Of course, with a major release, we can finally release some breaking changes, which is nice after seven years. 03:55.000 --> 04:00.000 If you're interested in how to migrate, we have a very nice migration guide there. 04:00.000 --> 04:05.000 That should help you out with everything, and if not, let us know. 04:05.000 --> 04:09.000 And I guess with that, we can start on the actual talk. 04:10.000 --> 04:12.000 Right. 04:12.000 --> 04:15.000 So we're going to go through a few of the changes. 04:15.000 --> 04:26.000 So there was a change to reggaex handling, which is certainly one of the most memorable ones for me. 04:26.000 --> 04:30.000 So, yeah, what does that do? 04:30.000 --> 04:36.000 Everyone uses Prometheus, so you should all know what this does. 04:37.000 --> 04:40.000 So I'll give you multiple choice. 04:40.000 --> 04:43.000 So it actually doesn't do anything. 04:43.000 --> 04:47.000 It's choice A. Who thinks that? 04:47.000 --> 04:51.000 There's a trick question. 04:51.000 --> 04:57.000 Who thinks it goes through every possible value carefully checking each one? 04:57.000 --> 05:01.000 I'll say no, nobody thinks there's no option C. 05:02.000 --> 05:10.000 So in prior to 3.0, it was option B. 05:10.000 --> 05:17.000 So it's completely obvious that it shouldn't do any work, because dot star matches everything. 05:18.000 --> 05:36.000 But, surprise, the goal code that was used to implement reggaex, the dot does not match a new line. 05:36.000 --> 05:43.000 So every time you run one of these queries, it would scan every possible value to see if it had any new lines in it. 05:43.000 --> 05:44.000 And you would not see those. 05:44.000 --> 05:49.000 But, nobody has, like Hansa, who's got new lines in their labels. 05:49.000 --> 05:54.000 Right. So this was just a stupid mistake. 05:54.000 --> 06:04.000 But, for me, Thias is, as culturally, is passionate about breaking changes, about compatibility. 06:04.000 --> 06:10.000 So we did not fix this bug until a major version. 06:11.000 --> 06:16.000 So that's, for me, that's the most memorable thing about 3.0. 06:16.000 --> 06:22.000 Once you decide to do a major version, we didn't do a major version just to fix this bug. 06:22.000 --> 06:29.000 But once you have decided, and we kind of decided it's about time, like seven years. 06:29.000 --> 06:32.000 That seems like long enough for a major version. 06:32.000 --> 06:37.000 Once you've decided you can break things, well, the floodgate's open. 06:38.000 --> 06:41.000 So we certainly fixed this one, and I was very happy about that. 06:41.000 --> 06:45.000 And those kind of queries, if you have a lot of values, they go faster. 06:45.000 --> 06:47.000 So, it's good. 06:47.000 --> 06:52.000 Okay. So there were a few breaking changes in 3.0. 06:52.000 --> 06:58.000 Luckily, they all fit on one slide. 06:58.000 --> 07:00.000 I could read it. 07:00.000 --> 07:03.000 We're not going to go through them. 07:04.000 --> 07:06.000 But, yeah, just to give you some idea. 07:06.000 --> 07:08.000 So that's the release notes. 07:08.000 --> 07:13.000 That's a list of everything that was changed. 07:13.000 --> 07:19.000 Some of them are sort of fixing stupid things. 07:19.000 --> 07:26.000 And other ones were things that you could already make it work better with a feature flag. 07:26.000 --> 07:28.000 But you had to know that. 07:28.000 --> 07:30.000 You had to turn on the feature flag. 07:30.000 --> 07:31.000 That was zero. 07:31.000 --> 07:36.000 We basically turned on all the feature flags that you really should have been using. 07:36.000 --> 07:40.000 But we did not turn on for compatibility reasons. 07:40.000 --> 07:46.000 So yeah, that's the breaking change story. 07:46.000 --> 07:50.000 Let's move on to elementary. 07:50.000 --> 07:52.000 We had a talk earlier. 07:52.000 --> 07:55.000 Some of you might have been in about elementary. 07:56.000 --> 07:57.000 It's the big thing. 07:57.000 --> 08:02.000 The big standard for instrumentation. 08:02.000 --> 08:05.000 As of 3.0, 08:05.000 --> 08:13.000 Prometheus by default will accept metrics in the open telemetry line protocol, 08:13.000 --> 08:16.000 OTOP format. 08:16.000 --> 08:21.000 So one of the big things about that is that the metrics are being pushed to Prometheus. 08:21.000 --> 08:23.000 It's always been a pool model. 08:23.000 --> 08:27.000 It sits there, pulling metrics in from wherever they come from. 08:27.000 --> 08:31.000 And OTOP is a push model. 08:31.000 --> 08:33.000 Because I don't know. 08:33.000 --> 08:36.000 I'm not going to be rude about OTOP telemetry. 08:36.000 --> 08:37.000 Only got three minutes. 08:37.000 --> 08:38.000 Yeah. 08:38.000 --> 08:42.000 That's going to take an hour out of the talk. 08:42.000 --> 08:44.000 So anyway, it is a push model. 08:44.000 --> 08:50.000 So Prometheus now accepts metrics being pushed to it in OTOP telemetry. 08:50.000 --> 08:55.000 So that's nice. 08:55.000 --> 08:57.000 One more for me. 08:57.000 --> 08:58.000 Yeah. 08:58.000 --> 09:04.000 So we decided to support all characters. 09:04.000 --> 09:08.000 That's why we refer to this as UTF8. 09:08.000 --> 09:17.000 The one character that matters to telemetry is the full stop, the period. 09:17.000 --> 09:23.000 The level names in open telemetry always have a dot in them. 09:23.000 --> 09:28.000 And Prometheus has always banned the use of the dot. 09:28.000 --> 09:33.000 And we thought, well, if we're going to allow one more character, why not let's just allow them all. 09:33.000 --> 09:38.000 So you can now write Chinese and your labels, you can write accented characters. 09:38.000 --> 09:43.000 Isn't the level names where they were previously restricted? 09:43.000 --> 09:51.000 But we do make you quote anything that is outside of the previous allowed set. 09:51.000 --> 09:55.000 So just put it in quotes like that example. 09:55.000 --> 10:03.000 That's in the migration guide and there's a whole special page on what do we mean by UTF8 support. 10:03.000 --> 10:06.000 So again, we throw it in the floodgates. 10:06.000 --> 10:08.000 You can use any character. 10:08.000 --> 10:12.000 You can name your metrics after emoji. 10:12.000 --> 10:19.000 I did submit a talk to coupon called status equals poop. 10:19.000 --> 10:23.000 But it was not accepted. 10:23.000 --> 10:25.000 It's a shame. 10:25.000 --> 10:27.000 Oh, I'm doing one more. 10:27.000 --> 10:28.000 Why am I doing so much talking? 10:28.000 --> 10:29.000 I don't know. 10:29.000 --> 10:30.000 I think I do. 10:30.000 --> 10:31.000 You do this one? 10:31.000 --> 10:32.000 Yeah, okay. 10:32.000 --> 10:39.000 So we're looking at another breaking change that was interesting. 10:39.000 --> 10:43.000 So we changed range selectors or how they behave. 10:43.000 --> 10:47.000 In Prometheus 3, you have a left open interval. 10:47.000 --> 10:52.000 Sorry, left close. 10:52.000 --> 10:53.000 Can't confuse. 10:53.000 --> 10:59.000 So if you have a sample that coincides with the left border of your interval, you get that sample. 10:59.000 --> 11:02.000 That's can be surprising in a lot of ways. 11:02.000 --> 11:07.000 So we wanted to change it, you know, make it left open. 11:07.000 --> 11:10.000 So that sample will not be included. 11:10.000 --> 11:16.000 And, you know, we thought it's a breaking change, but what can happen, right? 11:16.000 --> 11:20.000 The feature actually causes impact mostly academic. 11:20.000 --> 11:21.000 Oh, that's nice. 11:21.000 --> 11:27.000 So you have to get your time stamps really accurately allowing you to notice a difference. 11:27.000 --> 11:30.000 In the migration guide, we're sort of a little bit more insightful. 11:30.000 --> 11:33.000 We explain this a little more, you know, we do this. 11:33.000 --> 11:37.000 This might impact for some subcrew use cases. 11:37.000 --> 11:40.000 Some front ends that do certain things. 11:40.000 --> 11:43.000 But really mostly tests will be impacted, right? 11:43.000 --> 11:45.000 Easy peasy. 11:45.000 --> 11:48.000 Turns out, oh, maybe we should show a query like this. 11:48.000 --> 11:51.000 So this is an example of such a subquery. 11:51.000 --> 11:52.000 Right? 11:52.000 --> 11:56.000 If you don't know subquery, this is my pointer. 11:56.000 --> 12:01.000 This down here makes it a subquery, so it takes to you request total. 12:01.000 --> 12:06.000 Looks back for one minute and gives you a range vector with one minute steps. 12:06.000 --> 12:10.000 So you can see already, I forced this a lining here, right? 12:10.000 --> 12:14.000 We have a one minute interval, but we only have one minute steps. 12:14.000 --> 12:19.000 So the second sample would fall right on this one minute, a quarter. 12:19.000 --> 12:23.000 And it turns out that these queries are actually more common than we thought they would be. 12:23.000 --> 12:26.000 So some people have problems now. 12:27.000 --> 12:31.000 But I want to use this query now to actually look at the new UI. 12:31.000 --> 12:38.000 Because the new UI, besides looking much nicer, brings a really cool new feature. 12:38.000 --> 12:42.000 And that's the tree view and explain. 12:42.000 --> 12:45.000 So we have this query here, as I said, right? 12:45.000 --> 12:48.000 I wasn't lying, this doesn't return anything. 12:48.000 --> 12:52.000 And Prometheus 2, this would actually give you a graph. 12:52.000 --> 12:55.000 But we can fix it, and Prometheus 3. 12:55.000 --> 13:00.000 We add one millisecond to the range query, and cut out. 13:00.000 --> 13:03.000 Nice, right? 13:03.000 --> 13:07.000 But if we turn on the tree view now, we can actually look at what's happening. 13:07.000 --> 13:12.000 So the tree view gives you all the components of this query. 13:12.000 --> 13:17.000 And you can actually select the individual components and get the results for that. 13:17.000 --> 13:24.000 So I selected the subquery here, and you can see, oh, we only get a single value at a time. 13:24.000 --> 13:26.000 This is the old query, right? That now breaks. 13:26.000 --> 13:32.000 If we add that millisecond, it's two samples and the rate suddenly works. 13:32.000 --> 13:34.000 So that's really neat. 13:34.000 --> 13:38.000 I think that's the coolest feature of the new UI. 13:38.000 --> 13:40.000 I'm not a UI, I guess you can tell. 13:40.000 --> 13:43.000 There's more cool stuff, right? 13:43.000 --> 13:50.000 This actual explain tab also gives you a really nice explanations when your query doesn't work. 13:50.000 --> 13:57.000 So here, you know, that's a different query, but basically we try to match something that doesn't match. 13:57.000 --> 13:59.000 We don't get any results. 13:59.000 --> 14:05.000 In the old world, you know, you had to dissect your query and figure out, oh, where does this go wrong? 14:05.000 --> 14:09.000 The six-lane feature now tells you, oh, there's a label set that don't match. 14:09.000 --> 14:10.000 Fantastic. 14:10.000 --> 14:17.000 We should give a shout out to Julius Moot, who single handedly implemented all this. 14:17.000 --> 14:19.000 And his company Prom Labs. 14:19.000 --> 14:23.000 So if you want to, if you want to buy any previous training or consultancy, 14:23.000 --> 14:25.000 reach out to him. 14:25.000 --> 14:29.000 Well, he'll show up in one more slide or call it out. 14:29.000 --> 14:31.000 Right. 14:31.000 --> 14:37.000 The last one I wanted to show is this dreaded duplicate match group error, right? 14:37.000 --> 14:39.000 Who's here, have that ever? 14:39.000 --> 14:42.000 Not too many people, surprising. 14:43.000 --> 14:48.000 If you ever run into this, this can be a bit tricky to debug, but now with explain. 14:48.000 --> 14:51.000 It'll tell you, because it has access to your data. 14:51.000 --> 14:56.000 It'll tell you actually the match group and where it has multiple samples, 14:56.000 --> 14:58.000 and you can go out and fix it. 14:58.000 --> 15:00.000 So how neat is that? 15:00.000 --> 15:06.000 That's for the new UI, or back over to you. 15:06.000 --> 15:07.000 What? 15:07.000 --> 15:09.000 Went wrong. 15:10.000 --> 15:13.000 This is going to be exciting, because I can't remember what's on the next slide. 15:13.000 --> 15:15.000 What did go wrong? 15:15.000 --> 15:16.000 Nothing. 15:16.000 --> 15:17.000 Nothing, right? 15:17.000 --> 15:18.000 Well, no. 15:18.000 --> 15:25.000 So measuring what went wrong by what did we have to put out an emergency patch release for? 15:25.000 --> 15:27.000 Well, not match, actually. 15:27.000 --> 15:30.000 There were only three bug fixes in 3.0. 15:30.000 --> 15:31.000 About one. 15:31.000 --> 15:36.000 Nobody, nobody deploys 3.0.0.0, right? 15:37.000 --> 15:40.000 Always wait for the patch release. 15:40.000 --> 15:49.000 So the first one was actually introducing the thing that people, the most people had problems with, 15:49.000 --> 15:52.000 which is the left open for sub queries. 15:52.000 --> 15:56.000 But we did want to make it consistent, and we didn't know it was going to. 15:56.000 --> 16:00.000 I personally had never seen one of those one-minute colon one-minute queries, 16:00.000 --> 16:03.000 until the bug reports came in. 16:04.000 --> 16:08.000 Why would you do that? 16:08.000 --> 16:13.000 Yeah, so there was a memory leak, and we had forgotten to support 16:13.000 --> 16:16.000 UTFA on one endpoint. 16:16.000 --> 16:18.000 But that was it. 16:18.000 --> 16:22.000 That was 3.0.0.1. 16:22.000 --> 16:25.000 Right, so no back to yourself. 16:25.000 --> 16:26.000 Yeah. 16:26.000 --> 16:31.000 So with Prometheus 3, there was other stuff going on. 16:31.000 --> 16:34.000 Basically, somebody highlighted that, 16:34.000 --> 16:37.000 oh, somehow contributions are dropping. 16:37.000 --> 16:38.000 It's not as active. 16:38.000 --> 16:39.000 Let's change that. 16:39.000 --> 16:46.000 So we want to make the Prometheus community more welcoming to members or more contributors. 16:46.000 --> 16:52.000 And as part of that, there is actually a new governance document, I guess, we call it. 16:52.000 --> 16:54.000 In the works, that's not ready yet. 16:54.000 --> 17:00.000 But look out for that, and now's the time to join Prometheus. 17:01.000 --> 17:08.000 At Prometheus, I showed a slide how sort of the contributions developed running up to 3.0. 17:08.000 --> 17:11.000 And now we can actually look at 3.0. 17:11.000 --> 17:16.000 So this is the contributions by contributor for any given release. 17:16.000 --> 17:20.000 We can see the release numbers there at the bottom. 17:20.000 --> 17:23.000 And I mean, looks pretty good, right? 17:23.000 --> 17:29.000 3.0 was definitely a success in gaining new contributors and more contributions. 17:29.000 --> 17:37.000 This next graph there, this next not so bumpy one is Christmas New Year's. 17:37.000 --> 17:39.000 Everybody was off. 17:39.000 --> 17:44.000 And yeah, I mean, one contributor here sticks out a bit, right? 17:44.000 --> 17:50.000 But even without this person, this would still be quite a significant increase. 17:50.000 --> 17:53.000 And this is actually the UI addition here. 17:53.000 --> 17:58.000 So that was merged as one big lump, so that's joyous as work on the new UI. 17:58.000 --> 18:02.000 I said he would show up again. 18:02.000 --> 18:06.000 The same thing we can see, it's not just code, right? 18:06.000 --> 18:09.000 Not just a new contributions, but reviews are really important. 18:09.000 --> 18:13.000 So this is from the death stats, CNCF death stats. 18:13.000 --> 18:17.000 You can look at review contributions. 18:18.000 --> 18:21.000 Okay? 18:30.000 --> 18:35.000 Okay, I got to do this one handed and in the wrong direction. 18:35.000 --> 18:37.000 Give me a sec. 18:42.000 --> 18:44.000 I take the mic. 18:47.000 --> 18:51.000 Let's go over here. 18:55.000 --> 18:56.000 Okay. 18:56.000 --> 18:57.000 Back to it. 18:57.000 --> 19:01.000 Oh, and the script is tiny if you want to make that bigger. 19:01.000 --> 19:06.000 Very important to a project, right? 19:06.000 --> 19:12.000 If you don't want to contribute code, but you know, say, you know, an integration that we have from 19:12.000 --> 19:16.000 this course is for services coverage, so you can review stuff. 19:16.000 --> 19:21.000 We like outside input and pointing things that could be done better. 19:21.000 --> 19:24.000 So yeah, this also worked for 3.0. 19:24.000 --> 19:29.000 This, the bump here is really sort of this year at the end. 19:29.000 --> 19:36.000 Again, we have a bit of a drop over the change of the years, but hopefully we can get that back up in the coming year. 19:37.000 --> 19:39.000 So yeah, that was really nice. 19:39.000 --> 19:44.000 And I think it's worth a round of applause for all the contributors who made 3.0 possible. 19:55.000 --> 19:57.000 So, call to action. 19:57.000 --> 19:59.000 I mean, you can join us. 19:59.000 --> 20:02.000 This is picturing a PR. 20:03.000 --> 20:08.000 We just agreed to add 22 more people. 20:08.000 --> 20:10.000 Actually, we're doing two things. 20:10.000 --> 20:12.000 We're changing the governance. 20:12.000 --> 20:19.000 So that there's maintainers and a very much smaller steering committee. 20:19.000 --> 20:25.000 And so it used to be a bit of a problem adding people because it made voting harder. 20:25.000 --> 20:28.000 Because you have to get more and more people to agree. 20:28.000 --> 20:34.000 But because we're changing to have a steering committee, we're very happy to add. 20:34.000 --> 20:40.000 That's just the people a little bit easier to read. 20:40.000 --> 20:48.000 So these are, these are people who a lot of them were instrumental in getting 3.0 out the door. 20:48.000 --> 20:55.000 And a number are just working away on some bit, maybe not premises itself. 20:55.000 --> 21:00.000 Maybe one of the exporters, something in the ecosystem. 21:00.000 --> 21:04.000 Anyway, these people have been working hard. 21:04.000 --> 21:07.000 And we are adding them to the Prometheus team. 21:07.000 --> 21:10.000 We could have another round of applause for that, couldn't we? 21:10.000 --> 21:11.000 Congratulations. 21:17.000 --> 21:20.000 All right, we'll let it open for questions now. 21:20.000 --> 21:22.000 Anyone has any? 21:22.000 --> 21:28.000 Yes, I think we repeat the question. 21:28.000 --> 21:29.000 Oh, you could repeat the question. 21:29.000 --> 21:30.000 You did a shout. 21:30.000 --> 21:33.000 This is more of a joke, but when is more does zero? 21:33.000 --> 21:35.000 Oh, when is four does zero? 21:35.000 --> 21:37.000 Well, yeah, we have talked about that. 21:37.000 --> 21:40.000 We don't want to be another seven years. 21:40.000 --> 21:51.000 We thought, yeah, we sort of discussed, should we say that it's planned for one year? 21:52.000 --> 21:56.000 And then it probably take two. 21:56.000 --> 21:59.000 Yeah, but not another seven years. 21:59.000 --> 22:03.000 Yeah, quicker than that. 22:03.000 --> 22:05.000 More questions? 22:05.000 --> 22:09.000 If you can, please stay seated for five minutes because if everyone leaves, 22:09.000 --> 22:13.000 we won't understand the questions anymore. 22:13.000 --> 22:19.000 Oh, wow, we can just not talk too closely into this one. 22:19.000 --> 22:27.000 So, are there any changes planned for, for the open metric specification, 22:27.000 --> 22:29.000 connecting to the Prometheus 3? 22:29.000 --> 22:35.000 Are there any changes to the open metric specification related to Prometheus 3? 22:35.000 --> 22:37.000 I think that's a no. 22:37.000 --> 22:44.000 So there is work underway to produce an open metrics too. 22:44.000 --> 22:49.000 And maybe I should explain open metrics. 22:49.000 --> 22:55.000 So open metrics was an attempt to produce a standard, which was not tied. 22:55.000 --> 22:59.000 People like being avoiding vendor locking, right? 22:59.000 --> 23:00.000 That's a big thing. 23:00.000 --> 23:03.000 Nobody wants to be locked into a particular vendor. 23:03.000 --> 23:10.000 And in this world, Prometheus gets treated as a vendor, which is kind of nuts, 23:10.000 --> 23:12.000 because there's no company behind it. 23:12.000 --> 23:14.000 There's no billionaires. 23:14.000 --> 23:16.000 You're not, right? 23:16.000 --> 23:17.000 No, yeah. 23:17.000 --> 23:19.000 No, you just got added to the team. 23:19.000 --> 23:22.000 So, yeah, check in a week. 23:22.000 --> 23:29.000 So, yeah, so the open metrics was an older, older than 23:29.000 --> 23:33.000 open telemetry attempt to make a standard. 23:33.000 --> 23:36.000 And it wasn't as wide ranging. 23:36.000 --> 23:39.000 It was just metrics, didn't do traces and logs. 23:39.000 --> 23:42.000 It didn't catch on like open telemetry did. 23:42.000 --> 23:47.000 So effectively, all the excitement is with open telemetry. 23:47.000 --> 23:50.000 But on metrics still exists as a standard. 23:50.000 --> 23:56.000 And there is now a group underway aiming to produce an update of that, 23:56.000 --> 23:58.000 which I think will be a 2.0. 23:59.000 --> 24:01.000 Hi. 24:01.000 --> 24:06.000 Often times, I have difficulty reasoning about like writing unit tests is a, 24:06.000 --> 24:09.000 I find it a bit challenging from for Prometheus alerts. 24:09.000 --> 24:12.000 I was just wondering how we can use the new explain UI, 24:12.000 --> 24:18.000 or if you could throw the sample data to the explain UI that we throw to the unit test 24:18.000 --> 24:22.000 to help make it easier to reason about the unit test. 24:22.000 --> 24:24.000 So, let me take one. 24:24.000 --> 24:27.000 I'm not sure I get the question. 24:27.000 --> 24:29.000 I get it. 24:29.000 --> 24:33.000 So the question is about unit testing rules like the learning rules. 24:33.000 --> 24:37.000 And you notice that there's cold new information in the UI, 24:37.000 --> 24:41.000 and could that information be used to help unit test your rules. 24:41.000 --> 24:43.000 Is that the question? 24:43.000 --> 24:46.000 I think the unit test, like throwing the sample data to the unit test, 24:46.000 --> 24:48.000 make it easier to reason about unit test. 24:48.000 --> 24:52.000 Oh, you want to bring sample data from the UI into your testing rules? 24:52.000 --> 24:53.000 Oh, the other way around. 24:53.000 --> 24:56.000 Like if I have sample data that I throw at the unit test, 24:57.000 --> 25:02.000 you want to use the UI to explain why your unit test is failing. 25:02.000 --> 25:03.000 Yeah. 25:03.000 --> 25:04.000 Ah, got it. 25:04.000 --> 25:05.000 Right. 25:05.000 --> 25:10.000 I mean, if you have a Prometheus on top of your unit test data, 25:10.000 --> 25:12.000 you can probably do that. 25:12.000 --> 25:13.000 But it doesn't. 25:13.000 --> 25:15.000 I mean, there's a, I think the, 25:15.000 --> 25:17.000 I think the context is prompt tool, 25:17.000 --> 25:20.000 which has like, canned data for rules. 25:20.000 --> 25:23.000 And I think I would kind of like that as a good idea. 25:23.000 --> 25:25.000 So submit a feature request. 25:26.000 --> 25:27.000 Please. 25:27.000 --> 25:28.000 Or PR. 25:28.000 --> 25:29.000 Or just implemented. 25:29.000 --> 25:30.000 Yeah. 25:34.000 --> 25:37.000 We got one more up here, then we can come down here, I think. 25:37.000 --> 25:41.000 Is GRPC planned for OTRP? 25:41.000 --> 25:43.000 Is GRPC planned for OTRP? 25:43.000 --> 25:45.000 Since you said you supported. 25:45.000 --> 25:47.000 Nobody knows. 25:47.000 --> 25:48.000 Nobody knows. 25:48.000 --> 25:49.000 No. 25:49.000 --> 25:53.000 I don't think at this point, but certainly not an expert, 25:54.000 --> 25:56.000 but maybe the expert is running on. 25:56.000 --> 25:58.000 I think that's another one. 25:58.000 --> 26:01.000 We just need someone to do it. 26:01.000 --> 26:03.000 I don't think there's any philosophical objection 26:03.000 --> 26:08.000 to supporting GRPC for OTRP. 26:08.000 --> 26:12.000 Is there any development on the agent mode of Prometheus? 26:12.000 --> 26:15.000 So if I want to have like multiple Prometheus instances 26:15.000 --> 26:19.000 that remote white to the same central Prometheus instance, 26:19.000 --> 26:22.000 there was this experimental agent mode. 26:22.000 --> 26:25.000 But it never really worked that way that I expected it to be. 26:25.000 --> 26:29.000 It's still through in memory like nothing. 26:29.000 --> 26:34.000 And I ended up using the open-term issue collector instead. 26:34.000 --> 26:38.000 So we did have a slide about performance. 26:38.000 --> 26:40.000 Is that there? 26:40.000 --> 26:43.000 Yeah. 26:43.000 --> 26:47.000 So your question was about agent mode. 26:47.000 --> 26:50.000 And we declared it stable, didn't we? 26:50.000 --> 26:53.000 It's no longer experimental. 26:53.000 --> 26:58.000 But it is not relatively any more performant than it was before. 26:58.000 --> 27:07.000 However, Prometheus III, so these charts kind of show that the CPU usage 27:07.000 --> 27:14.000 and memory usage of Prometheus III is better than like a couple of years earlier 27:14.000 --> 27:17.000 and way better than 2.0. 27:18.000 --> 27:21.000 So yeah, we did a lot of work on performance. 27:21.000 --> 27:26.000 It has got better basically with every release. 27:26.000 --> 27:34.000 But the agent mode is not like a spilt like white speed demon. 27:38.000 --> 27:41.000 We've got two more minutes. 27:41.000 --> 27:44.000 Are there more questions? 27:44.000 --> 27:46.000 No, one over here. 27:46.000 --> 27:48.000 What are you talking? 27:48.000 --> 27:50.000 Let me say please do try it. 27:50.000 --> 27:52.000 We just released 3.1. 27:52.000 --> 27:54.000 3.2. 27:54.000 --> 27:56.000 For this last question. 27:56.000 --> 27:57.000 Hi. 27:57.000 --> 28:01.000 Any plans to allow exporting OTLP directly? 28:01.000 --> 28:03.000 So if I have a data store, for example, 28:03.000 --> 28:08.000 I don't accept OTLP that I don't need to translate it from remote by two. 28:08.000 --> 28:12.000 There aren't any definitive plans. 28:13.000 --> 28:16.000 Do we take the remote right two slides too? 28:16.000 --> 28:17.000 Yeah. 28:17.000 --> 28:20.000 So we have remote right two. 28:20.000 --> 28:25.000 That sort of wasn't really a plan feature for three, but coincided with it. 28:25.000 --> 28:29.000 And that does allow to even think about this now. 28:29.000 --> 28:33.000 So it's way more structured that can do the, you know, 28:33.000 --> 28:36.000 you know better than I do where the issues were. 28:36.000 --> 28:39.000 But that would allow implementing this now. 28:39.000 --> 28:40.000 Yeah. 28:40.000 --> 28:44.000 I mean, we looked at, so the question was about, are you going to implement remote, 28:44.000 --> 28:47.000 sorry, OTLP writing from Prometheus? 28:47.000 --> 28:54.000 And I certainly find it hard to tell what's the difference in value between that 28:54.000 --> 28:56.000 and the open telemetry collector. 28:56.000 --> 29:00.000 So if you can achieve what you need with the open telemetry collector, 29:00.000 --> 29:04.000 which paced in a whole bunch of Prometheus code, 29:04.000 --> 29:07.000 you know, go do that. 29:07.000 --> 29:14.000 If you cannot achieve what you need to, please file an issue. 29:14.000 --> 29:16.000 And once again, yeah, join us. 29:16.000 --> 29:17.000 Please contribute. 29:17.000 --> 29:20.000 Make Prometheus a more wonderful thing in future. 29:20.000 --> 29:22.000 Thank you very much. 29:22.000 --> 29:23.000 Thank you.