073: Heroku in maintenance mode and surfacing observability
Hello, Gophers. You're listening to GO podcast. I'm Demerik Saint Pierre.
Morten:And I am Morten.
Dominic:I was listening to our last episode, and starting to wonder if I should start to only say my first name now that you are saying your your first name. So I mean, it's a very I mean, even Danish people have trouble pronouncing my last name, so I I always just do Morten. It's it's Morten Wistesen.
Morten:Yeah. Right. Yeah. Maybe.
Dominic:No. No. That's it. Yeah. That that was funny the other day I was thinking.
Dominic:So if you are not on the Go podcast on Slack, you know, I I I keep saying that. But, yeah, if you want to join, we are trying to build something there that could be nice. So if you are on the Slack community or the the Gophers Slack community, you know, it would be nice to see you there.
Morten:And you also don't need any facial or governmental IDs to keep chatting there like you do with Discord soon? I don't know if you if you saw that.
Dominic:Well, unfortunately, I tried to go to Discord the other day. I think it was before Christmas. I started to do a lot of Gleam in October, November.
Morten:Mhmm.
Dominic:And they are on Discord. I was having a Discord server myself when I was actively developing static back in early twenty twenty and things like that. Imagine that I I'm not able to return to Discord anymore. And the because of accessibility reason. The craziest thing is they have this this captcha.
Dominic:And normally, when you are blind and you do captcha, you can opt in to have an audio captcha. You know, this is the only way for me because obviously, I cannot tell which of those images are x and things like that. And their accessible captcha is just terrible because it asks you to reverse reverse a word or something like that, but the word isn't I think it's an image that they are using, which I mean is I'm just unable to pass the captcha.
Morten:Yeah.
Dominic:So I I said, alright. You know what? Screw that. Discord is is out out out limit for me. But, yeah, I I was I was not aware that they were going to require that amount of thing.
Dominic:This this is crazy?
Morten:Yeah. I I I didn't read I was like, hey, if I just read some hacker news listings. Right? But, yeah, it seems like they are gonna be starting I think it was from March, to require IDs, at least.
Dominic:Like what? A driver license? Or or or, you know, when you say government IDs, I mean, is, like, are we talking about Social Security number and whatnot? This is crazy.
Morten:I think maybe I need to pull it up right now, but maybe it's biometric data that we were acquiring. So it discards biodata verification.
Dominic:So there will be, like, a face recognition of yourself and things like that?
Morten:Yeah. Let me just see here. So Wow. To access that would be scary. Yeah.
Morten:You need to verify your age. Right? So that is to access age restricted servers, unblood sensitive media, direct speech. Yeah. Facial age estimation video, selfie, or government ID.
Dominic:Yeah. It might be because of the Australia adopting this this law a little couple of months ago where, I don't know, like, under 14, they cannot access social networks and and whatnot. Maybe that's that. I don't know.
Morten:Might be. But they also they did have a breach, right, in Of course. In twenty twenty twenty twenty five, they had an incident where they leaked what was it? 70,000 user IDs. So, I mean, I have my little community for the my community for the for my little framework on Discord, and I don't I'm not gonna show my governmental ID or do any face stuff with with Discord.
Morten:I don't know.
Dominic:Yeah. Yeah. It's I don't I don't know what to think about that, to be frank. It's a lot to ask to to be to be honest. I'm I mean, wow.
Dominic:And and, again, you need to trust them. It's not it's not only to to show that to to a computer software. It's it's also knowingly knows that there's an entity that will now own this this thing. And like you said, anything can happen. So what what what then?
Dominic:I mean, wow.
Morten:Also, working in software, you know you know what security practices means in most companies. Which I mean, to I mean, if you don't work in software, you have probably never seen it. Right? But I mean, do do like, you don't need too many years where you're like, okay. So this is just production access.
Morten:Here we go. Be careful. Oh, yeah. Yeah. Yeah.
Morten:But, yeah, let's see what happens. But, yeah, you don't need that on on the Slack channel. So comment.
Dominic:Yeah. Yeah. You can you can comment still still safe to to to be there without supplying too many things. So one thing we we talked about tools last week, and it's it's funny because last week after the podcast, I I implemented a feature in in in static back end. Basically, I I needed to send email with attachment and things like that.
Dominic:And I have this this small interface that allows me to do, like, dependency injection and whatnot. When you are in production, you can use the SCS services and whatnot. And on development, I I had an implementation that was basically just printing the email to to the terminal. And now I wanted to test and I finally use MailPit. So Yeah?
Dominic:Yeah. That that is that is so nice. I I'm I'm wondering what was that the tool that you were talking about? That Yeah. Yeah.
Dominic:Okay. So so you are reading your email with that? Yeah. Oh, that that's interesting. I I thought it was like a developer tool to capture because now my my system let's say I I've created a a test in Go, and basically, I'm I'm just sending I'm just sending the email to this localhost thing.
Dominic:It's it's deployed on Docker. It's very nice. It's capturing the email and now I I have a web interface or better yet for your test. You have you have this API, a JSON API that you you can just you can just pull the last email that that was sent and you can inspect that from the test point of view. Yeah.
Dominic:And I was like, okay. I I have seen it. And when you mentioned it, I I did not click. But but yeah, it's it's something I wanted to use. But I'm wondering, you know, is it meant to be used in this situation that you are using it?
Dominic:Because now you were telling telling me that you were looking at your, you know, production email and whatnot. This is this is very different than what I I thought
Morten:Maybe it I misspoke then. I know I know it is only development I use Mailpit for. Yeah. Yeah. So it's it it ships with Antutrol in the way so you have a nice way to actually look your emails.
Morten:But, no, I don't I don't use it for any production stuff.
Dominic:Okay. Okay. No. No. Yeah.
Dominic:Okay. That makes sense. I I'm I misunderstood. Okay. So that that that is strange because when I installed it, I I I said, is this this the tool that Martin was talking about?
Dominic:And I I was I was not I was not sure. So that yeah. So that is very nice, to be frank.
Morten:Yeah. It's really and it's like the successor to something called Mailhog that is no longer active. Yeah. I think it's it's and, again, just being a single binary that you can run or a Docker image you can run. They do some basic, what's it called, spamminess?
Morten:No. Yeah. Yeah. Spam testing checks for you, so it works really, really well. And you can I typically sent an HTML version and a and a text version, so it's nice to be able to see both of them?
Morten:Yeah. Yeah. It's a really good tool.
Dominic:Yeah. Yeah. Totally. Very nice. And on a different topic, yeah, I I saw this last week that Heroku was kind of well, it was it was not very clear, but everyone is telling that they are basically shelving the product in a sense.
Dominic:So the the they say is that the the, you know, Salesforce said that they are kind of turning this into a maintenance mode or just applying security and whatnot. They will continue to support and but, basically, what they say is that there will not be any kind of feature development or any any new addition to to the tool. So a lot I'm I'm after that, I I've started I started to, well, hear a lot about that. And a couple of days after that, it was, you know, I'm I'm talking about my social network, feed. People started to say, oh, you know what?
Dominic:I just moved. I'm now on AWS or whatever. Ren render.render.com seems to have the the best approach. They they had the well, maybe maybe it's still there. I I don't know.
Dominic:But they they add this banner on on top and saying, you know what? You know, you can you can move your Iroku for without any downtime or something like that because Render is very very similar to Heroku in in a sense. So I I'm I'm pretty sure I'm pretty sure they they were kind of happy to hear that. So I I missed it, but I I I've seen it very, very quickly on everywhere. It's it's a major news in a sense.
Morten:Yeah. Yeah. I mean, it does it's funny to see all the providers just going out and migrate to us. It's also a lot of DigitalOcean providing, like, migration guides over to them from from DigitalOcean. But it is I mean, it's kind of what like, I don't know.
Morten:I mean, they did put maintenance mode, and, also, maybe people are scared of the of the fact that software can be can be done. I I mean, that may be also be the thing, they're feature complete, and this is, like, just a it didn't sound like that. Maybe that's how they look at it. I don't know.
Dominic:It's it's possible that it's that, but on their on their press release, they are they are clearly mentioning, you know, as as a decision from our stakeholder and whatnot, and they are mentioning their new AI thing on on Salesforce. So it really seems like when you are reading under the, you know, inside the inside the line, I don't know how to say that in English, but when when you are, you know, kind of trying to understand what they are saying, it's basically, you know, we from now on, we are putting all our efforts into this new AI things regarding the self you know, the Salesforce line of product. For me, it was it it was very strange. I did not really knew why exactly Salesforce acquired Heroku, and it just makes no sense because from my very very limited knowledge of Heroku, When you think about that, I believe it's safe to say that they were not having a lot of big organization there, you know, from well, may maybe maybe I'm completely wrong. I don't know.
Dominic:But from what I I can understand or at least see from the outside, it seems to me first of all, they had they had all all the Ruby on Rails developers since the beginning. But after that, it was mainly mainly SaaS, you know, could could be could be, you know, kind of decently sized software as a service companies. But, you know, the the big the big org, I I'm not sure they were there. So I I I'm not sure what was the play exactly for Salesforce because, you know, they are they are focusing on on bigger organization. So it was kind of doomed to to success because they they released some kind of products during during their acquisition time that, one was they, you know, you were able to sync the, you know, the Salesforce contact in in in a postgres database on Heroku for example.
Dominic:That is all good, but you still, you know, you still need to ensure that your customers or at least the new clients that you want to to to get are using Salesforce. And, you know, from my very, very limited point of view, that was two completely different target products. I mean, this that that was that was a little bit strange as as a move.
Morten:Yeah. That's actually true. I didn't I never thought about it, but when you you're talking about it, I remember my first one of my first jobs, we use Heroku. It's also a rails rails startup. And, yeah, it's it's right.
Morten:It's right. Like, enterprise probably I don't I don't wouldn't expect them to be on Rails. No. Sorry. Not on Rails.
Morten:On on Heroku. Right? They're probably already on, like, AWS GCP instead. So it's it's true. It is a weird move.
Morten:But they also cut everything from the beginning. Right? It's like no more free plans, cutting everything down, limit. Yeah. I don't know what the move was.
Morten:Maybe they build on top of it with the new AI stuff, which is, I don't know, scary? It's just an AI platform and a deployment platform. Yeah. I don't know. Is that right?
Dominic:Yeah. I I don't I don't know, to be frank. And I'm I'm I'm wondering also I I know that, well, first of all, you know, if if you take Basecamp, which moved from from the cloud recently well, recently, maybe it's a couple of years back. At some point, they they were they were pretty pretty loud about the about it. And when whenever DHH is doing something, a lot of Rails developers, you know, do that.
Dominic:And and they release this this camel or camel, I I don't know how to say it, kind of tool to help Rails developers that, you know, deploy to bare metal in a sense Mhmm. Very very easily from the command line. So I'm wondering, because I have I have I have to I have to guess that the majority of developers there were using Rails. Maybe maybe not like I don't know. Maybe not like 70%, but probably in the 40%, I would I would guess.
Dominic:Let let's say 35 could be could be a good number. And now I I'm wondering if that could have played a little bit in in a sense because DHH is so how can I say that? You know, when when he does something, there's a lot of people that that need, you know, that wants to do the same, which is which is fine. I mean, this is not a problem, but and and to be frank, the the tool the tool seems to be good. I I've seen a couple of video of of of the tool and whatnot, and it's true that when you don't want to manage a server, this this things, you know, this this small this small helper really helps.
Dominic:Personally, I I really love to maintain servers and whatnot, but there's a lot of developers that don't want to do that. And the the platform as a service as we know them, I don't know. It seems to there's a small shift currently, it seems.
Morten:Yeah. He's very passionate whenever he whenever he finds new stuff. But it's also if if you if you read their their book, what's it called? Shape Up? Is that the book, I I think it is, right?
Morten:Where he's like, pick a battle with someone bigger than you, and that's what they've been doing for, what, twenty or thirty years now. But it's interesting with the whole he also adopted Docker very fiercely. He got into that, and then Kamal came, and then he also got into Linux, and all. So Yeah. He really like, he's he's really shift like, when he likes something, he really shifted.
Morten:I tried out his Omachi OS, which is also cool. But yeah, maybe maybe they are switching. But I think with KML, you would it's only the deployments that you get. Right? You don't get you don't get actual actual server management.
Morten:You'll still need to keep it up to date and and security and all of that.
Dominic:Oh, okay. Okay. Okay. Okay.
Morten:I mean, I'm maybe I'm actually completely wrong here.
Dominic:No. No. I think I think you're right. But my point was mainly that since, you know, Basecamp decided to to say, you know what? The cloud is very pricey.
Dominic:We are going to to go bare metal. And now they'll they released that that tool? I mean, clearly, they they want they want people to to do the same thing and things like that. Then it it's it's funny that you are mentioning Linux and whatnot because I I remember DHH saying a lot of things, you know, maybe not against Linux, but now now nowadays is is, oh, you know what? You should use Linux at at at developer, but, hey, man.
Dominic:We we, you know, we are saying that since thirty years. Now it's it's really cool. It's really nice that now you finally arrive at this, but but come on. I mean yeah.
Morten:Yeah. It is funny. It's funny. Like, you almost like, when he comes with a new opinion, like like, give it give it give it a minute, and then and then see, because he really does come out passionate. I mean, that's also what I think is great about him and base thirty seven seconds.
Morten:Right? They are really passionate about what they do, and Yeah. They try to explore, at least. I think that's really cool from a from a company who's been doing software for so many years that they are not afraid of exploring new paths, which I think is very healthy. Of course, this whole rant around deploying stuff yourself, I I think from vast majority of companies, that is not true.
Morten:I mean, who who he doesn't know. No. We save $5,000,000. Yeah. Okay.
Morten:But the fact that you could spend $5,000,000 on cloud, say something about your scale. Right? I I maybe yeah. Like, I create a company. Let's say I make $50,000 a month with my company.
Morten:I I cannot run my own servers. At least that's probably not a good use of my time. I probably have the money, right, to buy some server racks and everything, but I need either do serverless or as what's what I do more is rent rent some VPSs and then manage those VPSs, right, chuck chuck in a load balancer in front, and then you are really you're good for a long time. So, yeah, the narrative is always super black and white and full on, but it was an impressive switch, right? That must also have been a lot of engineering to because I thought it was petabytes of data they had to move out of is free and maintain themselves.
Morten:So it's also very cool to see how they they take those experiences and turn them into tools like Camel. And not directly, but, like, the the deployment they do themselves is probably very similar to or close to what KML does. I think he talked about that before. So nice to see that that the tooling is is is also being provided to people in a open source format. And that's also, like, with t s DHH and his his new license.
Morten:They I didn't I don't know if you see his old SASE license Oh, no. That he he also picked a fight with the open source community because what like, open source open source means open source. It means here's the thing. You do whatever you want. Like, you do you.
Morten:Right? And what Osashi does is source available. Like, here is the code. Look at it. Run it yourself.
Morten:You cannot use it commercially. Right? And for some reason, he just picked a fight about that, where it feels like, hey. This is not open source. This is source available.
Morten:And then he turned that into a fight as well. And, like, why? You know?
Dominic:Yeah.
Morten:It's a funny it's a funny approach, but it seems to work. Right? He he creates a lot of hype around around their releases.
Dominic:Oh, yeah. Yeah. Totally. I think, you know, even even though it it kind of when when I hear him talk about Linux myself, it's it's like, But I'm I'm happy I'm happy he's doing that because it will it will bring more people to Linux, and yeah. Yeah.
Dominic:That that's always good.
Morten:If you are Linux curious, I can I can highly recommend PubOS? I think I think to listeners in general, if you are if listeners want to get into Linux, PubOS is a really good place to start. Gives you gives you a lot out of the box. It works well with NVIDIA GPUs. It has built in window managers the window manager.
Morten:Yeah. So if you don't wanna go down the Arch route, which is also Oh, yeah. Gotten easier. Like, Arch install is definitely an easier experience than what it used to be.
Dominic:Yeah. I I I ran Arch during I think it was, like, two or three years, and at some point, my system broke completely. After that, I said, screw that, man. I'm going with with the good old Damien. Yeah.
Dominic:And that that machine is still running beside me. Unfortunately, it's it's not my daily driver anymore because of accessibility, but it's still there. I'm I'm running I'm still running a lot of things out of this of this machine. Pretty pretty solid. It's it's impressive.
Dominic:I think correct me if I'm wrong, but PopOS is is beautiful as well. Right? It's nice looking.
Morten:Yeah. It looks really, really good. And they did they did a complete rewrite because it was always based on Ubuntu. And now I now I get into the gist of Linux that I don't really care too much about, so I'm not really that well versed. But something around GNOME or something, I can't really remember.
Morten:But they did a complete rewrite of the
Dominic:Was it Wayland because it it was it was using the new video driver instead of x 11?
Morten:It might be. I think they are I think yeah. The Cosmic, the new one, is on Wayland, I believe. And you can still go back to x 11 if you want to. Okay.
Morten:I can't remember the naming. Is it the disc? The no. It doesn't matter. I can't remember.
Morten:They did a rewrite, complete rewrite from the bottom up, and they created they created Cosmic, which gives you theming abilities also. So you can apply color themes to your to your entire entire environment. It looks great. You have a lot of configs available. For me, the big the big plus is that it comes with the Window Manager and also comes with a SANE desktop environment.
Morten:I used to run iFree, which is also great. Yeah. But there's just certain apps that I never really got working well with iFree. Like, when I had to do a Zoom call, it was weird. And I I don't it's just it's nice to have the the tiling window manager running and then being able to switch over to, like, an, quote, unquote, normal desktop experience.
Dominic:So That's Yeah. That's interesting. I I I was not aware that we could do that. I I was running I three as well. My I missed that setup a lot.
Dominic:You know? I was I was using in in tab mode as well, so everything was full screen for me. I was not using a screen reader back then, which was so quick. I mean, it was Yeah.
Morten:It's really snappy. It's also like, Omachi uses HyperLand, which is also it is faster. It's faster when than what you get on proper is on average, I think. But then again, I I just sometimes, Arch, I feel like it's for me, at least, there was younger Morten's game. He he would love it.
Morten:I I you know, I don't wanna get stuff done, and I don't wanna spend half a day fixing stuff if it breaks. I just wanted to I just wanna be in control of my own computer as I am with Linux, but I also don't wanna be dealing with Bluetooth issues or updating stuff or Yeah. Like, all the also, my recording environment, editing environment is on PopOS. So I have OBS running great. I have DaVinci Resolve running great.
Morten:Would it run better on Mac? Yeah. It it would. But then I'm also forced into whatever whatever Tim Cook wants me to do. Right?
Morten:So I think it's a nice it's really good middle ground. You can do some theming. You can get the window manager. You have folders and and and desktop epic like apps and everything. Yeah.
Morten:It's a it's a really impressive release of what they've done. And it's always funny because people were complaining on Reddit about about the cosmic release saying, this is not a release, and this is not ready. Whereas, like, but they also did something that people have not done for many years. Like, it's a complete Yeah. Write off from scratch, and it is very stable.
Morten:I've been running it since alpha. Yes. Sometimes there is some issues, but the fact that I could run an alpha as my daily driver while I was developing it was was super, super cool. So super stable now, I think. Still some kinks to work out.
Morten:I have some memory leakage issues with NVIDIA, but, I mean, they they it's also the first release of it. It's gonna be it's gonna be just it's gonna be better and better.
Dominic:Nice. Yeah. Do you play game? Do you play game any video games?
Morten:No. Okay. And I want to. I want to, but I have, like, this there's this game called Manor Lords, which is like a medieval city builder, and and I love those type of games. It's like Civilization for me.
Dominic:Oh, yeah. Yeah.
Morten:And, you know, it's the same thing when I need to get stuff done, and I also need to spend time with my wife and friends. If I just, like, start up Civilization, like, in five minutes, five hours has passed. You know? I I get so addicted to it, so I don't really do it anymore. I I typically jump on the I have a PlayStation, do a little bit of gaming there once in a while.
Morten:But I should I should be able to run most games now on PUBOS because they have to Yeah. It's crazy. Yeah. Yeah. That I
Dominic:This this team or Valve has has done an amazing work there.
Morten:Yeah. Yeah. So I I have a few games that I'm like, once once I get once I get some of my projects more finished, I will I will I will install them and start running them, but I think for now, I have I have left it out.
Dominic:Nice.
Morten:Yeah. But but, I mean, the fact that you can game on on on Linux now is also really it's really, really cool to see.
Dominic:Yeah. It's getting there. Especially with everything that is going on with Windows at the moment and and the push of AI in the in the UI, and nobody wants that. And there there's a lot of things that seems to be happening with, Windows 11 that that people you know, they they are basically pushing people over to Linux themselves. This is this is interesting to see.
Morten:It's the it's the year of the Linux desktop. It's finally it finally came.
Dominic:Now now that DHH is preaching that and Yeah. Yeah. I mean
Morten:We get it. Finally here.
Dominic:Yeah. It's interesting. Alright. So let's let's switch a little bit to the main topic. So I wanted to talk a little bit about observe observability and, you know, what what you are doing with that,
Morten:what
Dominic:I'm doing with that and things like that. So first of all, I I yeah. I've I've seen that in your in your project. You're you're using a library. I'm I'm not using anything in static back end.
Dominic:This this is where I am at at the moment. I'm deciding what I am going to to use and whatnot. So well, my first question is, did did you, you know, did you took a lot of times to decide how you wanted to add that to your project?
Morten:I mean, it's something I played around with for for a long time and and explored in different other projects or client projects I've done. And I always kind of just defaulted to OpenTelemetry because it's it's a thing I've been following for for for quite a long time now, and most of their go stuff is in stable. I bit I can't remember if the if the login package is still in beta. But it's basically what I always used in in projects. So I'm actually not super aware of other Right.
Dominic:No. But that that's that's the standard. Right? It's
Morten:a Yeah. That's the standard. Right.
Dominic:The, you know, suggested way to do that. My my first issue with with adding tracing to to my system is that, you know, I I don't want to trace all the requests. You know, that that that is not really something that I would like to do. Because let's say that you are starting to to put some tracing in all your request. I mean, this is a lot of allocation.
Dominic:This is a lot of things that needs to happen in addition to your normal call code. So are you are you first of all, are you are you tracing all the request that that happens in your system? And and if if not, then why you know, what requests are you are you picking? What's what strategy are you are you taking?
Morten:Right now, it's all requests gets a trace ID. Mhmm. But it's not all of them that are, what's it called? What's the right word? Telemarized?
Dominic:So so you're you're not sending all of them to your to your your system that will that will log them in in a sense?
Morten:No. I mean, it's more because if you work with the telemetry package, you can you can like, you can dig down almost as much as you want. And I don't do that for most requests. That's more only for, like, billing stuff or sign up stuff where I will go through the entire chain and add observability and traces and do spans and all of those Mhmm. Kind of elements.
Morten:But all requests are part of this package. Right? So they they do at least get a a trace ID. And then I can't remember the sampling rate I do. Also, I think for my projects, I don't have sufficient volume that this
Dominic:is gonna
Morten:be an issue.
Dominic:So you're you're picking a certain percentage of the request already. It's already like that. Yeah. Basically, random request, and you say, you know what? I want to pick, I don't know, 2% of of my request or something like that.
Morten:I actually, now I'm gonna have to check because I can't remember. Because this is also I think as you preface this, would would I used to use a lot of the Grafana stack for my observability. And maybe they have done better jobs late since I checked the last time, but that documentation has always been interesting. Yeah. I don't know if that's the right word.
Morten:It takes some time to set up, and then you have a lot of stuff that you don't need. But when I I did that, I just self hosted a Grafana dashboard. I self hosted the ingestors. Is that the right word? Ingestors, I think it is.
Morten:Whatever whatever it is, Grafana, because then they had some version now some tool, and then you have to use the new tool versus Alloy, I believe. And the I don't know. I don't know. So I I it's basically I'm I'm working on a new setup for this.
Dominic:Okay. Okay. Interesting.
Morten:That's building I hinted towards it. I'm building this SaaS project for self hosting service and application where I'm I'm gonna be exploring a new telemetry setup where I wanna be using wow. What's the it's a ClickHouse. I believe it's ClickHouse
Dominic:Oh, yeah.
Morten:Yeah. To store the data. So the whole idea is that whenever you create a server, you get, I guess, stored servers metrics, which is just some basic from node exporter to have an idea of, like, are your server healthy or not? And then create a server setup that uses click house or cluster of click houses, whatever, and ingest all the telemetry data there. I haven't fully deployed it yet, so I don't really know if it's something that is worth exploring.
Morten:But at least that's kind of my hypothesis that this is probably going to be good enough for at least my use cases and also probably other people's use cases. Again, it depends on the volume you have. Right? If you have a ton of volume, just go with Datadog and pay the Datadog fee. I don't know.
Morten:I'm going a little bit off track here, but that that is, like, kind of the the main approach I'm taking right now.
Dominic:Yeah. Yeah. I I it's not like I have I have a lot of volume or whatnot. It's just that, you know, at the moment, it's it's basically just logging for me, and I I feel like I'm ready to to go one step above, which is, you know, which is in implementing some some starting point for tracing and whatnot. So I I was starting to I was starting to try to decide, you know, my strategy.
Dominic:So so far, it's either well, I will I will start the tracing if if there is any kind of HTTP headers in in in the request, or I will I will trace everything and just at the end, you know, they they call that the tail I don't know. I don't I don't know what to call that. But, you know, let's let's say that there there's a 500 errors at the at the end and this this is what you will you will get sent to to your your telemetry server and whatnot. I I don't don't know the words exactly, to be frank. It's it's it's all it's all something I I'm I'm just getting into.
Dominic:So basically, like like I was saying, I'm I'm basically building smaller application, but I feel like it's kind of important to have those at least, and I'm I regret that now. Yeah. Because after after the fax is harder to to add than having that from having, you know, good telemetry from from the beginning of a project.
Morten:It's also, you know, whenever you have an issue and you don't know, like, it's just the fact that if you have metrics and you have trace data, it's just so much easy to dig into what's happening on your on your application because my in my framework, I also set up for the database layer. Mhmm. So you can actually also connect it a trace with a set of queries that happens on the in the database layer, which is also really nice of tracking it when you start to have a lot of a lot of data going
Dominic:and whatnot.
Morten:Yeah. Exactly.
Dominic:So you can basically grab grab the the SQL query and and try to optimize it. That that's what you are saying?
Morten:Yeah. You would have, like, an idea of execution time and is it what's this is it called a waterfall graph? I can't really remember. But you will see, like, the
Dominic:Yeah. Yeah. The entire the entire path. Yeah. Yeah.
Dominic:Yeah. Yeah.
Morten:And I we basically, the setup was with one of my latest clients where the guy who configured it all also used Grafana to set it up, and they were dealing with a lot of a lot of events simply from the fact that they were doing a lot of they are basically doing Google Sheets and AI. So there was a lot of things that needed to go through, and they also just self hosted the Grafana stack, and it seemed to work work fine. So yeah. So you you get, like, the entire layout, but the the thing is just the UI part that I mainly am exploring now because the I think the OpenTelemetry setup is really, really nice. And the if you just put what's it called?
Morten:What's it called? Prometheus and and and NodeExporter on your VPS, then you are, in terms of metrics, you are quite good. And then you need something that can ingest OpenTelemetry data with the I think there's a go go one you can just run. So that, I think, is fine. It's just the the UI part, I think, that's often a little bit tricky because that can be quite complex.
Morten:There is, from ClickHouse, a new UI that also looks interesting that I'm considered playing around with to to be able to better query because the whole thing about looking up your logs, for example, with the Grafana stack, has never really, at least to my knowledge, not been good. Like, you couldn't search your logs, which is which is big. Woah. Okay. I mean, you can't maybe I'm speaking out of out of my ass here.
Morten:I'm not
Dominic:Well, I I was looking at Onicom myself, which which is another, you know, viewer, I guess. And that that one seems to be nice to search, But I I was in the impression that that was a a basic functionality because it's basic when something something is is happening. You you need you need to search something. You need you need to start by that. You you all you don't always have the trace ID to to say, oh, you know what?
Dominic:This this request or this thing, this event that started it all, this is the culprit. I need to debug that. You you often don't really know this this ID. Right?
Morten:No. I mean, it it I think my experience was that it's probably possible, but not easy. I don't think they do, like, a lock indexing in the same way that you get with Honeycomb.
Dominic:Okay.
Morten:So you could probably still kind of get search working, but I don't think it's the same way where you can actually curate and look for all these kind of things. It is probably more around at least how I've always used it is get the trace ID, get the span ID, and then you get the the associated locks with with that trace ID, which is fine sometimes. But, again, as you say, sometimes you need to search for certain certain elements.
Dominic:Certain event or at least an error or things like that. So what what is your your your use case usually? So I guess that you are you using SLOG or SLOG from to to to do the login yourself?
Morten:Yeah. Yeah. Yeah. Okay. Full on So I
Dominic:I was using LogROS before, and now I I would I think I would like to switch, but I again, this is this is another thing that I would have loved if if the if Slug would have been there in in the Go ecosystem since the beginning because now Yeah. I'm not even sure that I want to change, but there's there's not much reason to change. But just to, you know, just to remove one dependencies is nice. I I I know that it's not really observability. It's just it's just the the logging part I'm talking about, but I I would guess that everything start there.
Dominic:You you might you might, you know, see something in your log, and and now probably the the trace ID is there. If if there's an error, at least you you can start from from that, but it's not it's not always the the situation.
Morten:No. I mean, Slack is great. It's it's I think it's awesome. Also, like, as most as much as possible staying in the standard library is is is how I try to to do it. And it has really good integration.
Morten:I'm also using Echo for the I don't know how Echo a router, or is it a webcam? I'm I'm not even sure anymore. Yeah. Don't know
Dominic:how to define it. I I
Morten:don't know. No. No. But it it works well with Echo.
Dominic:But wait. Wait. Wait. What what what was the the the decision be be in choosing Echo? Was was it was it because when you started the project well, the the new the new router functionality that we have now in in Go was not there, for example, extracting parameters from the URL and whatnot.
Dominic:So was that was that the main reason?
Morten:No. It was months of hard research. No. It was basically it was basically what I used from the beginning, I think. So it is I just always been familiar with Echo.
Dominic:Okay.
Morten:I can't remember. There's not really any justifiable reason I can remember because I I believe in the beginning, I was looking at I was actually using Gorilla when I started building web applications in Go.
Dominic:Yeah. Yeah.
Morten:And then that was on maintain, and then I looked at Fiverr and Echo, and I've I can't remember why I didn't like Fiverr. So I went with Echo, and then I've been using Echo for, I don't know, five years or something now. Four years, I can't remember. So it's kind of like a use what you know Yeah. Kind of thing.
Morten:Mhmm. At least I mean, it's true that you get the standard library does more now. I'm actually in one of my projects, I'm building a small API to run on servers where I'm doing this purely in standard library just to see how how it feels and how it works. Right? But I think for these bigger ones, especially when it's more framework y as I do with Enduro, it's very nice to have some of the stuff being being taken away and taken care of even though it's maybe not incredibly So
Dominic:what it brings you, a couple of middlewares, I would I would expect? Like Yeah. Be be because, yeah, the the standard library is kind of crazy these days. So I'm I'm wondering because I myself, I was I was also using Geralt first. After that, I kind of dabbled a little bit with Chi, and quickly quickly just dropped everything and just just went.
Dominic:Even even even when the the standard library was not able to to extract parameters and and there there was a lot of things, you know, you were supposed to handle the HTTP verb yourself and and, you know, all of that. I don't know. It was not really distracting me at some point.
Morten:No. Oh, yeah. I will have to I will have to rewrite a big project for web with the standard library for me to really know, to be honest with you. Yeah. Because now I'm thinking of it, I I can't really answer your question.
Dominic:Yeah. Yeah. Understand what you say.
Morten:Yeah. It's Because because it's one of those, like, I know it's there. I've looked a little bit at it. I haven't actually used it enough because I'm sure you're right. It's not that I don't think you are I don't think that it could it could probably replace it.
Morten:Right? It's also just a this is like, especially when you look at the framework, this is just built so much around Echo as a as a routing engine that it would require version two to switch to it. So Yeah. Which might be relevant in the future. I'm definitely not against ripping out stuff for for better.
Morten:I think because the main thing I'm getting now that you don't I don't think you get with the standard version is that you can define it with you have to define the the routes and the methods in a string. Right? You have to do, like, what get and then the route in a in one string, which I will probably want to I like to abstract that away. I think that's maybe a little bit too loose for my taste, but it's such a minor thing that you can always build something around that to to fix it or fix fix it, change it. I know.
Dominic:So it's it's not a string in when you're defining a route in echo?
Morten:I mean, they with the v five, you do, like, an add route, and then they have an echo route, which is now a struct. And then you specify like, it's just a struct that has method path name Oh. Handler middlewares, which which yeah. You just pass the the method from the the what is called HTTP package from the standard library. Mhmm.
Morten:I built a routes package for the framework, so all the routes are typed. So you pass that. You pass the name. You pass the handler. Yeah.
Morten:But yeah. I I don't actually, I think you have a good point. I don't think it's that that much you get anymore, maybe.
Dominic:Well, it it it's it's already built like that. I mean, you know, that that's the same thing from what I'm saying about about the tracing. You know, LogROS is is all implemented, and it's there. It's everywhere. And but but again, I mean, I'm I feel like the more I can I can remove dependencies, the the better the project will will survive the long term?
Morten:Yeah. Yeah. Yeah. So Yeah. Definitely.
Morten:That's also what I'm what I'm doing in the model layer of the framework is, of course, we're building around the SQL C, which I'm using the p d x package for Mhmm. Which have their own implementation of these p d types. So I'm trying to contain all of those types to the model layer, so everything else uses just the standard types. So trying to do some conversion there, which may be not really necessary, but I just only having a type that is from the standard library in general, I think, is really, really nice because it is just, you know, it's maintained.
Dominic:So you're you're mentioning is it is it about the null columns? So you now you are with you you you have, like, SQL dot null int, for example, or null string. Is that is that what you mean?
Morten:Yeah. Yeah. But also, it has a bunch of types that he needs to have or the the library needs to have, right, to to handle the the layers between Go and and, in my case, Postgres.
Dominic:So it's not using a float 64, for example, if if there's a decimal on on the database?
Morten:Not necessarily. And that's that's the thing. Right? So if I jump in and do I can't even remember if I have any. But, yeah, he it they they do use standard library types, but there's as soon as you get into something that's maybe if if you provide, like, a not null, they will maybe to like, if you do, I don't know, deployment ID integer not null, if they will just use int 32.
Morten:But if you use something like, say, created at with a time timestamp with time zone or something like that, they will use their own type.
Dominic:If that's
Morten:pg type timestamps, for example Yeah. If they do so again, with the let's say you have a deployment ID, in my case here, that is nullable, then they will use, like, a pgtype so they can handle the the null part, so the value and if it's valid and those kind of things. So I just try to contain those two two to one layer of the application so they're not everywhere so that everything else stays in the in the standard library as much as possible. Yeah. It was a long way of saying I I agree with you.
Dominic:Yeah. Yeah. Yeah. Yeah. But yeah.
Dominic:So this this is this is mostly where where I am, you know, regarding the the observability as well. So I would have liked to to add that at first, and now now I'm at I'm at a crossroad where I think I would like to have that. But, yeah, again, it's it's still, like it adds a lot of complexity, to be frank. It's it's not it's not as easy as I would expect.
Morten:No. No. No. It does take a lot. And, yeah, I've I've spent a lot of time on it some time ago, which is unfortunate that I didn't do it now so I could talk about it because now I can't really remember a lot of the decisions.
Morten:But it it it I think it's a it's a huge it is complex. There's a lot of moving parts that you need to be aware of and and consider. And you can you can mostly not deal with it if you are ready to pay. If you if you go some like, I don't know how expensive Honeycomb is, but I know that Datadog Datadog can get you if you're not careful the same. Yeah.
Morten:But then then it's literally just setting up, you know, the adding adding or, basically, just adding the the OpenTelemetry package to a little bit of setup, and then give give that to Datadog, and then you you are you are
Dominic:Yeah. But I no. I I think I would I would like to to have my own thing, but, yeah, it's a it's a separate thing. Would it would it would it be graphene? I I don't I don't know yet, but but, yeah, I think so where where are you actually viewing those those results yourself?
Dominic:So is it like a completely different server and whatnot? So how do you how do you deploy the the viewer of of that thing?
Morten:For a long time, that was just Grafana. So Grafana, they like, their web interface, can you can run yourself if you want. Just deploy it as a docker image, set up your sources, and then set up your different metrics, views, and then then you get then you get the data showcase. And I used to have some templates for it as well, so I could I could see memory usage, storage, how much storage that was left. I could also see number of requests, two hundreds versus four hundreds and five hundreds, those kind of things.
Morten:But that was that was mainly just built in Grafana Yeah. Which for for most of my stuff, is just too complex. I think it it like, it's if you are into to graphs and data and statistics, Grafana must be your your dream your dream product because you can do so much, and it's really complex. It can at least be really, really complex. Right?
Morten:So now yeah.
Dominic:Sorry. No. No.
Morten:Go ahead. Oh, no. So that was just so that's what I but what I've been doing and also why I wanted to move away Yeah. From it because I think it's too complex. That's also why I wanna explore ClickHouse because there is I can't remember if it was the guys behind SQLC who created this, but there is, like, an SQLC, like, package for ClickHouse.
Morten:Okay. So my kind of thesis that I wanna test out is that if I put all the metrics and traces and logs in ClickHouse, then I can build my own web UI around this this setup, right, and then use Datastar to just live update a graph or
Dominic:statistics or
Morten:those kind of things. And then for my use cases and fill out of, like, how I like to do things, I think it's gonna be much more maintainable, but also much more
Dominic:Right. It will be more accustomed to what what you want to see. And I'm not really I'm not really after graph myself. It's it's mainly trying to diagnose a request from everything that happens, and I'm talking about there there there might have been a lot of go routines that started there. So I I just want, like like you were saying, the and now I I I don't recall the the name the the name that you use, but the the you know, just just seeing a a good trace of what happens with with the request, especially if it failed and especially if I am to have a distributed system at some point, now I'm able to cross boundaries and and now just pass this this trace ID from one, you know, one services to the other.
Dominic:And now I have a clear view of what what is happening with this this trace ID.
Morten:Yeah. I've a a a great, great resource on this topic is from a guy called Luca Palmieri, who wrote a book called sewage production Okay. Which is a Rust book. It is fantastic. He does a great job of, first of all, teaching you how to write a Rust application.
Morten:And this is a Rust web application, which I now think is way overkill. But but it it is great because he has done a lot of this, like, heavy distributed systems in Rust, and he has a really good chapter around setting up telemetry, which is what I base most of my setup on as well.
Dominic:Nice. So
Morten:Rust, he uses more, I believe, it's more macros in his setup for for how he telemorized his his Rust application. But it's basically the same idea that you you you pick certain flows, certain request flows, whatever Yeah. Add some spans to it because I'm not gonna necessarily have any distributed system in the same sense. I might have multiple instances of my applications, but it's not like a bunch of microservices. Yeah.
Morten:Yeah. So, basically, the trace ID is just an entry point for me, whereas I can see from your point of view, if you if you have multiple services, tracking that request across those things is also really, really helpful. Yeah. Yeah. Totally.
Morten:But that should be that should be very feasible with with, yeah, with OpenTelemetry and then, you know yeah.
Dominic:Yeah. Yeah. Yeah. This is what I'm looking for. I'm I'm just, you know, I'm just not super clear yet on on what and how, but but, yeah, it seems to you know, it's it's not that complex, but you still need to think a lot about, you know, what kind of data you want to to see and how frequent you want to, you know, the sample to be taken and whatnot because Yeah.
Dominic:Yeah.
Morten:And but also Honeycomb should be a really good product. Have have played around with it a little bit, but not really used it a lot. I really mainly used just we used Grafana. Right. But but Honeycomb should be really great.
Morten:And I know I used to talk to this guy who did what did they do? It also doesn't matter. But they basically was a YC startup who worked with creating flame flame crafts
Dominic:Okay. I
Morten:believe, and got I don't they joined Grafana. They got bought by Grafana, but they did some really cool stuff. And they also had a lot of collaborations with Honeycomb, so I know from that guy that he was like, yeah. They're actually doing really, really good job. Right?
Morten:Yeah. So maybe Honeycomb is also a good a good good option to to explore in this case.
Dominic:Yeah. I I haven't started to to look at anything. Maybe maybe we should we should bring Matt Ryer to to get some Grafana point of view.
Morten:That would be great. And also to get a jingle.
Dominic:Like Yeah.
Morten:When the guitar I miss that.
Dominic:Yep. Cool. Alright. Alright. So any anything any closing thoughts?
Morten:No. I don't think. Not on not on telemetry, at least. I think we probably covered covered that quite well.
Dominic:Yeah. Yeah? Yeah. Sure. Yeah.
Dominic:Yeah. Yeah. I think I think it's it might be it might be something we we revisit at some point with with a guest and once I have I have implemented my my my, you know, my setup and whatnot. So so it's it's more exploring for me at the moment. Yeah.
Dominic:So that shows that I have not worked in in in project where that was that was a key part. So usually yeah. Usually, again, the the amount of volume is is usually very low for me. So, you know, good old good old logging is is pretty is pretty working at the moment. But but yeah.
Dominic:I it's something something I need to to check out at some point.
Morten:Yeah. I there's also like, in my first couple of jobs, a lot of them were on, like, GCP. GCP's Cloud Runner Cloud Run, which just gives you all of this basically out of the box. You have to configure the telemetry and a bit of stuff. Right?
Morten:But they have nice locking interfaces you can search. So it's also in later projects, have started to to work with it more and and set it up. But I think it's become more, yeah, more normalized in the last couple of years. I think that's one of the the people who worked on OpenTelemetry was actually out saying that, okay. Now people are actually thinking about it, which didn't used to be the case.
Morten:But they were like, why why do I need this?
Dominic:Yeah. Yeah. Totally. Yeah. Yeah.
Dominic:I think, yeah, I I think Elixir has something built in, which is really nice. It's all it's all it's all it's all there. You have a dashboard with your and you're you're seeing all all the your small pieces. I'm you you it's it's funny because you you talk about Rust, you know. For me, Rust and Elixir was was something that I wanted to jump into.
Dominic:For some reason, I have so many difficulties liking Rust or at least giving me the the kick in the pants to just just build something with it myself. So so yeah. That so that that that's why, you know you know, Gleam and Erlang the the Erlang ecosystem for me is is also extremely interesting. I I if I were to be frank, I I will say something crazy being on on this on this podcast, but if I were in need to build a really distributed system, I don't think I would go with Go. I think I think I would prefer to go with with, you know, Gleam probably and Elixir.
Dominic:Gleam and they're like, built on the LTP the the the LTP the the this this all machinery that is this is there for you. But I don't know what it is with me and and Rust, and I I invited John Arundel a couple of times in here, and he he started to to build some some Rust course and whatnot, and he wrote one book at least. And I I I kept saying, you know, it's I I don't know. I don't know. I I have I just can't.
Dominic:But but I started gleam and I was like, wow, this language is is my language. This I I like it better than Go in in terms of syntax, in terms of idea, but it's it's not super dare yet, I would say. I I don't know either what what it is. No. I keep I keep returning to Go for for most most of most of my development.
Morten:I think with the Rust thing, I think for best majority of application is just bringing a jackhammer when you just need, like, a a little normal hammer. Right?
Dominic:Yeah. Yeah.
Morten:Because I I think my skills and my knowledge around computers and software has gained tremendously for me exploring Rust as a programming language. Because it it you get more you get lower level, and you and I think a lot of the people who write Rust are super, super talented people who do some crazy asyncs, bare bones hardware no. Yeah. Hardware programming stuff. Right?
Morten:They really require some deep knowledge. So I think you can learn a lot, but I I just got tired of you know, it felt like every time you had to write 50 lines of code to accept a little request or something, there will be one line in Go. And then I didn't really need the performance gains. I didn't really need this I didn't need the extra security. The only damaged no.
Morten:Not damaged. What's it called? Maybe this is a Danish saying, like, you say your work damaged if you if you do a certain thing because of your work. I really I really love how you can express certain states in brass food food type system Yeah. So that you cannot get into a certain state Yeah.
Morten:Which is really, really nice. And there's this guy this person who wrote a article called pass no. No. Okay. I can't remember.
Morten:I will share it in the Slack if I find it, but where you literally just, like, don't validate. Ah, yes. Like, don't validate pass or something instead. So the validation is literally built into the logic of your of your types. So you have email type that cannot be an invalid type because it's And that's great.
Morten:I think it it makes a lot of sense on paper. But if you start to to do this in in practice, it just creates so much much so much extra work for you that is it really is it really worth that extra work compared to what you get for for the applications? Like, do they really need it?
Dominic:Yeah. Yeah. I for me, it's not really the type system because, like I said, I I really like Gleam, and Gleam is a is a functional, you know, type type language. And it's it's basically the very similar to what you can do in Rust, but maybe a little bit a little more interesting. It's for me, it's not really type.
Dominic:I I came from I did I discovered functional programming with Elm in in 2017, and this is where I I really started to to like this approach myself of building. For me, Rust is is all like like you said, and it's it's strange for me as well because I'm kind of fatigued sometime to have the browser being the output of of the program that I'm building. Because I've been doing that for, like, twenty seven twenty whatever. Twenty six years now, twenty seven year. The I always built web application in a sense.
Dominic:So it's it was always like like I said, the browser was was the end result. And even though I I did a lot of c at at, you know, when I when I was learning my craft at first, I even have a course on the assembly at first, but I'm I missed that, but it's I don't know. There there's something with Rust that I I it's like I don't have the time to or maybe I I what would be easier for me would be to have a a job. So if, you know, if I were to start at at a place and they use Ross and now I'm I'm paid to, I don't know, to force myself to to use the language, it's like compared to Gleam Gleam, I I I done I done that in my spare time. Like I was saying in in October, November, I I did I did the exorcism, really fall in love with the language.
Dominic:Wow. But again, me being me and mostly building web application, I don't know. There there's a lot of yeah. There's a lot of small things that that I found difficult. But but again, I mean, this is this is interesting.
Dominic:I I I still I still want to to make sure that I'm writing a lot of code in a lot of different languages still. Yeah. But yeah.
Morten:I think the other one I've been looking at was sick just because I wanted to,
Dominic:I don't
Morten:know, explore something different.
Dominic:Yeah. Totally. If they if they can if they can release the v one at some point. I don't I don't know where they are where they where they are, but No. It's true.
Dominic:Yeah. Yeah. This this seems this seems interesting to to be honest. Yeah. It seems simple.
Dominic:I I I love c myself. I can I can see myself writing some c even even today? Even with with the the crazy memory management and whatnot. But that's one one thing that brought me to go actually because it was so in my view, it was so similar to see when when you start to look at some code at first, especially with the struct and whatnot, you say, oh, you know, when you are coming from a object oriented world like I was from .net, and you you saw Go back back then in in 2014, it was like refreshing a little bit. But I I don't have that with Rust.
Dominic:I don't in terms of syntax, oh, yeah. Okay. The unwrap and things like that, the question mark and yeah. Compared to the if error not equal nil and go I mean, there's a lot of nice small things, but it's not enough. It's not that.
Dominic:And and I don't know what to build. I I don't need anything built that would requires me to use Rust. So that that's like you were saying, that's mainly one problem that I have as well.
Morten:Yeah. Yeah. I think it's also yeah. It depends on where you go in in in in your career, I think, because there's for me, the the reason why I just stick with go now is because I'm I'm just it's more the product building part Yeah. That I'm interested in.
Morten:So I'm also like, that's all because, like, we've said a little bit sick. I was like, yeah. I could build this, like, little desktop application that could help me transcribe stuff and maybe do some video stuff. But then again, I'm like, ah, you know, I have other stuff to that I wanna that I wanna do more. And I figured, like, five years ago, I would have just jumped on sick.
Morten:Yeah. To learn sick. So there's also I think it's you have these periods where, like, I I I was sure in, like, a couple of years, I'm probably gonna be like, okay. Now is the time for a new language to learn. Yep.
Morten:Yeah. I don't know. It's it goes through go through waves.
Dominic:Yeah. Totally. Totally. But Go Go is is really nice. But but yeah.
Dominic:Okay. Cool, Martin. So I think that's that's a wrap.
Morten:Yeah. Me too.
Dominic:Alright. So talk next week.
Morten:Talk next week. Bye bye. Bye.
Creators and Guests
