083: Lisette, inspired by Rust, compiles to Go with Iván Ovejero
Hello, Gophers. I'm Dominic StPierre. Today, Morten is not there, but we have a special guest today. Ivan well, Ivan. I asked you at the beginning.
Dominic:How how do you pronounce your name? And I I I did the mistake. Oh, yeah. So so let's let's introduce yourself a little bit. So you are the creator of Lisette?
Ivan:Correct. So, yeah, for a very, I guess, brief introduction, I'm a software engineer at AnyDen, which is this, workflow automation platform. And I've been working with Go for a couple of years. And it then is mostly a TypeScript shop, but, there are services, usually backend services, usually hidden backend services that are typically written in Go. And I ended up creating Lisette out of my love for Go, I would say.
Dominic:That's interesting. So would you say that it was the first time that you came across Go during, you know, working on those services and whatnot?
Ivan:I would say so, yeah. So basically I've been at my company for going five years now since it was a very little startup and there was really no nothing other than the main product. And as the company grew, we started creating, like, I don't know, developing cloud, having back end services to handle, the email setup, having back end services to handle, for example, some proxies to handle arbitrary code execution for users. Usually, use cases that required very small microservices in Go that ended up being, I guess, my first exposure to the language. I usually got the impression that, like, the first time I used it, I was just amazed by the simplicity and cross compilation, the compilator, very fast compilation times, very easy deployment story.
Ivan:But I guess I wanted a bit more.
Dominic:Were you did you trip about the verbosity compared to to TypeScript? So you're saying that the other services were were in TypeScript. So I I guess that's a pretty well, for some people, the verbosity can be can be a hard thing to to go over at first.
Ivan:So there is that. I usually like to think that, like, syntax is downstream from the semantics. Verbosity, I guess, would be kind of the first level. I'm also not exactly, like, extremely happy about TypeScript either. Like, would really like for JavaScript to offer algebraic data types and many of the things that are today, for example, in the TC39 proposal stage.
Ivan:But I always had this impression, yes, especially with Go, I guess, which basically enforces you to do certain style of error handling, which enforces you to be there's a there's a cost to being so consistent, which is you have to be on board with the consistency that's being enforced, even if it's extremely verbose. But I would say verbosity was not the first thing that came to mind. It would be more, I guess, the lack of expressiveness or, I mean, it's the subjectivity, right? If you have been exposed to certain like, ML family languages, you tend to really miss them when you're not working in them, Everything being an expression, having exhaustive pattern matching. In the case of Go, for example, you have pattern matching.
Ivan:Sorry, you have switch statements that are not exactly exhaustive, but if you're not really paying attention, you might think they are. And then in the future, you add another, let's say, pseudo variant to your pseudo enum, and you don't realize that the compiler is not actually there to save you from yourself to tell you, hey, this new variant that you just added, you need to actually be handling it in all of these different places. Otherwise, it's gonna fall through and you're gonna have a surprise at 3AM.
Dominic:Totally. Totally. So were you at all in so where where it came, this this ML I I don't know how to say that exactly, but when when did you did you discover program functional programming and whatnot?
Ivan:So, honestly, I don't quite remember when I got exposed to Rust. I have not used Rust for work at all. I think there were some there was already some resistance to introducing Go initially at any end. There wasn't that. Like it was typically a typescript shop and Rust was even further away.
Ivan:So I don't think I ever had an opportunity to get there. With Rust, I think it was more just playful experimentation and seeing what the all the fuzz was about and then realizing, wow, this is like all of these nice like it has a steep learning curve, but the affordances are just they win you over. Many of the things that Lisette has, it's basically just porting or migrating many of the things that I think Rust got right to the Go world. Just not reinventing the wheel, not revolutionizing anything, just reusing things that have been tried and proven. So honestly, I think it was just, you know, one weekend you want to try out, I don't know, OCaml, you give it a try, one weekend you want to try out a new language and you give it a try.
Ivan:And sometimes it really clicks. Basically you see why the semantics are the way they are and how they drive you to this, you know, the phrase that pit failure to be the semantics of rust, they bring you to the pit of success. They make it hard to shoot yourself in the foot, which is essentially what I'm trying to achieve with Lisette, to make it really easy for you to see what you need to do. And otherwise, the compiler is gonna yell at you, which is better than something failing at run time and you discovering it in the middle of a production incident.
Dominic:Yeah. Oh, yeah. Totally. I came across it it it yeah. We we will return very quickly to to Lisette.
Dominic:Let let me just, you know, say that. So I came across Elm in 2017 myself. Never done any functional programming before that. And the the learning curve was was kind of, you know, it it was not it was nothing compared to Rust or anything like that. But Elm being a a, you know, close to what pure functional programming is, gave me gave me some new perspective on what or you were talking about the expressiveness expressiveness and whatnot, and I was like, wow, this is this is very interesting.
Dominic:And from that time, you know, I never never really touched a typescript until very recently, sadly, but for for the front end. All of that to say that at some point, I I cross posted on on the Elm Reddit and the Go because I I was a fervent, you know, go back in user at that time. And Mhmm. I I couldn't I could not believe that those two were were not more closed because of the tooling at at that time. Elm, you know, were, very good at at, you know, giving great error messages and whatnot.
Dominic:But but for me, it was it was all about the tooling. I I was having a great experience on the back end with the tooling with Go, and I I was having a just a superb experience on on the front end now. And I I I I wanted to to plug the two together, and and that was not really well received from both side. It was it was kind of funny. And and funny enough is that I I tried to build a a a transpiler myself in, I I would say, 2020.
Dominic:It was called Flang. It was a it was a personal project. So basically, it was trying to have the Elm syntax, but it was going to transpile to go. And when I yeah. So it was it was funny because I I when I was doing research for for Lisette, there there there was so many people that that were saying that they were working on some kind of transpiler or compiler or something like that to to have different syntax for Go and whatnot.
Dominic:So when when I when I saw Lisette, was like, this is this is interesting. There there is something there. There has to be a lot of people that might not be exactly. That might be looking for something else. And Rust is is a beast in itself.
Dominic:And and the it's it's it's a strange world that you seems to wants to, to connect together. So why why you know, why this this this project came to life? What what happened exactly?
Ivan:So I think with all of these kind of projects that become kind of like larger than you expect, it became as just a weekend project to see, wouldn't it be fun to have, I usually had this experience of whenever I was working with Go, I was thinking fondly of my time with Rust during the weekends and vice versa. When I was working with Rust, for example, sometimes I didn't want to deal with, certain kinds of complexity that did not really apply to my, like first that maybe went over my head because my level of Rust wasn't where it should have been like back in the day as it usually happens to everyone, I guess, who starts off with Rust. But it did happen to me. So I was working in Rust, was missing the simplicity of Go, I was working in Go and I was missing the safety and the expressiveness of Rust. And I kept thinking what, like, what first of all, why has nobody done attempted this before?
Ivan:So I started looking, are there any kinda, like, toy programming languages trying to marry the two together just to try to find some, middle ground between safety and pragmatism? I did find a couple. Gleam? Okay, Gleam is an interesting case, but for a different reason. So if we're specifically about bringing these ideas to go, I was thinking more of Borgo, for example, which I think it was a project done two years ago by a developer called Marco Santeligrini, who basically is one of the projects that I really looked into and I thought, wow, this could be something.
Ivan:In the case of Gleam, I approach it from a different perspective. I think, like personally, I'm a huge fan of Gleam and of, Louis Pielfeld, the author. I think that Gleam is in a way kind of Lisette's spiritual sibling. So Gleam brings all of these, amazing taste, simplicity, ex superb developer experience to working in the Beam virtual machine. But it is, I guess, I mean, I'm not I don't know the first thing about Erlang to be able to say, like, exactly what would be the Go equivalent experience of working there.
Ivan:Yeah, to be completely honest, Gleam has been one of my biggest inspirations. Think from the language perspective, even from the, I guess, let's say community side, like, Gleam is one of the, if not the biggest, most recently launched language that has, like, so many people know and at the same time, if you think about it, it has zero institutional backing. And in my mind, that is just spectacular. It's amazing how a language like that without Microsoft, without Google, without anyone, just completely independent, can reach, the size and the magnitude that it has reached today. But, yeah, taking it back again to the to the programming language side, that is essentially what Lisette is for Go.
Ivan:Right? What, what Beam what Gleam tries to be for, the Beam runtime. Lisette is what is the equivalent for the Go runtime, trying to bring safety, expressiveness, and hopefully a bit of transparency. So the same idea that you can transpile from one language to the other. And if in the future you feel like, I don't know, Lisette is not more for you or you don't want any lock in, you just eject, grab your code, and you continue on, with your project as you were.
Ivan:So in my mind, I think, yeah, I've taken inspirations from, I think with projects like these, you're just standing on the shoulders of giants. And in many cases, just a mixture of all these inspirations that end up cooking up in your mind. But it all started with just, something that I wanted to exist.
Dominic:So at first, was was that something and I'm I'm not even sure. Is it a transpiler? Is it a compiler? What what am I what am I outputting at the end of the day? Let's say let's say I'm I'm building something very simple with Lisette.
Dominic:What is the output of that? Is it straight up Go code file?
Ivan:Yes. So, like, if you if you if we get really bit antic about it, it is a transpiler, not a compiler. I've heard people who have said the distinction doesn't really matter. If you're working in Lisette, you do a list build, you get a target directory, imagine as in Rust, for example. And in that target directory, you can have your Go project.
Ivan:And as part of list build, you're also building that Go project into a static binary. So, like this is basically so fast that you cannot even, it's not as in TypeScript that at least the TypeScript based TypeScript compiler that takes a while. In this case, it's really fast to just create Go from list and to create a static binary from Go. In many of the cases, you're gonna notice as you play with Lisette, it basically many for many of the things, we delegate to just the go tooling, is amazing and which really would not make any sense to, to just rewrite, for any purpose. It's not the case that Lisette basically writes its own, static binary and so on.
Ivan:That that was never the purpose. The purpose was this safety and expressiveness layer on top of what I already think, Go does perfectly well.
Dominic:So from there, I I guess that this this this weekend idea, you you started with a simple tokenizer and and tried to build an AST that that you wanted to to have. Was was that part long?
Ivan:Did you did did you
Dominic:did you reach the the basic syntax quickly?
Ivan:Yeah. So in my mind, I already knew what I wanted. I wanted basically a subset of Rust, the parts that I felt would add the most value. What I did was yes, as you said, basically, that's how we all started. I had already written some interpreters before just in fact, I think I was too naive when I started.
Ivan:I thought, yes, a tokenizer, a parser, like, the emit layer, these are not extremely complicated parts of writing a compiler. And so the thought came to my head, well, these parts are not that complicated. I guess it's probably not that complicated to add a type system on top of this. And that's where I was extremely, extremely wrong. It's one of those, I guess experiences where you're happy to have been wrong because maybe if I had known far in advance how much work it would have been, I would have thought about it like five times before I got my feet wet.
Ivan:But I think it was fine. I think it's just this incremental exposure to complexity and how you realize that basically all of the things that you are relying on on a daily basis for years and years and years, which is just the simplest, some type is not assignable to some other type, all the complexity that is there and also all this ancient magic, right? All of these are in the miller, these are algorithms from the nineties or even older that we're just relying on nowadays. And the good thing is that you start getting into this and you start realizing how, I guess, the line starts getting a bit blurry between linters and type checkers and maybe you can start, because of all this information that you're having by implementing a type checker, you can start doing away with a lot of the linting work. You know how in Go, a lot of the linters are post hoc.
Ivan:Right? You just run them after the fact, just to make sure whenever you can that, you're saving yourself from some food guns. But the good thing is that if you have access to richer type information, then your lens become much more, much stronger. Basically, this in the same path that you're type checking, you can do linting, you can add, can, your diagnostics, as you were mentioning in Elm, they can be extremely didactic and pedagogical. They can tell you exact, they tell, it's a conversation, not just a neuromyces.
Ivan:They are quoting, they are picking parts of the code that you wrote and embedding them into the help text, into the reasons why the error is there. And so I would say from this whole project, what took me the longest and what took me, I guess, make me grow professionally the most was just getting my head around how hint the melder works. And on top of that, how do you make it work with Go interfaces, which is in no textbook teaches that. So, yeah, I would say that's basically how it grew. And the good thing about doing it incrementally is that you add type checking and then you start seeing how Elm was able to, just become so spectacular adding diagnostics.
Ivan:You see how you can add a linting system that basically minimizes the number of passes that you're doing as you're just traversing the AST to validate it, to maybe show warnings. Maybe sometimes you don't know for sure that the user is wrong, but you have a certain it's fairly certain that you are and so you surface certain things as as warnings. And then after that is basically, well, you do have also, I don't know what the actual term is. I call it desugaring. You know how you can have syntax sugar?
Ivan:The compiler basically goes in reverse. And so if you have a pipeline operator, you de sugar into regular, vanilla function calls, if let as well. And so, it was a very incremental approach. Starting very quickly from the easy parts, getting bogged down for months into, like, getting hard algorithms right through your head and then going stage by stage in the pipeline. So you have type checking, you have the sugaring, you also have well, this is not part of the pipeline, but it's important as well, is formatting, right?
Ivan:Like nowadays, if you wanna be taken seriously, you cannot, I guess, put out a language with just some minimal tooling and formatting. It's kind of stable table stakes and to actually get formatting right, that is not at least for me, wasn't trivial. So you end up looking how various different Gleam, for example, or many others, many other compilers offer formatting and you have, for example, the Oxy compiler in the JavaScript world, you have, basically just like with Hilde Miller, these are variations on top of very, very old algorithms. In my case, it's Lindig, there's where you create a document out of your AST and so you can add indentation, you can, add line breaks, you can make sure that in certain cases, for example, in the pipeline case, right, you have a function, pipeline another function, pipeline another function, certain cases you want them on a single line, certain cases you want them broken up into, all the different steps in each in their own line. There's a lot of complexity there if you've never come across.
Ivan:I mean, usually at work you don't get asked and you write a formatter for me.
Dominic:Yeah. I know. You you are talking about that and I'm like, wow. There there are so many things that I just don't know exactly what they are precisely. So I mean, it's it's it's very it's very interesting.
Dominic:So okay. Okay. So you you fought a a little bit to have this this type. What was there any any period of time where your compiler was finding no error, but once transpiled to go, there there was there was, you know, some glitches or issues and whatnot. You know, you were talking about interfaces and whatnot in go.
Dominic:So I would assume that sometimes I don't know. It's it might not always always be as simple as just, oh, it's it's it's compiling on Lisette, but it, you know, should should work on Go? Exactly.
Ivan:No. No. You're exactly right. That has been, I guess, the life story of this project. Basically, the idea is if your project is compiling in Lisette, it's that that is the safety guarantee that, like, I'm building towards.
Ivan:If your project is building in Lisette, first of all, you have a lot of guarantees that are happening. If if it compiles, many of the things that I'm checking for, I'm already telling you all of those errors, you don't have them. And second of all, it is completely valid go. So if you reach a point where your Lisette build succeeds and then you try to run and I don't know, some variable is undefined, then there's a bug in the compiler where most likely of cases it's generating some go that it shouldn't. And you know how a go is, right?
Ivan:Even a variable that is unused is gonna prevent you from running, right? Yeah. Which is a very also a very touchy subject as well when you're iterating, maybe you do want to be able to compile so they can iterate rapidly. But yeah, that is basically, I think it's a nice thing about working with a compiler. It's a very closed universe, especially or in this case, which is two compilers.
Ivan:It's a very closed universe and so you can just use that to your advantage to get to correctness faster. One of the things that I did and I'm still working on is, for example, when you're working on a compiler, you or at least I relied a lot on snapshot tests. For example, in the emitting layers we were saying, right? When you're actually writing this the the the text that becomes Go, you end up writing just a lot of snapshots just to make sure that your pipeline operator actually compiles to something that is, valid Go. And after you do that, it might look like valid Go to your eyes.
Ivan:But, what I'm working on right now, for example, or probably this weekend, is grabbing all of this all of the output of the snapshot tests for the emit layer and actually running them through Go Run. Right? I want to actually make absolutely sure that Go accepts this because that's kind of a way to reuse, repurpose emit tests into you you co opt them into end to end tests in a way.
Dominic:Sounds it it sound it sounds very fun to to to be frank. I mean, I'm listening to that and I'm like, oh, there there's there's no web browser involved at any point in this and And while this sounds so to yeah. To me, pipeline is is so I I I would I would guess that you might you might win a lot of people with the the error verbosity of Go. This is not something I I have problem with. I don't although I do like, and I I I post a message on Blue Sky about that.
Dominic:I I like the I like, you know, well, Ross and and Lisette, by the same time, the question question mark operator, especially with my screen readers. So, basically, there's intonation now in my in my voice, which it's it's very because, you you know, symbols are always extremely difficult in in screen reader, you know, sessions and whatnot. So it's either two verbals or either you are missing a lot. So I was I I was I was looking at your co example, and I was like, oh, this is this is very nice because yeah. So to me to me, the pipeline and the the pattern matching is is is sounds great.
Dominic:So just expressing the this this might be something that, you know, gophers are not very used to to have a, you know, a better type system. Let let's call it that way. I'm I'm I'm I have no problem saying that because once you have tried some functional programming aspects, you you you discover that, okay. Yeah. Yeah.
Dominic:Go go might not be might not let me express myself exactly like I would like with my types and whatnot. And I this is what I like about your project. I I haven't tried anything yet. I will I I will be frank. I I I just look at the the homepage.
Dominic:So so for me, for now, the the pattern matching, the the question mark a little bit less, but but the pipe the the pipeline, wow, in Go seems to be seems to be great. So what what is what is the integration look like at the moment? So can I can I use the packages from from the Go ecosystem in Lisette and whatnot? What what what what is going on there?
Ivan:So, yeah, there are three levels to the interoperability story. The first one, which is basically what I launched with publicly, is interoperability with the Go standard library. Right? So the Go standard library offers 177, if I remember correctly, packages for And you to basically, you as a user today can just use Lisette with any of those 177 packages. And as you probably saw in the homepage, the example for the pipeline operator literally is using, standard library methods in ways that you have never probably seen but it works.
Ivan:It's basically go in the end and just rearranging things behind the scenes. So that is level one. That works today. That does I mean, if you're interested, I can explain a bit of the compiler magic that goes into that. But, basically, that's what you can do today.
Ivan:I think it's 99% of the cases, it it works. Then you have level two, which is what people like most of the people, most of the reception I had when I launched this, most of the comments, came back to this, which is we want third party libraries, right? I mean, you want the entire Go ecosystem. And I usually say like, I hear you but like one step at a time, especially like this is not a one person on nights and weekends, but essentially that is coming. So the idea is okay.
Ivan:I guess I do have to explain it so that it makes sense. Let's say that you have any Go package. It could be a ghost channel library package. It could be a third party package Go package, and you want to use it from Lisette. We're talking about two very different type systems.
Ivan:One, let's say, bit richer than the other, but you still need to represent whatever is coming in from the Go world into the Lisette world, being able to all the safety checks, the linting, everything we talked about has to work on something that is coming from abroad, let's say. And so the way that it works is Lisette is mostly but not completely written in Rust. There are a couple of modules that are written in Go. There's a reason for that. There's this I called it, for the lack of a better term, bindgen or bindings generator.
Ivan:It's this Go package that is using the Go standard library that has affordances for you to basically traverse an AST, do AST manipulation, inspect it, just basically grab a chunk of Go code and derive intelligence from it without myself having to actually write all of that because I'm just using a ghost on the library package to inspect and introspect ghost under library packages, which means I can iterate over a 177 ghost under library packages, find all of the public symbols, and symbols here means functions, constants, methods, iota, what have what have you, whatever is being actually exported by that package, see those signatures, map them over to Lisette. So basically bindings generator, the core of it is having these mappings, right? So, like what kind of patterns in Go are equivalent to a result, what kind of patterns in Go are equivalent to an option, what kind of patterns in Go are can be made equivalent to an enum. And so what the binding generator does is grabs this foreign language, creates all of these bindings. If you think of, for example, TypeScript, TypeScript type definitions, the .d.
Ivan:Ts files. And based on that, that's how you inject all of this type information back into Lisette so that first of all, it can run all the safety and expressiveness that you need. And at the time that it comes to basically output to emit, go back out, it knows basically it can just merge the two worlds together. And that same principle of generating bindings for the ghost tunnel library is what I'm going to use for, third party libraries. There's a big difference there though, which is with third part like the ghost under library is written, I guess, by the same group of people over a long period of time with a lot of consistency.
Ivan:That is not exactly true of the world. But there is I mean, it's Go, right? There is a certain baseline level of consistency. People usually don't get too creative with their type. Like, they're used to, like, returning a tuple where one value is a result and the other is the error.
Ivan:That's really there is a long tail of of, creativity, let's say, but in most cases, you can on, people following Go standards. And so what I already have today, but I haven't actually documented it yet because it's work in progress, is this command, the list add, which basically, as you would expect, keep it as simple as possible. It reuses go get under the hood to get the package. Under the hood, runs the bindings generator, generates all those mappings for you, caches them in your disk, and then, basically allows you to use any Go third party package as you would use, the Go standard library. The long tail there is the problem, though.
Ivan:This is something that will basically will take a few months to to to actually get to the bottom of, which basically means the closer the package you're trying to use is to the ghost and the library conventions, the more likely it is that it's gonna work, but there are no guarantees. Right? People are completely free to do whatever it is that they want with, their packages. They were basically just like with JavaScript, never people didn't write JavaScript libraries thinking in the future that they might be used by TypeScript. It's TypeScript we have to accommodate and not the other way around.
Ivan:In many cases, for example, you might need heuristics and they might be wrong and so you need overrides, which means, for example, in certain cases, you think that you might be able to map something to our result, but actually, no, because of whatever domain, the library is using, maybe it's legitimately returning a tuple that might look like a result but isn't. And so what I'm doing, and I already did this, it's very interesting. There's this BigQuery dataset from Google with a lot of information on just the Go ecosystem at large and especially about, dependencies, right? All sorts of dependencies, popularity of dependencies, who is importing who. So basically you can create a bit of a graph or even a rank list of, the most popular packages.
Ivan:That's what I've been doing. And so, basically my machine just is installing so many Go packages and running bindings generator and seeing what fails, what can I find? So over time, at least working your way, like, down the list from the most popular to the least popular, you can start, first of all, checking if it all works, checking what doesn't work and fix it on the compiler side. Basically, the compiler can have a list of overrides so that I know that, I don't know, GorillaMax is gonna require some exceptions here and there. I can just hard code them for a specific version.
Ivan:And as we know, like new versions get a slash v two or v three. So, this is gonna be specific, immutable to a specific version as long as they're being respectful of semantic versioning. And, I think that's gonna be, like, quite a lot of work. I'm actually, hopefully, I don't know, I hold a bit of hope that if people find it useful, they will be contributing into the compiler, whatever it is that they find. The good thing is that it's really easy to just find these exceptions and work them into the compiler.
Ivan:I'm also thinking of ways to allow people to add overrides without having to go through me. So in the same way, for example, like in TypeScript, you can write your own you can override type definitions with whatever it is that you find. So, basically, the the compiler uses them without you having to contribute it back, which is, I guess, a second up. It's nicer for everyone to have access to them, but people would still be able to. And and then that's all level two.
Ivan:And then there's level three, which is everything we've been talking about is Lisette's primary. Right? You have a Lisette project and you're working in Lisette and Lisette is your world and maybe you bring in Go as a visitor into your world, but it remains Lisette. Then you have level three, which is the opposite.
Dominic:Mhmm.
Ivan:So level three is you inject Lisette into an existing code base. Right? I don't know what the proper term for this would be, like mixed source projects or Go primary projects, Go host projects, I don't know. But basically, let's say that you have an existing code base and you want to start injecting Lisette. Imagine if you were I don't know if you have experience with Java and Kotlin, but that's similar kind of symbiotic relationship where you would have your Go code base, you open up a list directory, you start writing your Lisette your Lisette module.
Ivan:And the difference here would be, for example, if you think of the TypeScript and JavaScript example, it's on a file basis, right? But Go, the boundary in Go is more package based. So my idea, as is again a bit further into the future, well, that's why it's level three, would be that you have Go owned packages and Lisette owned packages inside of your entire Go project. And so you have your newly set, let's say, embedding, you build, that that list build is gonna generate Go files. Those Go files, where are they gonna go?
Ivan:To your same project, respecting your your file paths. Basically, you wouldn't even be able to tell from your imports that some of the Go files are generated from lists versus some Go files are native, so to speak. And what you would be able to do that way would be in your classic Go, you would be importing from the transpiled Go and vice versa. And so the idea that would be, I guess, the the the final the final monster, which would be full interoperability inside of a Go project. All of this would be reusing what already exists.
Ivan:So basically, if you want to add, let's say, a Lisette dependency to a a Go first project, you would be able to. If you want to add a Go dependency to your Go first project with a Lisette embedding, you still would be able to, so that it's as seamless as possible, with this distinction. Right? The invariant would be certain, packages, certain, like, subset of files are all Lisette owned or all all Go owned, if that makes sense.
Dominic:Okay. Okay. Yeah. Yeah. Yeah.
Dominic:Yeah. That's a lot to unpack, I would say, though. So I know that. That's great. That's great.
Dominic:So I I think I think we we do have a good a good, you know, vision from your from your part now. I'm I'm a little bit curious. So I I want to return to two things. First of all, the the the build generator, when when would it run? Would would it run when Lisette detects that, oh, you know, I I particularly, you know, I would like to use this Go package, and now it detects that this this is new.
Dominic:I haven't built any binding yet, so I will just run it myself. Me myself being the compiler, or it would be the user that would need to to run something manually to say, you know what? I I want to use this package and now, by the way, run run the build generator?
Ivan:Right. So to be completely honest, this is a bit so far away into the future that I haven't
Dominic:thought too
Ivan:much about the developer experience. No. No. It's not. I mean, what I'm trying to say is, I guess when I reach that spot, I'm gonna try to use my best taste and then try to make the DX as streamlined as possible.
Ivan:It may be the case that we can do this automatically. It might be the case, I guess, for simplicity and so the developer doesn't get confused that we expect the developer to just run this build, which in many cases it's less than, I don't know, for mid sized project I've seen like two hundred milliseconds, so it's almost acceptable. Okay. There has no I have not done any performance optimization work either, so that might also be improved in the future. But yeah, that would be, I guess, my thinking.
Ivan:Let's reach there, see what makes the most sense. And I guess that's the cool part of having a less than V1 project. Yeah. If you just make a mistake or if you make a decision and many people are telling you that for a good reason, that with solid reasoning, that they disagree with that, maybe you can take a step back and reassess. So I definitely do not have all the answers I just have.
Dominic:Totally fair. Totally fair. Yeah. I I thought I thought it was it was kind of in in the process. It's it's a good testament to Raz though that that you you said that you haven't done any performance, you know Exactly.
Dominic:Work or attention and and you still have the this two hundred millisecond is is pretty pretty damn good. I mean, wow. This is this is impressive.
Ivan:I mean, if you wanna take it further, I'm also not the I I'm much better at Go and TypeScript than I am at at Rust. I'm absolutely positive that someone who had a higher level of Rust would even, like, easily be able to, like reduce allocations, lots of work. The good thing about Rust, especially for your compiler work is it's almost like tailor made for compiler work. All of these, it's a bit of a meta point, but all of these nice affordances you want to add to a new language, it already has them. By having them, it shows you how useful they are, especially we're working with ASTs that as you might imagine, for example, in Lisette's case, it just has so many variants, all the different AST nodes that you might have.
Ivan:And it's just the fact that it's using exhaustive pattern matching, for example, with Maranget's algorithm, it allows you to see, okay, well, now I want to add a new feature to my language. What are what are all the spots I'm breaking if I am adding a new AST procedure? What am I breaking in the formatter? What am I breaking in the sugaring pipeline? What am I breaking in the time checker?
Ivan:And that's just all taken care for you. So the more you work on it, the more you get convinced about the benefits.
Dominic:Nice. Maybe you will be able to how does it go when a language is written in itself? You will be able to bootstrap or something like that?
Ivan:That would be fun, but so that's called self hosting. Oh, yeah. Yeah. I've always found it I mean, there are, I guess, different answers to this. To be clear, it's definitely not one of the goals of Lisette
Dominic:reach the
Ivan:point where it's self hostable. From a, I guess, there's the vanity argument, there's the showing that you're production ready argument, There are just so many things that you could do. Think for a project of this nature, like aiming to be a transpiler, I think the proof is in just having real, I guess, the other day someone showed me, I guess, what would be the first Lisette project out there in the wild.
Dominic:Yeah.
Ivan:And I think if the motivation for self hosting a program, your own language would be to just have some proof of how production ready it is, To me, the proof is in the putting. The proof would be in there being actual Lisette slash Go projects out there in the wild, powering companies, powering, processes that are actually making money for, for real purposes. Would I like to have had infinite time and money and all of the things that I want for Lisette already done? Maybe.
Dominic:But it might not be as performant as a Rust based, right?
Ivan:So one thing that I'm noticing, for example, Gleam as far as I understand, it's also Rust based and it's not self hosted. I think it's not a trend that I'm seeing in very recent languages that are coming out, at least for the size of the ones that I've been keeping track of. But, yeah, I guess it's it's been on my mind, but I guess it's it's realistically not something that I would ever get to. I think I
Dominic:think Rock started with with Rust that they They turned to Zig. Yeah. That's that's very interesting. I'm I'm a I'm a huge language geek myself. Never never never had the the intention or whatever to to write, but I'm I'm always very, very interesting.
Dominic:Okay. Let let let me return because I want I want to return. Maybe we will we will go back a long time ago, but but still very interesting to me. So the pipeline. You were mentioning that you needed to do a couple of things to the the gold standard library to have them work.
Dominic:And I'm I'm curious to to hear, was there, you know, if we take the the string package, I I think most of the functions kind of accept the the first parameter as being the the string, the string source. So I would I would guess that that would that would kind of already helpful to to to do the pipeline. So I I guess my question is, I I suppose that the pipeline in in Lisette is taking what what's the the previous the previous operation inject the in the in the first argument or it's the last argument?
Ivan:So when we say pipeline, right, when we say pipeline, we mean going from the very beginning, which is like scanning, tokenizing up to emission. The way that type definitions play into the pipeline, especially type definitions that come, for example, from the ghost standard library, you generate them beforehand. In the case of the ghost standard library, they are literally I I generate them into you can go to the the set source and find them. They are bundled into the compiler. They are strings embedded into the Rust binary.
Ivan:And at the point in time when you reach, I guess, let's say, the first sub stage of type checking, which is essentially registering all the symbols that you have. If your program is importing, let's say, the strings package from the ghost hunter library, that's also what you're registering. And you're in order to do that, to know what to register, you look at the type definitions. That's the point in the pipeline where the type definitions, play a part. How those type definitions were generated, that's, I guess, in a way, it's this ancillary system that it's not even part of the compiler.
Ivan:Right? It's just this Go package that assists in in preparing the the type definitions that the compiler will be consuming. And the only reason it's like that, it's because it's just simpler to use Go to parse Go rather than to write my own Go, AST parser in Rust, which I looked at and I I don't think there was any if I found something, it wasn't extremely advanced, I thought, why not just rely on this extremely battle tested package and use Go for what it already is good at.
Dominic:Yeah. That is that is so interesting. So what is the you know, what is the next step very short terms? Let's say, the first three months or six months, whatever, you know, what what what is going on? Do you need anything?
Dominic:I would I would I would suppose, you know, people trying the the the language and give you give you some feedback, probably?
Ivan:Exactly. So this is this is very early days. Like, I'm under no delusions of where the project is at. And usually when you're starting out, first what you want is correctness and feedback to lead you to that correctness. I've done a lot of testing myself, I've done a lot of automated testing.
Ivan:When you write a language, it's imagine when you write a feature in your job and it needs to interact with so many other things that you really need to think about, think through every interaction. In a language, everything interacts with everything. And there are so many gaps of correctness that you just even you do as a single person, there's just so much that you can anticipate. And so I've been really lucky in that the launch got, I'm not a very public person, but the launch got a lot of publicity, like it ended up in Hacker News and many other places and people started like coming in and bringing their ideas and many of them, I guess in the Lisette spirit, they're very concerned with correctness and they actually are using the language not to build things, but to break the language, which I really appreciate. Because then they come back and they say, wow, I created this match statement, this match expression, sorry, where every arm is diverging, which basically means, yeah, one panics, the other they all return never.
Ivan:And this is like exotic, almost deliberate, almost malicious input that I never thought about, but like you have to handle it. And I'm really appreciating that. So basically, I think from like, I don't know, three months, six months, I don't know how long this might, but it will take as long as it needs to take. I want to make absolutely sure that as we were saying earlier, if your Lisette builds, then you have a guarantee that your Go is valid and it's gonna run and you're not gonna find Go runtime errors surprising you because the whole purpose, the whole, I mean, expressiveness is one thing, but I focus more, I think most of the benefit in practice is gonna come from safety. And so being able to just bolster all the work that goes into making sure that all of this is correct would be phenomenal.
Ivan:And once that is in place, what I've been also doing in parallel as people have been reporting these and luckily they are very easy to fix because they are usually small edge cases that haven't thought about. What I'm doing in parallel is continuing to work on interoperability, what we said about level two earlier today, which was just this long tail of various packages and making sure that all the exceptions are in place so that when you try to import them, you have to think about nothing other than your business logic and not why is this type signature not matching what I'm expecting, which also will take a while. I think that the fun challenge there is how do you get to cover all that long tail without how do you automate the work of detecting all of these inconsistencies so that it doesn't take twelve years to cover the Go ecosystem. Although like I have some ideas how to go about automating that, but again, it's very early days. It's just to give you an example today, Lisette does not have pre built binaries for all the platforms.
Ivan:Actually requires you to do cargo install and build from source yourself because I haven't had time to put all of that into my it's just a matter of finding the moments. But I also want people who report all these bugs and I usually do releases every day. I want people who report like bugs or open PRs to have a really nice experience and feel like, okay, I reported this and you blink and it's the next day and it's already fixed and you can continue on with either your crazy exploration or your actual building of the project together. I'm not in any rush. I think I really find joy in something like this and I think it's also very easy to kind of burn yourself out if you want to do everything, especially if you don't have a team with you.
Ivan:So I'm just trying to enjoy the process. Think it's like with the language, it's just the biggest, it's just a mountain of a project. And so if you don't kind of like find a joy in all of this and try to turn all these crazy challenges into things that can become solvable, then you're just not gonna make it. So basically, that's what I'm thinking, like, long term, adding functionality at the same time, consolidating correctness and and then just being very attentive and then listening to what people have to say. I have my opinions of what kind of I have a lot of opinions of what Go should look like, but I want people also to tell me to validate or to falsify my assumptions on whether, like, should we even be evolving Go or not?
Ivan:Yeah. It is a very heretic thought, That's
Dominic:a touchy subject in this community. Yeah. I think it's
Ivan:at least worth exploring, seeing how far we can get, how correct we can get. And from there, like literally, nobody's forced to use one thing or another. I just find it a fun question to explore. To me, it's always been this burning question. How far can we evolve go while still preserving all of the amazingness, all the good things that make it so productive in the first place.
Ivan:Right?
Dominic:Oh, yeah. Oh, yeah. Totally. Yeah. Yeah.
Dominic:That that that's also
Ivan:yeah.
Dominic:From from my point of view, the the what is going on at the moment is that there is so many languages that pops. I don't I I'm sure you've heard about Sky. There were there were Dinkgo a couple of months ago. Have you heard about Sky, actually? The it's a it's a Nelm like language.
Dominic:It's an ML language that also compiles to Go.
Ivan:I've heard of it. I haven't explored it too much yet. I am much more familiar with Dingo, for example, which is one of the approaches that I explored when I was building Lisette. Yeah, I should definitely, from what I, I read it very briefly, so forgive me if this is not extremely accurate, but from what I understand, it was more a very opinionated way to build full stack app. It was a bit more than a language even, right?
Ivan:But again, I did not delve into any depth.
Dominic:Yeah. Yeah. Yeah. But but that that's my point. I mean, the the the is so many noise at the moment.
Dominic:It it it's it's it's hard to it's probably hard to so that yeah. That's what I'm I'm happy to hear you to say that that, you know, you're taking your time and I mean, it's it's a tough crowd probably. I I would imagine. The Go community, you know, when even just talking about, you know, web framework and whatnot and things like that. I mean, just just there there is there is some there's some, you know, quote unquote gatekeeping and whatnot, and people like to like like the way Go is simple and whatnot.
Dominic:And again, I mean, when you when you'll when you try a for for people that did not tried any any kind of language, like Rust and and myself, I'm I'm doing a lot of Gleam. I mean, Gleam for me is is is just good and what I what I'm preferring at the moment because I like the beam as well. So you were you were mentioning that so it's not it's not only the it's not only the language, but most importantly, you know, what are you getting with this language? And I I think what what you are building in a moment, the fact that we would inherit from all the goodness of Go is is just very, interesting. It's it's a good it's a good place.
Dominic:I I think you you have something. Yeah. But again, I don't know. Did you did you receive any kind of, a lot you know, negativity and things like that so far or or it's mostly positive?
Ivan:Oh my god. I received everything.
Dominic:Yeah. Okay.
Ivan:So I am yeah, exactly. I am not a I'm not a very public person. I do have a blog that I post, like, once every two years. I am not a very public person. I also did not post all of these hacker news posts and so on, just basically people who found it.
Ivan:And I've heard everything. I've heard from people who said I was actually working on this exact same, like 99% of what you already did, so I'm actually switching to this to see how it fits what I actually wanted to do. I've heard from people who said this should have existed years ago. I've heard of people who said you should have never done this. I've heard people, I've had people sending me emails saying how displeased they were at how I was spending my private time.
Ivan:I try to take it with a lot with a big grain of salt. I think it's like politics, religion and programming languages. Everyone has an opinion and it's always gonna be controversial and for your mental health, you have to learn, like extract all the kind of constructive criticism. Usually very hungry, I'm always very hungry for constructive criticism.
Dominic:Yeah.
Ivan:And if you experience any kind of drive by snark, all this low quality negativity, let's say, allow yourself to like, flows through you and it doesn't become part like, enters one year and out the other. What I have had a lot of luck though, and this is something I really, I guess, even going beyond languages just in general, I've had a lot of luck with actually emailing people who had blogs that talked about how they wish Go were like this or that or, like all of these ideas that I'm bringing in, I'm, as I said before, I'm not inventing anything, I'm just porting things that a lot of people wish, maybe not everyone, but a lot of people wish Go hat. And I've had a lot of luck emailing these people and just very briefly saying, found this article that you wrote. I think it resonates quite a bit with what I'm trying to do with this Hindi language. If you want to take a look, I'd be, like, super thankful if you had any thoughts here and there about it.
Ivan:And people actually wrote back and where I'm talking about people who were, I guess, even like pseudonyms, I'm talking about people like in, like well known in the go community, people who have actually been very like delved into the source and then challenged my assumptions, asked sharp questions, that make you think, that maybe make you see that you haven't anticipated certain things and you need to, like, take a step back and then attack the problem from a different angle. Maybe some of the type bindings generation is not very precise in certain cases that you maybe were not thinking of because there are so many to think of. And that's exactly why I emailed them, because there's no audience, there's no public, there's no performativity, let's say. It's just one and the other person. And I mean, some other people say, oh, that's very nice and nothing else and that's fine.
Ivan:I think it's another way also to gather feedback. I mean, it doesn't have, it doesn't everything have to be public, but you take what you get. Yeah. It's a process. I'm still getting used to this whole handling all sorts of opinions and and trying to, in the most mature way, go about it.
Ivan:Yeah.
Dominic:Yeah. Very, very interesting project to be to be honest.
Ivan:Thank you.
Dominic:Ivan, this is this was great. So I I I hope you find the podcast not not so hard. I know you you had some doubt.
Ivan:I was nervous. Yes.
Dominic:I I would I would love to talk to you maybe maybe three or six months from now, see where the language is. I would love to to test it myself to be to be honest, I'm I'm having so much difficulties myself to just going with Rust. I I don't I don't know. I I do have some blockers. I'm there there's no excuses.
Dominic:There there's no excuses. So when I when I saw when I saw you the, you know, the the syntax and whatnot and the benefits that you are trying to bring to go, I was like, that that is that is very interesting for me.
Ivan:If it helps, I've heard from two users who took up Lisette and told me exactly that they were taking up Lisette only because they wanted to get more used with Rust just as a gateway into learning So that I never would have anticipated this.
Dominic:Yeah. That that might be a small step in between. Yeah. Yeah. I can see that.
Dominic:Thank you. Thank you so much. Do you have any closing thoughts? So what what would you know, we will have the links. So it's it's lisette.run and what, you know, what would you recommend people do?
Dominic:So we install we install the compiler and and just write some code and and that's it?
Ivan:Exactly. So basically, if you go to the repository, there's a quick start subsection of the document which takes you, like from not knowing anything to just creating a very small project, installing the compiler and just using it. There is an LSP, there is as much tooling as I can provide for a young language. And yes, if you're curious, give it a try and please know that like any thoughts that you have, the issues are open, the GitHub discussions are open. I'm very active.
Ivan:I find it very fun to have exchanges with all sorts of reactions from people. So, yeah, definitely feel free to give it a try and
Dominic:have fun. Perfect. Thank you very much for coming to this episode.
Ivan:Thank you, Dominic. It's been a pleasure.
Creators and Guests
