Big blocks or small blocks? This is the fundamental question of Bitcoin scalability. The argument for big blocks is also known as on-chain scalability. Under this strategy, each block in this append-only chain of Bitcoin transaction blocks would grow in size to be able to support lower transaction fees and higher on-chain throughput. A set of Bitcoin users who supported this idea forked Bitcoin to create Bitcoin Cash, a version of Bitcoin that has a larger block size. The argument for small blocks asserts that scaling Bitcoin does not require a larger block size. Under the model of the small blocks, the scaling demands of the Bitcoin blockchain would be handled by side chains. A side chain is a network of person-to-person payment channels that only reconcile with the Bitcoin blockchain to checkpoint batches of transactions. These side chains can be connected together to form the lightning network. Lightning network is hard to implement. To implement a lightning network requires solving real-world distributed systems problems that are unprecedented. It's much more complicated than deploying a blockchain with a larger block size. In addition, opponents of lightning network suggest that this will lead to a centralized banking system being constructed on top of Bitcoin. Opponents of lightning network fear that instead of a decentralized payments network, the world of a lightning network would be a lower-cost version of the present-day financial system, a world in which JP Morgan and Blockstream would partner up to battle Coinbase in a decentralized but actually centralized war for control of the unbanked. These big blockers, the people who are arguing against the lightning network, they argue that the new banks on the lightning network will be just like the old banks. They're going to be censorious of transactions, and they're going to be held in the domineering palms of the global financial kleptocracy. At least that's the position of the big blockers. So why bother with the lightning network approach? Why are we building this inelegant, kludgy system of off-chain, potentially centralized banking 2.0 complexity? Why not just increase the block size indefinitely and keep things simple? And even if we increase the block size today, couldn't we still deploy lightning network in the future while appeasing the transaction volume of today? Well, one major reason is that growing the block size does have a cost. The bigger the block size, the more demands it places on any node that wants to maintain a record of those blocks. And if you grow the block size today, you forego the experiment of seeing whether a small block size plus lightning network could in itself handle the transaction volume of a global decentralized financial system. Of course, this framing of big blockers versus small blockers is a conveniently polarized reduction of a much more granular reality. To believe that there is no subtlety between the two sides of this big block versus small block debate is to underestimate the number of dimensions involved in this argument. This is an unfortunate side effect of rigidly programmed Twitter bots that are on one or another side of the big block versus small block debate, and also a general political atmosphere of 2018 in which your lines in the sand are demarcated by which subreddit you choose to affiliate with. That said, my impression is that the more experienced engineers in the cryptocurrency ecosystem are overwhelmingly on the side of small blocks plus lightning network as the most promising approach to scaling bitcoin. Take whatever side of the debate you want, a single line of bitcoin core code that is actually committed speaks much louder than an avalanche of tweets. In today's episode, Jameson Lopp joins the show to explain why lightning network is an appealing engineering construct. We play the devil's advocate and contrast lightning network with a big block approach, as well as a big block plus lightning network approach. Jameson also describes his experience working within the Ethereum ecosystem and gives a sober explanation of some of the issues that Ethereum scalers may themselves encounter. As political as this issue is, what I like about the big block versus small block debate is that people get really impassioned around it, and when they get impassioned, it's more entertaining to watch them debate, and when people are more entertaining, it makes a dry subject more accessible. So if you're having trouble understanding cryptocurrency scalability, I recommend checking out the past interview we did with Roger Ver. There's also an interview out there on a podcast that I will link to in the show notes where Jameson and Roger, Jameson is today's guest, Roger is a previous guest on the show, but there's another podcast that I listened to that, again, I'll link to in the show notes, where they had a debate over this subject, over big block versus small block, lightning network versus no lightning network, and it was really impassioned and super useful. So if you're having trouble getting into this topic, I recommend checking out that episode that, again, I'll link to in the show notes. And if you're looking for all of our episodes about cryptocurrencies, you can check out the Software Engineering Daily app for iOS or Android. We have all of the episodes in there, we've got related links, we've got conversations that people can have with each other about the episodes, and they're all searchable. You can save them offline. We made the Software Engineering Daily apps for our Software Engineering Daily audience. And if you're curious about those back catalog episodes, then you can check out those apps. As always, you can send me an email or send me any suggestions for the show or whatever else is on your mind, jeff at softwareengineeringdaily.com. I'd love to hear from you. Every team has its own software. And every team has specific questions about that internal software. Stack Overflow for Teams is a private, secure home for your team's questions and answers. No more digging through stale wikis and lost emails. Give your team back the time it needs to build better products. Your engineering team already knows and loves Stack Overflow. They don't need another tool that they won't use. Get everything that 50 million people already love about Stack Overflow in a private, secure environment with Stack Overflow for Teams. Try it today with your first 14 days free. Go to s.tk.daly. Stack Overflow for Teams gives your team the answers they need to be productive, with the same interface that Stack Overflow users are familiar with. Go to s.tk.daly to try it today with your first 14 days free. Thank you Stack Overflow for Teams. Jameson Lopp, you are an engineer at CASA and a noted Bitcoin writer. Thanks for coming on Software Engineering Daily. Thanks for having me. We've had some previous shows where we talked about the basics of Lightning Network, some pros and cons of various approaches to scaling a blockchain. And today I'd like to start with the focus on why Lightning Network is the preferred scaling mechanism of the Bitcoin core developers. Then we can talk some about the actual state of the engineering of Lightning Network. I know you're not a total expert in the field, but I think you're enough of an expert to give us some overview. And to give some context in relation to other episodes of Software Engineering Daily, I had Roger Ver on the show a while ago. He gave his argument for a larger block size, why that's in his view better to scale than Lightning Network. Because I had Roger Ver on the show, I actually got flooded by anonymous crypto Twitter people who were either harassing me or supporting the episode, depending on what side of the bot line they fell on. Can you give a brief overview to these two alternative ideas, the larger block size versus Lightning Network? Sure. It really comes down to trade-offs, right? And so that's why it can get pretty heated, especially when people start saying, oh, that'll never work. It's because there's so many variables at play that really what we're talking about is what trade-offs we're willing to make in order to get it to quote unquote work. And then of course, a lot of times you have to define what work really means in the first place. But the trade-offs that I find that Bitcoin scaling debate really comes down to is prioritizing a low transaction cost for the user at the potential expense of a high validation cost for someone who is running their own node to make sure that nobody is breaking the rules on the network versus prioritizing a low validation cost of the entire system at the possible expense of high transaction cost. And that is really what this has all come down to. Of course, this is not only technical but also economic arguments, a lot of philosophical and ideological arguments. So there's no objective correct way to scale Bitcoin. It really comes down to what trade-offs you're willing to make. And even the trade-offs themselves, as you laid them out, those are hypothetical trade-offs. We don't actually know what would happen in practice if the Bitcoin blockchain scaled up as a block size and the community was run wild in that world versus having Lightning Network implemented and having the world run wild similarly. What gives us such confidence that we know what the outcomes would be in these two hypothetical scenarios? Well, I see it more from a security standpoint. And so my standpoint has changed over the years where several years ago, I was actually more on the scale everything on the blockchain camp. And that's because my perspective at the time was from a capacity planning perspective. I had worked with Big Data, Hadoop clusters, HBase clusters and whatnot for a number of years doing large scale analysis for marketing companies. And I looked at the Bitcoin blockchain from a capacity planning perspective of saying, you know, we need to have a lot of headroom available otherwise the user experience degrades and that's bad for Bitcoin. But on the other hand, now I'm actually looking at it more as a security perspective of Bitcoin as a whole ecosystem is setting this block size cap, not because necessarily it would kick a bunch of people off the network because they couldn't run a two megabyte block size on their nodes, but more because it's very hard to control these networks. And so once you do make a change to increase the capacity, it's going to be very hard to sort of reverse position and go back if you make a mistake and you pushed it too far. So I'm these days leaning more towards being very conservative about the changes that we make to the protocol simply because I think that it needs to be more of a like aerospace engineering perspective taken to sort of the caretaking of the protocol and the network rather than a move fast, break things perspective. And it's interesting if you look at the crypto ecosystem more broadly, you can definitely see that there are some projects out there that are more on the move fast, break things side and they can be interesting to watch. But I just don't think that that is the right way to engineer what is supposed to be a sound monetary system. So your position is more about one of humility. We know that the current Bitcoin block size is working decently. You know, we can see, we can imagine a world where off chain scaling works. And even if we don't have it just yet, it's not worth putting the cart before the horse and saying that because we can't, we won't have lightning network for another two years or five years or 10 years, we shouldn't threaten the experiment by increasing the block size. Right. It's making these trade offs of we're protecting Bitcoin, the ecosystem and the network at the expense of having a poor user experience at times when there's a lot of demand or a lot of congestion on the network. And you can definitely go the other way with it and try to make the protocol more malleable, more flexible. And that's why it's actually interesting to see really this, you know, side by side like AB experiment, though now it's not just AB, it's like 40 different forks of Bitcoin that are all out there and trying to compete. It's not all that dissimilar to what has been happening for the past like seven years where a lot of altcoins have come out and just tweaked a few parameters and said, okay, now we're going to be able to be a lot better than Bitcoin at doing this one thing, whether that's throughput or privacy or what have you. And at the end of the day, Bitcoin still manages to retain the top position. And you know, you could speculate as to why that is. But I think it's mainly because it has proven itself to be very robust and is not going to be changing for the whims of a few people. And probably people are making an implicit bet on the same conservatism, the same humility that we're talking about here. It's as much about a bet on the technology as it is the ethos of the people who are making commits to the technology. Yeah, I mean, and it's a diverse ecosystem. I definitely hate to ascribe power to the developers. And this happens very often, especially when debates get heated is you start seeing people looking for scapegoats or trying to find like centralization of power so that they can blame somebody. And for all of the blame that is being put out there against Bitcoin developers, and you know, I don't even like saying Bitcoin Core developers, because there's like seven or eight different implementations of Bitcoin. And I hold no allegiance to Bitcoin Core as an implementation over any other implementation. If they decided to start doing something with which I disagreed, I would be the first one to jump up and say, you know what, I'm going to switch to using some other implementation because I don't agree with what they're doing. It's really about the ecosystem as a whole, because these developers can't just make changes. They have to propose changes, get consensus within their own developer group, and then put them out there for the rest of the network to decide whether or not to adopt. So it's interesting to see what happens with these network splits, where you have, you know, 15, 20% of the users on a network disagree with a certain change or certain lack of change, and then they split off and go create their own network. When you say there are seven or eight versions of Bitcoin, are you talking about the Bitcoin cash, Bitcoin gold flavors of Bitcoin? Are you talking about things that are compliant with Bitcoin itself? Right. I'm talking about implementations that are compatible with the Bitcoin protocols. And so of course, you know, this just talking about this gets really difficult to discuss because like you said, there's also dozens of different actual networks that call themselves Bitcoin something. That's one of the interesting aspects kind of going back to, I guess, the governance or lack of governance of these systems. It really is this new crypto anarchy that's emerging on the internet, and we're still trying to figure out how to deal with some of the weirder aspects that happen when nobody controls even the usage of a name of a thing. There are full nodes, there are miners, there are simplified payment verification nodes, there are a variety of node types on the blockchain. What is the weakest type of hardware that should be able to run a miner or that should be able to run a full node? Should you be able to run a miner on a Raspberry Pi? Yes, this is one of the like fundamental ideological questions. I think that you don't find people that are really setting specific monetary goals. It's usually one extreme or the other. It's either we want as many people as possible to run a full node, possibly even on a mobile phone with a very crappy data connection, versus the other extreme of we don't really care who can afford to run a node because anyone who is doing enough economic activity will figure out a way to afford to run a full node. If it costs tens of thousands of dollars a month to run a full node, then that's still okay because then the businesses that are making the most money can afford to do that and it'll be worthwhile to them. This kind of goes back to what we were saying earlier is there's no real objective correct answer on that. It's just a matter of your opinion. With the larger mining pools, we already see a notion of sharding within the mining process. You have miners that are teaming up together to look for a valid block. I say sharding. That's probably an improper use of sharding. You could imagine a more sharded version of mining where you could have a trustless system where me and you and 500 other people in an IRC channel are all running some open source software that is maintaining some shard of the Bitcoin blockchain across our phones and therefore we have a sharded full node essentially. Maybe it's on IPFS or something if we want to get super fancy. Do you think that's a plausible model to getting everybody running a full node? I haven't really seen much research on anyone trying to shard a full node. The most node sharding research I've seen seems to be on the Ethereum side. Intuitively it seems like that is going to change the security model, but it's going to really matter on the details. I think a way that you could consider the nodes or the network to be sharded in a trustless fashion is actually via sidechains where you can then have these pegs between the main Bitcoin blockchain and some other blockchain and if you're not using that sidechain you don't care. You don't have to validate it. If you are using it then you might care and you might want to validate it, but that way you can have specialization between different blockchains that are trying to do certain things better whether it's privacy or throughput or what have you and then only the people who actually care about that particular functionality would then have to worry about validating it. And when I propose something like sharding in the Bitcoin ecosystem today, just saying it to you, it betrays a lack of understanding of the engineering complexities of implementing something like sharding against the Bitcoin blockchain and I think there's a lot of lack of understanding of the finer engineering details of how hard it is to maintain. If you think up some magical solution to the current Bitcoin projected scaling difficulties or present day scaling difficulties depending on how you look at it, it's very easy to conjure up solutions that sound like they work, that sound quite good in a white paper, but probably would be less good if you ran the experiment, tried to roll it out, tried to get enough engineers behind you. How do you develop an intuition between the scientific laboratory things that sound good in a white paper versus things that are actually realistic? Yeah, that's tough, and actually I would say that there's probably a lot of people in academia who have been displeased with how their work in this space has been received or not received as it were, mainly because there's a lot more to deploying changes on these networks than just coming up with a great idea. You actually have to convince the people on the networks to adopt it, and that can be a much more challenging thing than just writing a paper and doing an experiment and getting peer review on it to show that it is theoretically a good idea. You actually have to be willing to engage with the technical community and basically go through a gauntlet that is not guaranteed to end well for you, and sometimes that results in people then going off and creating their own network because they got rejected from getting those changes into Bitcoin or whatever other network that they were trying to change. This is once again the lack of governance aspect of it. For the average person, it can be pretty baffling to try to keep track of proposed changes. There can definitely be a lot of lingo and game theory and other understanding that's going on under the hood that's not obvious if you're just reading these papers or discussions between developers. That's when I think people start gravitating around the other technical folks in the ecosystem who are better at dumbing things down. We are running an experiment to find out if software engineering daily listeners are above average engineers. At triplebyte.com slash se daily, you can take a quiz to help us gather data. I took the quiz and it covered a wide range of topics, general programming ability, a little security, a little system design. It was a nice short test to measure how my practical engineering skills have changed since I started this podcast. I will admit that though I've gotten better at talking about software engineering, I have definitely gotten worse at actually writing code and doing software engineering myself. But if you want to take that quiz yourself, you can help us gather data and take that quiz at triplebyte.com slash se daily. We have been running this experiment for a few weeks and I'm happy to report that software engineering daily listeners are absolutely crushing it so far. It has told me that everyone who is taking the test on average is three times more likely to be in their top bracket of quiz scores. And if you're looking for a job, triplebyte is a great place to start your search. It fast tracks you at hundreds of top tech companies. Triplebyte takes engineers seriously and does not waste their time, which is what I try to do with software engineering daily myself. And I recommend checking out triplebyte.com slash se daily. That's t-r-i-p-l-e-b-y-t-e dot com slash se daily, triplebyte, byte as in eight bits. Thanks to triplebyte for being a sponsor of software engineering daily. We appreciate it. Okay, I want to talk in some detail about Lightning Network. And by the way, if anybody out there is listening who works at like Lightning Networks or perhaps Blockstream. I did a show about Lightning Networks with somebody from Blockstream with Rusty Russell from Blockstream. I think it was like two and a half years ago before I even understood how Bitcoin worked. So that show was not great, but it was, it was decent. But anybody out there that really understands this stuff was working on this stuff in a lot of detail. I would love to do a show because again, I know you're not working on this day to day, but it's kind of like, I was really looking for somebody and I just found you. I heard you do a podcast or two about this. So let's talk about Lightning Networks for a bit. So Lightning Networks, as I understand are a way of networking together payment channels. So you and me set up a payment channel with each other against the Bitcoin ecosystem. And Todd and Sue also set up a payment channel between each other. And Jameson also sets up a payment channel with Sue. Therefore, I can make a transfer of money to Jameson, Jameson can make a transfer of money to Sue and therefore there is an implicit payment channel between Jeff and Sue. And all of this can take place off chain. Explain if, tell me if that's correct, like give an explanation for how Lightning Network works. Yeah. Reasonably accurate. I mean, in order to create these channels though, you are making on chain transactions. You're basically anchoring into the Bitcoin blockchain or Litecoin blockchain or really any network that has support for a few primitive functions like time locking and multi-sig. Because we need to be able to construct these hash time lock transactions, which are used to update the state of a payment channel. So when you're creating a channel between yourself and Alice, then basically the two of you are entering into a multi-signature arrangement that you can then update between the two of you privately and without broadcasting to the rest of the network. And you can update the values of basically how the value is balanced between yourself and your counterparty in that channel. Then there's a lot of game theory involved around making it a very bad idea to try to cheat somebody and post a stale channel state that would give you back more money than you're supposed to have. That's where a lot of the time locking comes in. There's also various recovery transactions that basically allow you to steal the money away from someone if they try to steal it from you incorrectly. And this works pretty well between two people, but it's not that interesting. This technology has existed since I think 2012 or so. What gets interesting is then when you start connecting all these payment channels together and you start routing money through them in a way that is more, I guess, opaque to the end user. They can just say, you know, I want to make a payment to this address and I don't really care how it gets there. Let's just let the software under the hood figure out how to route it. And now since we're doing this all off the blockchain and only transmitting the data between the few parties that are interested in it, it's orders of magnitude more scalable. You basically become limited to network latency and the actual processing requirements are very low. So I think that some folks like Christian Decker have said that you could update a payment channel hundreds of times per second. I think that's pretty much going to come down to whatever the network latency between the two parties in that channel is coming out to. But from a more macro perspective, this is much more appealing for scaling a distributed decentralized network than just throwing more data onto the whole network itself. And the comparison that I often use is actually with the internet itself of this is a pretty tried and true method of scaling distributed networks where if you're looking at the way the internet is constructed with layer zero, the ethernet layer, that is a global broadcast to all protocol where you put data onto the ethernet and it goes to everyone else on that ethernet network. And of course that would never work if we tried to scale it up to the entire world. We wouldn't be able to do things like we're doing right now with streaming audio because we would have to simultaneously send and receive everybody else's audio streams at the same time and obviously our ethernet cards would get maxed out and the fiber would get maxed out. Instead, you know, what they did to scale the internet was they added layers on top of it and specifically TCPIP, the routing layers, make the internet much more scalable. And what you're doing there is you're just finding the fewest number of entities on the internet that need to know about given piece of data in order to get it to the desired destination. So when you're sending data to someone, it doesn't have to go to everybody else on the internet. It just gets routed through a minimum number of parties and of course if any of those parties stop working or start trying to screw with you, then you just route around them and find some other ones. And that is really what gives the internet, you know, an extra level of robustness. And it's really the same type of thing with Lightning where instead of routing just data, though it is just data, it's now a special type of data that is representing value. So now we have this financial network that is much like the internet itself and distributed, permissionless, and pretty robust, though that's what a lot of the arguing that's going on right now is about, you know, how robust will it really be? And there's definitely still plenty of issues that need to be overcome. But I've seen a lot of different people that are devoting their own resources in order to improve this protocol. So I think that we're going to keep working on it until it becomes good enough for actual like production money use. Like internet infrastructure, things like emails and seeing web pages and loading videos. These problems did not get solved by liberal university professors and cypher punks. They got solved by corporations. And so what we have today is you and I talking over a VoIP interface that was created by a company talking on a browser that was probably, well, in my case, at least created by Google running on pipes that are Comcast and other corporations. And the worries of somebody like Roger Ver would be, okay, if we take this delegated model of scaling, where we move it off chain away from the purest form of scalability, which well, in perhaps Roger's view of purity, you threaten the decentralization of it. You delegate these scaling problems such as, how do payment channels discover and get connected to each other? That's a hard problem. We're delegating that to Blockstream or Lightning Networks or whoever else. Or who knows, maybe it's software people can run themselves. It's peer to peer. That could be great. You've got problems like people flicker on and offline, and I think there's some other trust issues. What's the argument that Lightning Network will lead to a degree of centralization that will threaten some of the core goals of Bitcoin? Well, it's an interesting thing to hear people talking about centralization of the Lightning Network. And actually, if you go back to my internet analogy, you can make a case that TCPIP centralizes the internet because now you're only routing through a small number of parties. But I think the counterbalance to that is that if there are other parties available and you can route around the bad actors, then that's an important counterbalance to any quote unquote centralization. And so this centralization and the fact that there are going to be economically larger nodes and economically smaller nodes and kind of expect to see like a power law distribution that results in a scale-free network topology, it's mostly important because we're not talking about actual custodianship of the money. So it's not like you might try to send your money through a node operated by Coinbase and they're going to freeze it and steal your money. They could refuse to relay it along and then you have to find another route. But I don't think it's quite the same level of centralization problems that we would actually expect to see if we scale on-chain because then scaling on-chain is going to drastically increase the cost of who can run the full nodes, which are actually determining the rules to the network. Now that seems to be a lot scarier because now the individual no longer has control over what rules they're agreeing to. It seems like, you know, giving that quote unquote custodianship of network rules to a small number of actors is a scarier proposition. But of course, you know, this is once again just kind of going back to ideology and what you think is the most important aspects to preserve. I completely agree with you. So, and I think if I can reframe your argument back to you, the arguments that Lightning Network leads to centralization is a portrayal of a world where banks stand up their own Lightning Network. Maybe it's powered by Blockstream or in partnership with Blockstream or powered by Lightning.Networks. And we're all routing all of our payments through new banking infrastructure that's owned by the same people. You know, maybe we, you know, the main chain is still open to cypherpunks and whatnot. But if you wanted to transact only through the main chain, it's going to be a much worse experience. It's going to be much more expensive than if you were to transact through payment channels. That's like kind of the centralization case. That's the scared bear case. The bull case is that the world with no on-chain scaling with the small blocks, the small block world is that nodes remain easier to run. Not that they're easy to run today, but they're not going to get harder at least. And in addition, it's not like, I mean, we don't necessarily know to what degree there will be centralization of the Lightning Network. For all we know, there will be indie banks. Like maybe me and you can go start a bank and have a Lightning Network between us and we can charge whatever we want. Maybe we have to charge slightly more than Chase if Chase has a Lightning Network-based banking system, but at least it's permissionless banking. Right. And that's actually my plan is that I intend to be operating nodes on the Lightning Network. And you know, if other banks, if like traditional financial institutions want to come in and start routing money, then they're welcome to do that, but they can't stop me from participating. So I'm not worried about that. And how resource intensive do you think it'll be to run a Lightning Network? I don't think it's going to be that intensive from a computational standpoint. I think the bigger question is going to come down to what level of economic risk are you willing to take because, you know, running a routing Lightning node is going to be running a hot wallet. And so hot wallets are a security issue you need to worry about. It's really, I think, going to come down to the economics of, you know, how profitable can it be to route money through the Lightning Network versus the level of risk of operating this hot wallet. So that'll be another thing that, you know, people are going to learn over time. And of course, you know, crypto security is a constantly moving target. We are getting better at operating more secure hot wallets, but there's still quite a ways to go. I really want to take the devil's advocate argument to as much of an extreme as we can because I just, because I want to settle, I want to settle this as much as I can for the Twitter bots that are somehow transcribing this podcast and processing what I'm saying. So if I'm a person in a developing country or a poorly regulated financial system like Venezuela, why is a world with Lightning Network superior to one with big blocks? So it, you know, it depends on the user experience and what they care about. Now, if you're living in Venezuela or a country that has, you know, hyperinflation, your top priority is probably not being fully self-sovereign and invalidating that no one is breaking the rules because you're already living in a system where the authorities are breaking the rules so terribly that you have this huge incentive to put your value anywhere other than your nation state currency. So for a lot of people that are in that situation, they may not care about the specific state public security model that they're being given. They just may care more about the like the store of value aspect and of course in any of these permissionless systems, they're pretty volatile, but they're at least, they're not guaranteed to go down in the way that, you know, hyper inflating nation state currency is. Okay. And the problems that you've outlined with Lightning Networks, there's a great article I read by you that I think was from like, it might've been from 2012 or maybe, maybe it was 2016. It was a little bit dated, but you outlined a lot of problems with Lightning Network. I don't know if they've been solved today, but there are things like that we touched on earlier, like the fact that people can flicker on and offline, the fact that payment channels have this discovery process, this discovery problem, which is sort of like the problem of service discovery and a, you know, that a company has to face. What are the biggest unsolved practical engineering problems in Lightning Network or are you up to date enough to kind of know some of those problems? Yeah, there are. There's actually another type of flickering problem that actually came up recently and just got fixed where there is this fee negotiation process that happens when you're routing money. And there was an issue recently actually that resulted in a number of premature channel closures happening because nodes on the network got out of sync and were disagreeing with what the appropriate level of fees were. And so trying to negotiate fees with another node who was then asking for order of magnitude higher fees would actually result in you closing the channel on them because it was suspicious and spammy. And so there's, that's just an example of like a lot of these tiny little details that are going to need to be worked out over the long run. I also in that article, I think that was two or so years ago, I was just noting that, you know, it'll, you'll need to worry about different types of denial of service attacks where there, you know, there could be an attacker who is very well funded and tries to flood the network and basically unbalance a lot of the payment channels and force them to close. This is a layer on top of Bitcoin. And so you, you can't create a layer that is more secure than, than the foundation, of course. So there, there's going to be new game theory, new types of attacks that, that need to be tested and figured out how to mitigate. So that's why it's not going to be this overnight process of, oh, okay, everybody, the lightning network is deployed and, and you should just go ahead and use it. But rather it's going to be a long process of, well, the lightning network is already out there and there are thousands of people that are using it, but it's still in a beta testing phase and we wouldn't recommend putting a large amount of money into it. We've done a lot of shows on testing and deploying internet infrastructure in the context of a specific company. So if I'm a SaaS company, I probably have a continuous delivery pipeline. I've got these different environments. I've got a test environment or staging environment and I have a pipeline. I, you know, if I'm deploying new software, I first deploy it to my test environment. I run some tests on it. Okay. It's looking good. I click, you know, move along the pipeline to the staging environment and then I have automated tests that run in the staging environment. Maybe I've got some manual people who are doing some tests in the staging environment as well. If it works, then I push it to production and then we have my SaaS company has been updated and it's in production and that's great with lightning network. We're talking about deploying software that has a wider reach, perhaps is quite difficult to roll back. If you roll it out, how does the deployment process of something like lightning network compare to the deployment process of conventional software? Well, so you do, you do start out pretty similarly of, you know, you'll, you'll write your code, you'll write your unit tests, you'll have peer review on the code itself, which is hopefully very rigorous. And then there are both local test networks and reg test networks that you can set up. And then there's the actual test net, the, that most people are using and, and you know, hopefully any issues get caught while it's still in the testing phase, but this is one of the reasons why, you know, life cycle for changes to these crypto asset networks tends to be a lot longer because you, you want it to be baking in adversarial test environments for as long as possible. Because like you said, once you get to the production ready phase, even then it's not, it's not that you just deploy it because you don't have the ability to deploy it. It's that, you know, you make the software available to deploy and then you're going to see probably like a bell curve distribution of, of actual upgrades that happen where there will be a few early adopter techie types that, that upgrade very quickly. And then over the next month or two, you'll probably get most of your upgrades. And then there's going to be a very long tail of laggards. And really as soon as you've made it publicly available and have the first few people that are out there and upgrading it, it becomes almost too late to roll back in many cases where a, a reverts is going to be pretty damaging to, to anybody who gets affected by it. And to what degree can I transact on a lightning network today? You can, you can go set up a lightning wallet on test net or main net. There are at least a few dozen providers on main net that you can actually buy goods and services through. And I have links to all of those on my website and also linked to some other lightning resources. How does that experience compare to transacting through Coinbase or some other non lightning network payment provider? It tends to be a lot faster in terms, in terms of just the responsiveness because there's no need to like wait for confirmations or anything like that before the provider gives you whatever it is that you're purchasing though, because it is still pretty new and there are some kinks to be worked out. There are the occasional payment failures though. I haven't experienced them myself, but I have seen some other people have basically routing failures. So it's not, it's not guaranteed to work at this point for sure, but it seems like the vast majority of time people are having a good user experience. There are these different implementations, lightning.networks block stream. I'm sure there are several others. What are the subjective design decisions? What are they making trade-offs on? Well they're all trying to be compliant with each other. I think the main real differences that I'm aware of are the actual languages that they're being built in. And so the C lightning implementation, which is being written by some of the blockstream folks tends, it seems like it's, it's more the like Bitcoin core style, like a C plus plus development, whereas L and D by lightning is actually more like the BTCD style where they're written in go tends to be a bit more readable developer friendly and whatnot, but possibly possibly not as performant. And then there's Eclair, which I'm actually not familiar with what that's built in, but I think those are the three main implementations that have the most developers working on them. Once lightning network is deployed, let's say it works, it's secure. What kinds of platforms? What kinds of cool things could be built on top of it? So this is actually one of the reasons that I'm excited about lightning more than just for having cheap fast payments is that I actually expect to see a proliferation of, of laps as some people have called them, you know, various lightning enabled apps to interact with each other because now if you're, if you're building something on these second layer networks, it's, it requires a lot less permission for you to play around with them. It's not like you need to try to get like consensus changes or even, even policy changes at the network level. So I've, I've been seeing from my, my former coworker, Alex Bosworth, actually, he, he seems to be a big lightning idea, man. He's always talking about new ideas of stuff that he's working on, but I'm most interested in seeing a decentralized exchange happening and that's for a number of different reasons, but mainly to try to get rid of some of the systemic risk of the current centralized exchanges, but also just from a privacy standpoint, I think that anything that you're doing on these second layer networks is arguably going to have better privacy than doing out in the open on-chain Bitcoin transactions. Now it's certainly debatable, but it does seem to be a network that's taking a much bigger focus on privacy. Citus data can scale your Postgres database horizontally. For many of you, your Postgres database is the heart of your application. You chose Postgres because you trust it. After all, Postgres is battle-tested, trustworthy database software. But are you spending more and more time dealing with scalability issues? Citus distributes your data and your queries across multiple nodes. Are your queries getting slow? Citus can parallelize your SQL queries across multiple nodes, dramatically speeding them up and giving you much lower latency. Are you worried about hitting the limits of single-node Postgres and not being able to grow your app? Or having to spend your time on database infrastructure instead of creating new features? Available as open source, as a database as a service, and as enterprise software, Citus makes it simple to shard Postgres. Go to citusdata.com slash s-e-daily to learn more about how Citus transforms Postgres into a distributed database. That's c-i-t-u-s-d-a-t-a dot com slash s-e-daily, citusdata.com slash s-e-daily. Get back the time that you're spending on database operations. Companies like Algolia, ProsperWorks, and Cisco are all using Citus, so they no longer have to worry about scaling their database. Try it yourself at citusdata.com slash s-e-daily, that's citusdata.com slash s-e-daily. Thank you to Citus Data for being a sponsor of Software Engineering Daily. To improve your point, I think to strengthen your point, if I transact on the main chain entirely from a set of payment addresses that are associated with my wallet, you could maybe re-piece my transaction history and figure out what I'm up to. If you had a lightning network that acted as a dark pool, then it could pool together transactions and make larger transactions with the main chain to checkpoint to the main chain, and everybody is shrouded in the additional privacy features of that lightning network. Yeah, and it's going to get interesting once we see more complex types of routing become more common, especially when it becomes easier to break up your payments and basically spread your payment out kind of across a large swath of the lightning network, and then it gets sort of pieced back together at the end. The lightning network, if we get fast payments implemented and they're working well and they're cheap because they're on the democratized world of different lightning networks competing with one another, then we could see an explosion of programming languages on top of that, or we could have maybe an Ethereum-like language that simply has payments as a first-class citizen within it, just like Ethereum, except it's interoperating with the Bitcoin blockchain. How would that lightning network-enabled Bitcoin decentralized app platform, how would that compare to Ethereum itself as a platform? There's a million different ways that this whole thing could go. There are some people that are already starting to try to think of what would layer 3 or layer 4 type applications look like, but there's also a fair amount of innovation going on with scriptless scripts and better smart contracting languages, like Simplicity, for example, seems to be making some good headway there. Also better and more performant smart contracting on Bitcoin itself with the Merkleized Abstract Syntax trees and Taproot and Graphroot. These things are all still probably a year plus away, but I'm actually fairly confident that over the long term, we're going to see robust and yet developer-friendly smart contracting languages become available for use in the Bitcoin ecosystem. It's obviously taken a lot longer for Bitcoin to get around to this as opposed to Ethereum because the developer mindset is a lot more conservative and I guess not as interested in the move fast, break things type of development process, but it's definitely going to be interesting to see which of these but ultimately is more popular, like the one that is very well marketed and very developer-friendly or the one that is trying to be more secure and an edging towards developer-friendliness. I don't think that we're going to have to worry about seeing that play out for probably at least another year. At least another year, that's pretty fast as far as I'm concerned, right? Yeah, I mean, a lot of this stuff really does come down to your personal time preference and I think that's also a big part of the scaling debate is that a lot of the people who want to increase the block size, they see it as a more urgent issue where if Bitcoin doesn't do this, then it's going to get overtaken by some other network that can scale better. Whereas the more conservative folks are like, we know Bitcoin is going to be fine regardless of how many people are unhappy with trying to transact through it. Yeah. So we talked about the low-level development issues in Bitcoin. We're talking a little bit about Ethereum now. You had a great article about the challenges of building low-level Ethereum infrastructure. Tell me about those challenges. What were you learning? I think you just spent some time like building on Ethereum, right? Because I think you had to implement something for the company BitGo when you were working there. Tell me about the challenges of low-level Ethereum building. Well, I mean, there were a number of surprises from the operational standpoint. It was a nightmare trying to run fully validating nodes. It turns out that most people in the Ethereum ecosystem are not fully validating the whole blockchain. They're actually using this warp syncing feature that is just getting the like Merkle root of the entire state of the system from a recent block and starting to validate it from recent history and assuming that, you know, most of the nodes are honest and the hash power is honest and whatnot. And so that's kind of a trick that Ethereum has gotten put into most of the nodes just to make it usable. If you're trying to do full validation, there are certain types of hardware that is literally impossible for it to ever catch up and get to the current tip of the blockchain. So you end up playing around with a lot of configuration parameters and different hardware types in order to get fast enough speed. It really mostly comes down to disk IO because of all the state change operations. From a kind of even lower level perspective, just dealing with 256-bit integers and 256-bit math can be a real pain. If you're looking to do that, like database operations of like atomic addition, subtraction, whatever operations with 256-bit numbers, you're probably going to have a bad time. I'm not aware of any that support that natively. We ended up having to basically do the arithmetic ourselves in memory and single thread those operations so that we weren't, you know, accidentally corrupting the data that was currently in the database. And then at a higher level of just the smart contracts themselves, smart contract security is still very new and it's a real nightmare if you're doing what we were doing, which is writing our own multi-signature implementation. Interestingly enough, anyone who's familiar with the parity multi-sig contract and the different issues that it had, the first issue that it actually had was a bug that an audit found in the BitGo smart contract. And that actually was found and fixed like over a year ago, like almost a year before BitGo even went live. And then, you know, we had been live for about six months and that parity hack happened and we looked at it and we were like, oh crap. You know, this is such an easy thing to overlook. It was kind of the way that our whole smart contract development process went where we got an audit, we fixed things and then we would go to a different firm, get a different audit and they would find different things. And it was just a process of like this programming language and the Ethereum virtual machine. It's all so new that almost nobody really understands how it works. And the problem that I have with Solidity, for example, is that it is incredibly user-friendly. It's developer-friendly, easy to get up and running, but it's also deceptively friendly where simply leaving out a keyword can result in all the money in your smart contract getting stolen or frozen or what have you. And so that's why there are a number of different folks who are working on better standardization, better tools, sort of provable smart contract security, stuff like that. It's just, there's a lot left to be done before I think that this type of development is really suited for storing millions and billions of dollars. It's just really hard to be completely sure about the security aspects of it right now. Many people would decry the tribalism within the cryptocurrency ecosystem. You wrote about the advantages of tribalism. What are the advantages of tribalism within cryptocurrencies? Well, I think that it mostly comes down to network effects, is that you need some sort of culture that is kind of keeping people interested and keeping people on the same page as to what you want this project to value and to thus evolve in a certain direction. That's kind of one of the more fascinating sociological aspects of this whole ecosystem is if you don't have authoritarian type of governance, how do you attempt to govern the network or at least attempt to corral people together to try to get them to collaborate rather than just dispersing and fracturing into a million different networks that have probably far less value than if you collaborated and worked together. We see some very interesting results from that. Some of them are very toxic and there's a lot of fallacious arguments going on. But then there's also a sort of lighter side of things with like the memes and people coming up with sort of inside jokes to keep folks entertained. I try to participate in the more optimistic side of that without attacking too many people though. Sometimes you see some really terrible decisions that are made by some of the developers on various networks and it's hard to believe that they're able to get away with these things and still have networks that are valued at billions of dollars. There's a lot of nonsensical things that are going on too that I think maybe you could attribute to this system being overhyped in certain ways. It's certainly fascinating one way or another. It keeps me intrigued and basically plugged in constantly. That makes two of us. I wish we had more time today and as I said before the show, you're welcome to come back on the future. I really enjoy your sentiment in terms of talking and writing about cryptocurrencies. I do want to give you a couple of minutes to talk about your company, CASA, that you're building. I'd love to know what you're doing and what the long-term vision of CASA is. Sure. We're starting out with a Bitcoin vault product and anticipate eventually expanding that to support a lot of different crypto assets. Really what we're trying to do is bring the promise of being your own bank back into the crypto asset world where I think that it's always been possible to be your own bank, but the learning curve is so high and the amount of time you have to put into it and the technical requirements, they're just so onerous that the vast majority of people don't do that. Even people at my technical level that I've spoken to, a lot of them do not go through all of that rigmarole because either they just don't want to put the time into it or because they don't trust themselves. I think this results in a lot of people just keeping their money with third-party custodians and that is antithetical to, I believe, what the premise of this whole movement is. What we're trying to do is bring the usability and ease of use of a mobile phone app and combine it with the security properties that you get with a hardware signing device. When you create a CASA wallet or CASA vault, it's a three out of five multi-signature wallet and you're going to have three different hardware devices that you geographically disperse. You'll also have your phone with a key and the secure element and then CASA will have one key as a emergency recovery mechanism. Just having these keys mostly on dedicated hardware devices that are not in the same area is going to protect you against hacks, it's going to protect you against many different forms of loss, acts of nature or even physical attackers that may come into your home or business or what have you. We're really just trying to provide a comprehensive solution where we think through as many of the possible loss vectors and build them into our security model where we are taking the best practices and the experience that we've built up over the years and putting them into the software to help guide the users to do the right thing. What I think is going to be the very interesting experiment over the next year or so is to see how well that works, see how much of the best practices we can put into the software so that we can lower the level of touch that is required because we're starting out as a very boutique solution, about a $10,000 a year price point and if we can lower that level of touch and make it more automated, then we're hoping we can bring the price point down and make it more widely available. Well as soon as you become the wealth front of cryptocurrencies, you can count me in. All right. Jameson, thanks for coming on the show. It's been really great talking to you. My pleasure.