Cryptocurrency security is a concern to anyone who has a significant amount of money in the form of Bitcoin, Ethereum, or other crypto assets. Most Bitcoin is held in either a Bitcoin wallet or a Bitcoin bank. Your Bitcoin holdings are recorded on a public ledger. You access these holdings by authenticating with your private key. A Bitcoin wallet could be described more accurately as a Bitcoin key ring. Securing your Bitcoin wallet is about securing that private key. And just as there are many different ways to secure any individual piece of text, there are many ways to secure a Bitcoin private key. A Bitcoin bank is a term that can be used to describe institutions such as Coinbase. Coinbase takes the technology of the Bitcoin wallet and wraps it in additional layers of security, identity, and failover that we associate with banks and large technology companies. By using a Bitcoin bank, you sacrifice the autonomy of managing your own private key. On the bright side, you don't have to manage your own private key because private key management is annoying. But of course, by giving up that private key management, you are giving up some autonomy. A Bitcoin bank gives you the downsides and the upsides of working with a centralized service provider. Jameson Lopp is a cypherpunk and a cryptocurrency engineer at CASA. Jameson was previously on the show in another popular episode relating to cryptocurrencies. CASA, the company where he works, is a company that is building long-term cryptocurrency storage and secure key infrastructure. In this episode, we explore how Bitcoin wallets work, how to secure them, the common threats, the scams and the hacking attempts of Bitcoin, and what he's working on at CASA. Jameson Lopp, you are a professional cypherpunk and an engineer at CASA, which is a platform aimed at cryptocurrency holders. Welcome back to Software Engineering Daily. Great to be back. I want to talk to you today about cryptocurrency security, specifically wallet security. Let's start with the discussion of wallets. What is a Bitcoin wallet and why does a Bitcoin holder need one? Well, a wallet is not the best name thing. It's probably better to call one of these things a key ring or a key management service. But basically what you're doing is you're creating software that helps you manage both the private keys that unlock your crypto assets, allow you to spend them, and allows you to manage other aspects of your actual money. So in Bitcoin, that basically means managing the unspent transaction outputs, which are the real quote unquote Bitcoins. Those unspent transaction outputs, are they held in the wallet or are these publicly accessible? Let's just do a little bit of review on the interaction between a Bitcoin wallet and the public ledger. Yeah, so the UTXOs are basically the database of currently available spendable Bitcoins that you get from parsing the blockchain itself and looking at the entire history of the blockchain. And so the blockchain is just this series of inputs and outputs. Every transaction is spending some inputs and then it's creating new outputs. And by following along the entire history from the very beginning, the Genesis block, we are able to delete the transaction outputs that get spent, create the new ones that the new transactions get created when they're creating the transaction and broadcasting out onto the network and then getting into the blockchain. And eventually we arrive at the current state. So if you could actually think of transaction on the Bitcoin network as a state transition function, where you're consuming some transaction outputs and then creating new ones. And eventually we have the current state at the current tip of the blockchain. When I initialize a Bitcoin wallet, what do I need to do? Well, all you're really doing is creating those private keys. And there's a lot of different ways to do that, but there is a standard now for what they call hierarchical deterministic wallets, where you're really just creating this root master key pair. And then you're deriving all of your public private keys from that master key pair. And that is good for a number of reasons, but especially because it means that you only really need to back up that master key pair. And in fact, there are other standards around that that allow you to just back up a 12 or 24 word seed phrase that is then used to derive the master key pair itself. So when you create a new wallet, there's nothing in there except for some public and private keys. But of course, there's not going to be any value in the wallet if you've never used that set of public and private keys before, because nothing will have been sent to those addresses. So you do not need to scan the entire blockchain in that case and find all of the transactions that are associated with this wallet because you know that it's unique. Yeah, if this is the first time that you've ever used the wallet. Now, if you are importing a seed phrase or a bunch of key pairs that are from a wallet that you used previously, then you will need to go back and do some sort of scanning so that you can pick up those unspent transaction outputs. What you're describing there, if I'm already a Bitcoin user, I've got one wallet that I want to leave at home at all times. And then maybe I've got another wallet that I can walk around with and use occasionally. And I want them to be copies of the same account structure. If I was setting up that new wallet that was a copy of the same address schema as the first wallet, what do I need to do in that case? Well, in that case, what's going to happen is the wallet software is going to either from the very beginning point in time, or in some cases, you can tell it to only start from a certain date if you know when you first started using the wallet. But that wallet is then going to have to go out onto the network. Or you could, of course, have it connect to your own node, but it's going to have to connect to some full Bitcoin node somewhere out on the network, and then start querying that node. And it'll basically walk through every block that has ever been put into the blockchain. And it will send a query to the node saying, you know, here's a sort of fuzzy filter. I'm looking for any transactions that are related to any addresses that match this fuzzy filter. If anything matches, send it back to me, and I'll, you know, inspect those transactions more closely locally. And so you can do this pretty quickly because you can, this is actually called SPV, the simplified payment verification, which was actually described in the Satoshi white paper many years ago. And by using the the Merkle tree route that is in each block header, you can actually get a strong cryptographic proof from the node that specific transactions existed in specific blocks without having to download all of the transactions that ever happened. Now, without getting too deep into the details, there are some privacy issues here. There are some potential censorship issues, but it works pretty well most of the time. And of course, there are a number of proposals out there to improve upon this and make it even more censorship resistant and more private. Let's save the SPV versus full node discussion for a little bit later in the conversation. Sure. Focusing just on the wallets for now, what's the spec for hardware that I need? Can I just run this on my MacBook or on my phone? What are the requirements for secure Bitcoin wallet software? Well, that is quite a broad question. And if you know anything about computer security, you know that nothing is really 100% secure. So you really want to try to do is to minimize the attack surface of whatever wallet you're using. And of course, there's always going to be performance trade-offs and convenience trade-offs. And so generally, the more secure your setup is, the less convenient it's going to be or possibly less performant. So when it comes to running a wallet on regular hardware, like a laptop, a desktop, a mobile phone, what have you, those are always going to have more potential attack vectors because you're running a full-fledged operating system. There could be malware that gets on it. It's just like a million different things that could go wrong and could result in you potentially losing your private keys to some attacker or the attacker basically spoofing addresses, swapping them out when you're not looking and you accidentally end up sending money to an address that belongs to an attacker rather than your intended recipient. So for the average user who has more than probably $1,000 in crypto assets, I recommend going out and getting one of these hardware devices, whether it's a ledger, Trezor, KeepKey, ColdCard, what have you, because those are going to give you basically the highest level of security that you can get while still having a pretty good user experience. And in fact, we're building on that at my current company, trying to merge the user experience you can get with a mobile app wallet with the security that you can get with these hardware key management devices. And those devices are generally more secure because they're custom-built hardware that does basically nothing except managing the private keys and a secure element and doing the transaction signing operations for various cryptocurrencies. So it's basically impossible to get malware and other bad things onto those devices because they simply don't support other operations. That is, unless there's some kind of supply chain interloper, one thing I saw recently was that you have a store, the Kasa store, where because you're a dedicated security company, I think there's an argument to be made that it would be more secure to buy a Trezor wallet from your Kasa store than from, I don't know, eBay. You probably don't want to buy a Bitcoin wallet on eBay. You probably don't want to buy one from a random person. Now, the types of attacks that we're seeing when people buy a hardware wallet off of eBay or some other or some other more peer-to-peer marketplace, it's not that those people are actually going and tampering with the hardware itself. It's usually much less sophisticated where these people are buying a hardware device. They're then opening it up and putting a wallet seed phrase on there that they know the seed phrase. And then they're shipping it off to other people and basically giving them instructions telling them, oh, don't reset the device. Just keep using the seed phrase. Here's your seed phrase. And then of course they wait for someone to deposit money on it and then they steal it all. Now, I'm not a hardware expert, but if you go onto the Trezor and Ledger websites, you should be able to find more material where they actually have a number of tamper-proof mechanisms built into the hardware itself that make it very, very difficult for someone to open up the hardware and tamper with it without the hardware itself noticing. It's like the firmware basically checks the integrity of the hardware every time the device turns on. And while of course anything is possible with enough resources, like I think if you go to some three-letter agency that has like electron microscopes and all kinds of other equipment, they might be able to tamper with this type of stuff. But in general, you get a really, really high level of security that the device itself has not been tampered with. This is the Bitcoin wallet that we have now thoroughly discussed. There's also a term that is commonly used, Bitcoin bank. I think this might be what you could describe Coinbase as. What is the stack for a Bitcoin bank? How does a Bitcoin bank account compare to a Bitcoin wallet? Well, normally what we're talking about when we call something a bank, it's a large custodial service. And so they are managing Bitcoin and crypto assets for a large number of people. So that basically means they have to keep all of these private keys safe on behalf of others. And that means they're going to have much more sophisticated multi-layer storage solutions. When you're talking about Coinbase, for example, which last I heard they probably are holding at least like 5% of all the Bitcoins in existence, you can go onto their engineering blog and they have a post from a year or two ago where they talk about some of the security mechanisms they use. That basically means that the vast majority of these private keys are going to be held offline in a highly secured vault where you have to have multiple humans that are going in at the same time to open up the vault and get into it. And it's basically this huge Faraday cage with air-gapped computers on it. And whenever they want to send money from this cold storage into their hot wallets, then it's a very long convoluted process that has many, many checks and balances to ensure that they're safe, you know, not only from hackers and external threats, but even from insiders who might be incentivized to try to steal the private keys and, you know, jet off to some non-extradition country. In this description, you've outlined a difference between cold storage and hot storage. You've given some examples and I know cold storage and hot storage are extreme characterizations that there's actually a lot of granularity between what is cold and what is hot storage. Can you describe the types of accounts that would be classified as cold storage or hot storage and what falls in between? Yeah, so generally when we say cold and hot, we're basically saying it's connected to the internet or it's not connected to the internet. Obviously, when you disconnect a device from the internet, you are creating this moat or this air gap so that hackers cannot get to it. Like, that's the safest you can possibly be from hackers is just pull the plug. Now, when people think of cold storage, they usually think of a paper wallet. I actually don't recommend that people use paper wallets unless they're extremely advanced because there's just so many things that can go wrong with them. But I generally say that a hardware key management device is cold storage because the device itself, while you still plug it into a computer or even a phone in some cases, the way that these devices are set up is that the private keys can never actually leave the device even though there is a cable connecting the device to another computer. The data that is transmitted between that cable is very limited and the keys never get transmitted off the device. So, the other side of things, of course, is a hot wallet where you actually have these private keys on a machine that is either constantly connected to the internet or sometimes connected to the internet, but it's just a piece of software that is managing the private keys on a general purpose computer. And the problem that you have there is that even though a good wallet is going to keep those keys encrypted in a file on disk at rest, at some point in time, you're going to have to access those private keys. You're going to have to decrypt that blob of data in order to be able to sign a transaction with it. And even if that's only for a matter of milliseconds, if there's some malware or something else sitting on the computer waiting for you to do that, then it can swipe your private keys and steal all of your crypto assets. And that's the primary problem that we run into with hot wallets. And that's why exchanges get hacked all the time is because they have to be running hot wallets in order to be able to do automated withdrawals. Failure is unpredictable. You don't know when your system will break, but you know it will happen. Gremlin prepares for these outages. Gremlin provides resilience as a service using chaos engineering techniques pioneered at Netflix and Amazon. Prepare your team for disaster by proactively testing failure scenarios. Max out CPU, black hole or slow down network traffic to a dependency, terminate processes and hosts. Each of these shows how your system reacts, allowing you to harden things before a production incident. Check out Gremlin and get a free demo by going to gremlin.com slash S E daily. That's gremlin.com slash S E daily to get your free demo of how Gremlin can help you prepare with resilience as a service. Do you have prescriptive or proscriptive protocols for who should have what kinds of wallets? Cause I assume, you know, somebody who's completely unsophisticated with cryptocurrency, you would recommend one type of wallet set up for them versus somebody who is a security expert. And there's probably also the dimension of how much you are storing, what percentage of your net worth you're storing. What are the different combinations of wallets and configurations that you recommend for different types of people? Yeah, I've, I've written about this a little bit in general. I kind of broke it down into like three different categories that probably require different levels of paranoia. And so your, your very entry level category is just spending money, you know, a hundred dollars, couple hundred dollars, basically whatever level of cash that you might normally be walking around with in your pocket. That is an amount that would be fine to just keep on a regular hot wallet on your mobile phone or, or any other device, because it's probably not going to ruin your life if something happens and you lose that. Now the, the next level of storage when you start getting more intermediate level is, is more like a small investment. So if we're talking about like a month salary or, or more, you know, getting into the thousands of dollars range, then that's when it starts to make sense to invest 50 or a hundred dollars into one of these hardware devices. And that's going to give you the best security without having a huge investment of time or money. But then the third level is when we start talking about like life changing amounts of money, like hundreds of thousands or millions of dollars worth of crypto assets. And that's when you're probably going to want to devote more time and resources and money into a more complex and more redundant system, because you're going to be worrying about not only loss due to hackers and external attackers, but even more commonly, most of the loss seems to happen just due to user ignorance or negligence, not having a sufficient level of redundancy and robustness against any type of natural disasters. And so that's really the type of level wallet that I've been working on this year with Casa. And we think that there's a pretty big niche for that, where a lot of people started out with that first or second level. And then as these bubbles keep happening in cryptocurrency, they wake up one day and now they have a level of wealth that they never thought they would have to worry about managing before. So what are some of the, I want to ask about the scams and the thefts and stuff, but maybe we should first cover that the user level incompetency. What kinds of mistakes do people make? Particularly, you already mentioned paper wallets and obviously with paper wallets, you could just lose the paper wallet or spill water on it. You can have a computer hard drive where you've got your private keys stored somewhere and then you accidentally lose the computer. We've heard many stories about that. What are the common scenarios where people just make mistakes? Yeah, I mean, I think it's usually someone either not creating a backup in the first place and they're just running a hot wallet on a computer and then there's some sort of hardware failure, or they have one backup somewhere and something happens to that. So we've even heard, there was a really good story. I think it was a Wired journalist or someone a year or so ago where he had a backup of his seed phrase, but it was just written on a piece of paper in his desk and apparently the maid came through and thought it was a scrap piece of paper and threw it out. And so it's usually just not having good IT practices and not having multiple offsite redundant backups. And then even if you do have good backup practices, then a lot of times people won't actually test them. And so they might have some flaw in their backup and recovery mechanism where, sure, they have backups, but then when something happens and they go to try to recover it, they find that flaw and now it's too late. And so we're just, we're trying to think through all of these different possible failure scenarios because I think most people just think about, oh, my house might burn down or there might be a flood or something. And so all I really need is one of those, you know, metal storage devices where I etch the seed into that. And I think those are good against a lot of common threats, but I actually had a stress test blog post that I wrote recently where I showed that there are a number of things that can go wrong that can even make those metal devices sufficiently unreadable, even if the metal itself isn't completely destroyed. And then even more recently, we've been hearing things of, for example, bank safety deposit boxes getting opened and thrown out without the owner necessarily even knowing what's going on. And so I think there's also a lot of people who are just putting their a single backup recovery phrase into a bank safety deposit box and not thinking that, oh, there are actually potential failure scenarios for that as well. And the scams, the Bitcoin theft scenarios, the tricks that people can wage against each other. We've seen so many of these over the years. What are the most common types of scams and Bitcoin theft scenarios that you've seen? Well, the most pervasive right now, and I don't know how successful they are, but there sure are a lot of them on Twitter with the various giveaway scams. I think that's getting some of the more naive folks to send their money off thinking that they're going to get more money back as a result. There's also a lot of exit scams from seemingly legitimate businesses. And this is why having a long history and reputation is also pretty important. If you're going off and using some exchange or other service that's only existed for a few days or weeks, then you never know. They might just amass a lot of deposits and then disappear off into the ether. But in general, you just have to go with your gut of, you know, if this sounds too good to be true, it probably is. There's just such a much higher reward for scammers in the crypto space because these are bearer assets. And so once you send them, there's no way to call them back. And so I think that's why more and more scammers keep flooding into the space. Okay, let's start to talk about how to build a more secure wallet and money management system, since that's what you're focused on at Casa. So when Casa was getting started, what were the predominant mechanisms for storing cryptocurrency? And what made you think that there was some gap and there was some set of remedies that you could provide that would be a sweet spot in terms of security? Well, over the past few years, these hardware wallets have become more popular. But we were just hearing a lot of horror stories of people having millions and millions of dollars on this one single device that even though it's incredibly secure against various types of attacks against the device itself, just the fact that you have a single device creates, of course, a single point of failure. And we kind of came to the conclusion that there's a bit of a usability flaw even with these high security devices, because if you buy one of them, the first thing that it's going to do is it's going to display this 12 or 24 word recovery phrase to you. And then it just says, keep this in a safe place. And while that seems simple enough, it's actually a huge ask, I think, of average person to understand all of the ramifications of security and good IT practices of keeping 12 words safe against all kinds of different possible loss scenarios. And loss, of course, doesn't necessarily mean attack and loss to some other person. It can just mean data loss that makes the thing unrecoverable. And so what we found when we started asking around is that there were a lot of OG crypto folks out there who were just petrified to be responsible for storing their private keys. And so a lot of them were just leaving them with custodians, because they had this thought process of, well, if I leave all my bitcoins at Coinbase or some other exchange that's pretty reputable, then they might actually be more safe there, because there's whole teams of people that are devoted to the security of the cold storage that is being managed by these companies. And I think that's a pretty logical line of thinking for yourself that very well may be more secure and more robust against certain failure scenarios. But of course, it kind of goes against the whole ethos of the system, because now you're just creating more single points of failure where we're now putting billions and billions of dollars into these Bitcoin banks, and that is creating systemic risk. So we wanted to try to kind of push back against this centralization of this aspect of the ecosystem and make it easier for people to be their own banks and still be able to sleep at night without having to spend a large portion of their life actually thinking and worrying about all of the stuff that's required to be a good bank. To the point of centralization, let's talk about centralization for a little bit. This is slightly off topic, but I think it's important to think this is slightly off topic, but I heard recently that Bitmain, which is the company that mines most of the Bitcoin in the world, I think they control at least enough hash power to have a 51% attack if they wanted to. Is that problematic? Is that fundamentally problematic to the health of Bitcoin? Yeah, I mean, it's hard to actually measure their level of hash power because they also have all of these other business connections. Now, they definitely build the vast majority of the machines with the hash power as to who controls the specific machines. It gets a bit murkier, but it's not good in the first place. I mean, it's not optimal. It would certainly be better if we had like four, five, six or more companies that are actually producing these ASICs and then distributing them more widely than they are distributed at the moment. Now, the main thing when it comes to 51% attacks, you kind of get into the game theory behind it, is that I'm not worried that Bitmain itself will try to 51% attack the network because they would be shooting themselves in the foot. They would cause such a massive loss of confidence in Bitcoin if that happened that the price would plummet and however many billions of dollars worth of Bitcoin that Bitmain has and of course, their probably future revenue would be severely impacted. Now, it is more concerning from the fact that, well, this is a single company and they could then become a target of nation state and some nation state might come in and seize all of their operations and then say, okay, well, we want to try to kill Bitcoin now, so we're going to put a gun to your head and force you to 51% attack the network. Of course, it's hard to kind of put a level of risk on what that scenario might actually be, but we don't want that to be possible, if at all. If that occurred, you've probably run this scenario out in your head a couple of times, would there just be a fork and Bitcoin would figure out how to deal with it? Yeah, and this is kind of the fundamental strength of Bitcoin is that while we have this great machine consensus protocol that is automating the consensus of the current state of the network, if that fails and it has failed a few times in the past, then you have a layer of consensus below that, which is the human consensus. So, if machine consensus starts screwing up, if the network ceases to operate, then all of the major players on the network and really anyone who's paying attention and cares enough is going to start conversing and saying, you know, this is bad, we need to fix it. What's the best way to fix it? So, if for example, there were some 51% attack that was ongoing and it looked like there was no reason to believe it was going to stop, then you would see people come together and say, okay, what protocol change needs to be made in order to stop this attack? And so, the extreme example, of course, would be a proof of work algorithm change, which would make all of those ASICs essentially worthless. Thanks for describing that and going there with me. And one reason I took that slight deviation is because there are some people who listen to the show that are still not completely convinced that Bitcoin is a thing worth paying attention to. And so, I like to occasionally revisit some fundamentals, particularly the kind of thing that you just described, which is essentially a really strong durability guarantee of the network, which makes it so beautiful. And I hesitate to use the word indestructible, but it's hard to imagine something that will destroy it at this point. I mean, you've had the best security experts in the world try to think of a way to destroy this thing or to disrupt it forever. And I haven't heard one yet. So, it's pretty strong, strong durability guarantees. Yeah. Well, I mean, it kind of goes back to the cypherpunk manifesto and some of the ideas around what they were doing. And if you read that manifesto, I believe there is one section that says, we believe that sufficiently widely distributed software cannot be stopped. And so, this system, while it does have its points of centralization that could be used to attack, it's fundamentally driven by all of the humans who are interested enough in maintaining and improving the system. And those people are so distributed around the world. And this is why I've been doing so much global travel over the past few years is because they're in jurisdictions that are very diverse and composed of individuals with wide diversity of perspectives. And so, it's just the issue of like, there's not enough people or attackers out there who can kick down all of the doors of all of the people who are helping to maintain this network. It's just, it's too distributed. It's like trying to attack a swarm. Coming back to the discussion of Bitcoin bank centralization, now that we have absolved ourselves of the question of Bitcoin centralization globally, you've described some fallibilities of the Bitcoin bank model as it exists in something like Coinbase. And Coinbase is a paragon of the community. I love Coinbase. I've done several shows about it. I have used Coinbase in the past, but we can agree that there is some centralization. It's a victim of its own success, essentially. And you look at that and you see something, you see an opportunity, perhaps a more decentralized Coinbase model is what I'm hearing you kind of allude to. What's the solution there? What's the way to build a Coinbase-like entity where you have security guarantees that are in some sense associated with the reputation of this large organization, but you don't have the failure scenarios inherent in a large organization? Well, this is one of the reasons why I'm so excited about the developments in the decentralized exchange space. And so there are already a few decentralized exchanges out there like BISC, but these are basically software that are creating their own peer-to-peer network and creating order books and allowing you to find other people that you trade with directly. And so this once again means that you're taking on a little more risk and responsibility yourself because you're having to manage the private keys and run the software. But it means that we're no longer creating these large honeypots with enormous amounts of value where an attacker only has to get into one door in order to sweep up a lot of money and harm a lot of people. And so by further distributing that value around and once again creating more doors that attackers have to get through, then it's decreasing the systemic risk in the system. But of course right now the software is a bit clunkier. It's not going to be as user-friendly and it's probably going to be a number of years before that type of software is able to compete on the usability front with something as centralized as Coinbase. So does this relate to what you're building at CASA or do you see this as a different question? Well with regard to exchanges it's definitely different. Now the thing is Coinbase is both an exchange and a wallet. So they're kind of doing multiple things. At CASA we're more focused on the secure store of value side. But you know there's definitely some overlap there where we want people to be pulling their value out of third-party custodians who could make any number of mistakes or even be coerced by other larger entities that are more powerful and you know result in loss. But it's a multi-prong I think type of evolution that we see a lot of different teams working on a number of different projects in order to try to further decentralize the various aspects of the space. So the problem that you are focused on specifically right now is long-term store of value. Correct. Okay and can you contrast how you're thinking about solving that problem with the solutions that we've already discussed so far? Well one of the the big differences that if you want to take on the responsibility and be your own bank as it were in the crypto space then right now you have a lot of education that you need to take on in order to do it correctly. And what we're trying to do at CASA is to facilitate the technical side but also provide a lot of support just on the human side where you know we are providing more of a boutique experience helping people set up and maintain the wallets themselves but without us actually having control over more than one out of the five keys to their vault. Okay and how far along are you in the development of the long-term storage software that you're building? On the bitcoin side it is up and running and we've already onboarded a number of clients and of course learning more as we do that every time we onboard something you know we find a little small other thing that should probably be tweaked and we're really focused on scaling up the onboarding and then scaling out towards supporting all of the other popular crypto assets that our clients are demanding to have the same level of security around. What are some examples of design decisions and implementation decisions that you've tweaked over time with customer interactions? Most of it is going to come down to figuring out like the simplest way to describe you know a series of actions that they need to go through so we're trying to shove as much of the technical complexities under the hood as possible but there are times when customer might hit an edge case where you need to give them a warning or give them a choice some sort of decision to make so you know instead of you know if they have a wallet that's full of dust for example instead of saying something like oh you have too many UTXOs and you're trying to send a value of this and it's not possible to compute you know figuring out what a more human readable message could be so that they have an easier decision to make and so you know giving them some options of saying like well if you want to send this level of value you may need to wait this many hours in order for your transaction to complete type of thing. By the way when you say a wallet full of dust you're talking about fractions of bitcoin that are so low that they're beyond beyond a nomination of anything that would be of value in the real world? Well yeah so it you can get into both technical and economic edge cases when you're dealing with wallets and I actually I wrote an article I think two years ago called the challenge of unspent transaction output selection and it's just all of these edge cases that users can get into because they don't understand what's going on over the hood and they can basically fill up a wallet with a lot of UTXOs that in certain situations if the fee market goes up to a certain point you actually have to build in some logic that we've been calling economically unspendable transaction outputs where you then need to figure out how to kind of guide the user in the right direction to help clean up their wallet without having to expose them to all of the complexities of what's actually going on inside of the wallet. Digital Ocean is a reliable easy to use cloud provider. I've used Digital Ocean for years whenever I want to get an application off the ground quickly and I've always loved the focus on user experience, the great documentation, and the simple user interface. More and more people are finding out about Digital Ocean and realizing that Digital Ocean is perfect for their application workloads. This year Digital Ocean is making that even easier with new node types. A $15 flexible droplet that can mix and match different configurations of CPU and RAM to get the perfect amount of resources for your application. There are also CPU optimized droplets perfect for highly active front-end servers or CI-CD workloads and running on the cloud can get expensive which is why Digital Ocean makes it easy to choose the right size instance and the prices on standard instances have gone down too. You can check out all their new deals by going to do.co.se daily and as a bonus to our listeners you will get $100 in credit to use over 60 days. That's a lot of money to experiment with. You can make $100 go pretty far on Digital Ocean. You can use the credit for hosting or infrastructure and that includes load balancers, object storage. Digital Ocean Spaces is a great new product that provides object storage and of course computation. Get your free $100 credit at do.co.se daily and thanks to Digital Ocean for being a sponsor. The co-founder of Digital Ocean Moisey Oretzky was one of the first people I interviewed and his interview was really inspirational for me so I've always thought of Digital Ocean as a pretty inspirational company. So thank you Digital Ocean. So since you are building wallet management software there are these trade-offs that you need to make between how easy you make it for somebody to transact with that money versus keeping it secure because if you expose the user to some vulnerability through two-factor authentication of a cell phone provider that makes them vulnerable to a cell phone takeover attack then that might be more convenient than something that is a more stringent security protocol such as perhaps Google Authenticator. It's not a great example because Google Authenticator is just about as convenient as going through a cell phone based transaction provider but I think the thrust of my point stands. You know you've got these trade-offs between security and convenience that you've discussed a little bit earlier. Tell me where you're trying to play in that that set of trade-offs or what's what are some specific trade-offs in that security versus convenience that you've made. Well we're definitely prioritizing security because if you fail on the security front there's no going back. The general mantra that we have is that it's better for the user to lose access to their wallet for some period of time than it is for an attacker to gain access for even a few milliseconds and so there certainly are situations where a user might trip over some security alert that causes their wallet to become unusable for a little while or they they might you know misplace some of their hardware devices and then require a CASA assisted recovery for example and you know that process could take hours or days to actually go through and then go through but we think that the trade-offs there when we're talking about you know vault type products that are potentially this person's you know life savings that you usually you don't need to access your life savings on a very short notice and so that's an acceptable level of inconvenience. I want to revisit the discussion of SPVs and full nodes. I'm not sure if it relates to product that you're building at CASA does it? Does the decision to validate or build a wallet through an SPV versus a full node does that impact your customers at all or does that impact you or do you think of this as just a completely disjoint concern? It is all related and so one of our primary tenets I guess at CASA is to help maximize user sovereignty and safety and so so far we've mostly been talking about safety but the sovereignty aspect is important and that definitely comes in when we're talking about running a full node you know validating that the money you received is in fact you know abiding the rules of the network that you are agreeing to and it also comes down to trying to minimize the trust between the user and our own service and so you know right now by any number of measures CASA itself is fairly centralized but we're going to continue to work on pushing as much of the security and trust out to the edges so that the clients themselves are performing more of the the validation and not relying upon any data that CASA servers are giving to them in response to a query so that's more of a medium to long-term vision but it's definitely all related. Speaking more generally for people who are managing their own wallets or they have different solutions for managing their currencies than CASA how should they think about SPVs and full nodes? Why do you have some you know strong opinions about the costs of using SPVs? So the the trade-offs of using the simplified payment verification is that you are making an assumption that the the blockchain with the most hash power is the correct blockchain and a lot of people are fine with that assumption and it works well in the vast majority of cases but there are a ton of things that miners could do for example to to start creating a blockchain that does really screwy things but still has the most hash power so it's you know one of the reasons I'm not a fan of that is because I don't think that we should be trusting the miners you know they have a job which is to basically time stamp transactions as part of the global consensus but I think that the onus is on the rest of us to provide checks and balances and make sure that we're not trusting them to be doing that job we should be you know verifying all of the work that they're doing. There's also some major privacy issues where I believe Matt Corallo and Jonas Nick have shown that the the bloom filter mechanisms that get used by SPV they were originally thought to be fairly private but over the years we've come to determine that they're actually not they're they're terrible at privacy and and it's very easy for someone who's running nodes out on the network if they wanted to actually figure out which addresses belong to you by inspecting those bloom filters and so you know once again it comes down to this convenience of you know it's really easy to start up a wallet on your desktop or your mobile device and the SPV consumes very few resources on the client side and it's it's pretty fast to to get synced with the network but you are making these privacy and security in a way trade-offs so that's why there are some improvement proposals out there for better client-side filtering where I believe it was Lalu Roast Beef has this mechanism that he has proposed where the client would basically be downloading the data but filtering it locally so that no one actually knows what data you're interested in which should give you more privacy on on that standpoint let's talk more about casa and building a business around cryptocurrency I have a number of questions about security so casa has I'm guessing what 10 or 15 engineers how big is the company at this point uh yeah we've got around a dozen or so employees so that's a big organization I mean it's a sizable organization it's not tremendously big when you have your your security product set up your your cryptocurrency provenance product set up do you get it routinely audited by external security people do you get it audited by people who can help you understand how to that you're secure against natural disasters for example the tail risk questions how do you make sure that your cryptocurrency holding product is resilient beyond all of the tail scenarios absolutely so yeah we we get you know regular external third party audits of our whole code base and infrastructure the trickier thing though is when we're talking about the the robustness of like any given client's wallet is that because we are giving more sovereignty more control to the user we can't force them you know to follow best practices we can certainly advise them and try to build features into the software itself to try to guide them in the right direction but so for example when you create a wallet with casa it's going to create one key pair on your phone that gets secured by apple's like secure element key ring encryption functionality and then it will create one key pair that are kept offline by casa for disaster recovery but then you're going to have three different hardware key management devices so like treasurers and ledgers that you have purchased and then initialize and set up and so as a three out of five multi-sig solution now obviously the point here is that you want the wallet to be multi-sig requiring multiple signatures in order to spend from the wallet you want it to be multi-device you know with preferably diversity of different devices to once again prevent single points of failure and then one of the most important things for redundancy is you want it to be multi-location you want these keys and devices to all be geographically spread out and while we're certainly going to tell this to our clients we can't force them you know not to keep all of their hardware devices in a drawer in their desk at their office or at their house or something and that's just like one of the the trade-offs that comes with giving more control and more responsibility to the end user. Tell me about the product roadmap for casa let's say you get this cryptocurrency holding problem solved to a great degree where you're just getting lots and lots of customer balances that are putting their money into long-term storage what's the next product that you build and what are the guiding principles for the longer-term product development strategy? Well the extremely high level view from casa is that we want to be the best personal key system that is available for people and so right now of course that means crypto assets but we believe that if the world keeps going in the direction that it is where it becomes less about having you know centralized authentication and identity and more about having your own public private key pairs that prove that you should be able to access certain things that having robust key management solution is going to be important not only for your financial life but for many other aspects of your daily life so we want to continue building out you know secure but usable key management software so you know obviously we're going to be supporting any of the the popular crypto assets that our clients want to support but we think that long term it's going to go a lot further than that and i think that one way to make this more clear is if you actually know the history of casa and casa the guys who really started working on what became casa were originally about a year and a half ago working on a block stack app called bedkin and it was basically a decentralized airbnb on the block stack platform and they had started working on this for a few months and eventually realized that there was no really good key management solution and they they weren't sure how they were going to get people to use this app if they weren't going to be able to easily secure and manage the keys to control access to the app itself and so that's where they pivoted and they decided to go a layer deeper into the problem as it were and then perhaps someday be able to pop out and and start building and facilitating other things that are using this type of distributed infrastructure but that is kind of the the high level vision of you know you could call it crypto anarchy if you will building many many different things on a more distributed platform that requires public private key cryptography to use on a day-to-day basis there's a product called keybase that we've done several different shows on and i feel like keybase was pretty prescient in realizing that key management is going to be something that consumers are going to have to deal with and it's an open question as to what level of technical detail the consumer is going to want to get into what do you think of the keybase solution how does that compare to your ultimate vision of what casa would provide with key management well i do like keybase especially just from the way that they have that key tree where you know you can authenticate new devices and new accounts and they're basically signed off by your existing keys to add them and i think that that type of thing is great in terms of the flexibility of having sort of a digital cryptographic identity i guess the the only thing is i'm not sure that keybase is supporting any you know hardware specific key managers it seems to be all in the software right now so it's still somewhat vulnerable from that standpoint okay do you see yourself becoming something like a bank in some regard because if you have all these balances that customers are depositing with you do you see that as something to be lent against i don't know that we would have you know that type of financial service because we're not custodial so we don't actually have control of the asset now there are a number of services out there like unchained capital or block fi or salt or what have you where you can do these you know crypto backed loans and i've actually used them myself and they work pretty well but i don't think it would be possible for casa to do that because we wouldn't have the ability to actually like you know freeze someone's money and and prevent them from taking it away if if they were trying to to back you know some other loan or service with their assets and and i don't think that casa wants to be responsible for actually custodying any of these assets you get into a whole new level of like regulations and issues with the traditional financial industry when you start custodying assets so that's the nice thing of you know spending three years working at bitgo and now working at casa is that while we are providing financial services we are only really regulated from the sense that we are a software as a service company and it just gets rid of a lot of headaches and it might even make more sense for a company like that to be built on top of casa like i would love if if i'm using block fi which by the way i don't know anything about that sounds like a company i should inspect more closely in that other one you mentioned i think unchained capital sounds interesting too but those sound like companies that could use casa i mean they probably don't want to have to build the security that you're building yeah what other i guess just to wrap up what are the other platforms that you think could be built on top of a company like casa i mean obviously today it's more for individuals but are there other services that you think people could build on top of casa it's tricky because you know we really are more focused on individuals at least at this point in time you can get more complicated if you start building like enterprise treasury management logic into the software and a better example of that is bitgo and and the software that they have built where there's a lot more permissioning and multi-user management around it and and so you know it's certainly possible for casa to go into that direction but you know for now we we kind of figure there's already enough enterprise focus companies out there and not necessarily enough individual focused ones so anything is possible especially when you start talking about some of the more complex scripting operations that are going to get built into the bitcoin protocol over the coming years you know we already have some ideas of stuff that we would like to implement around there to make the vaults themselves even more secure and robust for now i think we're going to be busy enough just uh trying to to build a user-friendly vault product and so where it goes from there let's just say we have some ideas but they won't be public for at least a few months okay well jameson thanks for coming back on the show and sharing your time i really enjoyed talking to you thanks for having me cloud computing can get expensive if you're spending too much money on your cloud infrastructure check out do it international do it international helps startups optimize the costs of their workloads across google cloud and aws so that the startups can spend more time building their new software and less time reducing their cost do it international helps clients optimize their costs and if your cloud bill is over ten thousand dollars per month you can get a free cost optimization assessment by going to d o i t dash i n t l dot com slash s e daily that's d o i t dash i n t l dot com slash s e daily this assessment will show you how you can save money on your cloud and do it international is offering it to our listeners for free they normally charge five thousand dollars for this assessment but do it international is offering it free to listeners of the show with more than ten thousand dollars in monthly spend and if you don't know whether or not you're spending ten thousand dollars if your company is that big there's a good chance you're spending ten thousand dollars so maybe go ask somebody else in the finance department do it international is a company that's made up of experts in cloud engineering and optimization they can help you run your infrastructure more efficiently by helping you use commitments spot instances right sizing and unique purchasing techniques this to me sounds extremely domain specific so it makes sense to me from that perspective to hire a team of people who can help you figure out how to implement these techniques do it international can help you write more efficient code they can help you build more efficient infrastructure they also have their own custom software that they've written which is a complete cost optimization platform for google cloud and that's available at reoptimize.io is a free service if you want to check out what do it international is capable of building do it international are experts in cloud cost optimization and if you're spending more than ten thousand dollars you can get a free assessment by going to doit-intl.com slash se daily and see how much money you can save on your cloud deployment wow so