Responding to and recovering from a +$50m crypto hack of Haven Protocol - The Inside Story

This is a guest post by AHawk. AHawk discovered Haven Protocol in 2018 and has been a community leader for the project since 2019. As a crypto investor and enthusiast, he believes the concept of a Monero-based private stablecoin ecosystem will truly revolutionize how people protect their financial privacy and interact with crypto in the years ahead. 

 

Oh No... What Just Happened?

It is every developer's worst nightmare. A crypto project’s hell. 

In the early hours of the morning, when most normal people would be sleeping, this small core team was working at a break-neck speed.

The entire core team had just learnt about an exploit that had occurred on their protocol, and it was a race against the clock to minimize the financial impact. 

With a herculean response time and a full patch in less than 8-hours, the exploit would be almost completely stopped. But it was not without a wild goose chase, a hacker in their midst, and massive initial financial loss.

This is the true story of a small dedicated team, a community, and a hack that initially cost over +$50-million in damages. 

This is the story of the Haven Hack, and I’m going to tell you what it was like, from the inside.

Backstory

It started with a question. One that many crypto-investors overlook, but is critical to consider. I first started asking myself this question when I began to invest in cryptocurrency in early 2017:

If you wanted to protect your trading gains from both volatility and from a privacy perspective, how would you do so?

Perhaps you have new found wealth and do not want to become a target for scammers, or nefarious actors. 

Perhaps you live in a country where cryptocurrency is illegal, banned, or actively targeted.

Or perhaps you live under a government that wants to, has tried to, or might plan to control how you spend your hard earned income - eliminating your ability to preserve or build wealth for you or your family.

The reason behind your drive for privacy may vary, but the question still remains. How do you maintain your privacy, whilst protecting your financial gains from the instability of the market?

A cold-storage wallet is only part of the equation. As records and addresses on the blockchain are immutable and fixed forever on the chain, they can all be leveraged to uncover a vast amount of information.

And from the financial stability point of view, it has become increasingly clear that stable coins like Tether can freeze accounts if needed, meanwhile skirting around the uncertainty of actual 1:1 ratios of stablecoin to dollars.

It is clear that large tech companies and governments are moving towards all pervasive surveillance that will eventually enter the financial realm - threatening our ability to freely transact as we so choose.

Privacy in the real world has always been a concern of mine. Privacy in the crypto-world was never any different. And yet it has never (until recently) been solved.

Diving into the world of cryptocurrency, fascinating as it was, still left me with this nagging question. Which is what led me to the Haven Protocol project in early 2018.

On the morning of the attack, after nearly 3.5 years of painstaking work, this hack threatened to destroy everything that we had fought for.

Not One, But Two Points of Attack

It was early April 2021, when the core Haven team outlined a long-term vision to make the project truly decentralized and run entirely by the community. This was something that had always been a priority. 

By this time, a growing community had already developed, and the launch of new, private synthetic assets on the Haven network was just around the corner. 

But only a month after the initial launch of Haven’s synthetic, privacy focused stable coins, the hack happened. 

On the morning of June 23rd, our core team found out that an unscrupulous miner had attempted to modify the code - looking to take advantage of an unknown vulnerability in our miner-reward-validation code. By modifying the code, the hacker would be able to create more substantial mining rewards that were not legitimately due to the miner.

While patches were rolled out the same day, and the exploit was rendered unusable, this led the team to discover another vulnerability that had been exploited twice by the hacker. The resulting effect was that hundreds of thousands of counterfeit stable assets were minted, resulting in the unusually high selling volume of XHV (Haven Protocol’s token) on our partners' exchanges.

If left unchecked, the unknown (and potentially unlimited) inflation of xAssets could have sent the protocol into a death spiral - crashing the price of XHV and resetting years of hard work.

Immense Pressure

In a community run organisation like Haven, people come together from all walks of life to contribute to a larger idea. To push that idea forward.The concept of financial privacy unites the Haven team, everyone on the core team has given way to the project, sacrificing personal goals, and ambitions to deliver something with profound consequences for all. 

From foregoing positions at thriving tech startups, to leaving successful businesses - the desire to see financial privacy become a reality has brought everyone on the team together to pursue the same goal.

Seeing all at risk in an instant, while knowing that you and others have put in years of hard work, gives you a sinking feeling that is indescribable.

But the devs were literally working until the job was done, no sleep, no down time, no rest. The daily team meeting was like a war room, with a laser focused sense of direction and purpose I cannot begin to describe.

And while we were furiously working out how we would solve the problem, I was eerily confident we would.

Because although the hacker was very skilled, the team’s drive to keep the project going, to make financial privacy a reality, was something that I knew would overcome the situation.

Community, Collaboration, and Counter Attack

The response to the hack had to be, and was swift, precise, and deliberate. 

First, our core team contacted all exchange partners (including KuCoin, Binance, TradeOgre, and Bittrex) and requested that the exchanges close the XHV wallets. This was done to prevent the attacker from depositing and selling an unknown amount of XHV from the creation of the fraudulent xAssets.

Next the team took the unavoidable and admittedly extreme measure of disabling an aspect of the Protocol responsible for conversion metrics - this move essentially stopped the potential for any more funds to be converted and withdrawn.

With the hacker in our midst sowing seeds of doubt (and yes, you can still read the actual conversation between the hacker and one of our core team members here), the community was in constant discussion with the team.

Knowing that the hacker could still influence a vote to enable further exploitation or exchange of counterfeit tokens, the project still needed to trust the community with important decisions. A vote was proposed on how to deal with the hack and help recover the millions in stolen assets.

Voted in with overwhelming support, the community agreed to a full rollback to a blockheight before the attack, with many members sacrificing short term gains for the long term vision of the project. 

The rollback essentially reversed transactions that occurred after the attack, but due to the quick and decisive action taken to limit the use of the protocol early in the hack, it helped the project recover millions with limited impact to holders.

Why Does It All Matter, Why Fight So Hard Against An Attack?

Ultimate because privacy matters. 

“Arguing that you don’t care about privacy because you have nothing to hide is no different than saying you don’t care about free speech because you have nothing to say.” - Edward Snowden

It matters because it allows you to think freely. It matters because it allows you to choose a religion or belief that you connect with. It matters because it allows you to control your financial assets, spending or saving in a way that is best for you or your family.

Cryptocurrency has always held the promise of delivering on a private financial future. Being able to hold your financial wealth privately, no matter how large or small, should be your right and ability if you so choose.

And while this is something many will say Monero provides, it only allows you to transact privately - not hold your assets in a stable, unwavering form. Offering privacy without being subject to the ups and downs of the market, is something that at this point in time, only Haven offers.

Moving forward from this hack has required a lot of learning, and many improvements to our process.

Haven 2.0, Lessons Learned

As any small team with lightning fast growth can attest to, sometimes, when growing quickly, mistakes are made. We recognized that in order to move forward in a strong way, and learn from a +$50-million hack that nearly crushed the project, changes would have to be made.

The first is that the power and support of the community is now leveraged to a far greater extent. With Bug Bounty Programs in place (up to $100,000 worth), and improved communication, the community is more instrumental in finding, solving, and working together to build the project.

Additionally new validation methods have been put in place, alongside our core team’s complete overhaul and refactoring of the Haven codebase - an aspect that helps with peer review, auditing, and future protocol development. Importantly this step completely removes the potential for another attack using the attack surface that got the project into this position.

And finally our core developers have leveraged world renowned, 3rd party experts in cryptography and Monero, to validate and test new code that is deployed. The team contracted CypherStack for the audit of both the Mint and Burn Validation maths and the new code.

With the world changing at such a rapid pace, we believe that Haven 2.0 can be the future of private money, and may provide the victory for financial privacy that we should all be looking for. 

But to build it, one thing is clear: a private, stable, and resilient protocol needs a strong team and the input and support of an unrivaled community. 

Bio:

AHawk discovered Haven Protocol in 2018 and has been a community leader for the project since 2019. As a crypto investor and enthusiast, he believes the concept of a Monero-based private stablecoin ecosystem will truly revolutionize how people protect their financial privacy and interact with crypto in the years ahead. You can learn more about the Haven Protocol and the community by going here: https://havenprotocol.org/