End the Lie

Financial crisis revealed and resolved in the world of Bitcoin

Decrease Font Size Increase Font Size Text Size Print This Page

By Adam B. Levine

Contributing writer for End the Lie

(Image credit: Adam Crowe/Flickr)

(Image credit: Adam Crowe/Flickr)

Monday, March 11th saw calamity strike at the core of the Bitcoin system – over the last 3 years the number of users has increased exponentially, and the value along with it.  In the past we’ve seen large public thefts, but Monday’s event was fundamentally different.

Note: be sure to read Adam’s previous article, “Ulterior motives behind the Copyright Alert System”

Bitcoin is basically one big public distributed ledger (known as The Blockchain) where the ownership of every Bitcoin ever created is tracked, accounted for, and verified automatically by various participants in the system.  Transactions can be detected within a few seconds, but merchants generally take the hour or so to gather 6 confirmations to ensure the payment cannot be reversed.

Other participants, upon receiving news of the ownership transfer (payment), check the payer’s account to ensure that they haven’t already spent them publicly.  A confirmation is the return message saying “Yes, this all seems to be in order, the value is truly available to be transferred to the intended recipient and we’ve updated our ownership records to reflect that.”

With that background in mind, on Monday the blockchain split in two without warning.


Why’d it happen?

Call it growing pains – Berkeley Database (BDB) was used for versions of the Bitcoin software .7 and below and as luck would have it, will accept a maximum number of changes per block.  The protocol’s self adjusting difficulty seeks to issue a block every 10 minutes.  As Bitcoin usage has increased, so have the number of transactions contained in the average block.

So what happened?

On Monday, a single block went through the system containing 1700 transactions, and was rejected by the participants still running on .7 due to how many changes it made.  Had everyone been running .7, the block would have been rejected and the transactions processed in (hopefully) smaller blocks.

Because an upgrade to .8 was rolled out several months ago that switched to a different database system without the same limitation, we found ourselves with two separate blockchains (ledgers) each disagreeing with the other’s interpretation of reality.   Those running .7 saw too-large blocks as invalid, while to those running .8, everything was normal and the larger blocks were acceptable and built upon.

Cause and effect

An alert went out immediately (at 1:30am UTC) from the mostly volunteer open source development team letting merchants know to not accept Bitcoins for the next few hours while the system sorted itself out.  To miners, they asked that everyone revert back to .7 so no incompatible blocks will be generated. The average user was unaffected.

Crisis isn’t good for anyone, so participants quickly fell in line wanting to resolve the issue as soon as possible.   Within 8 hours things were back to normal;  If you live in America, chances are pretty good you slept through the whole thing, including the 23% drop in price, which vanished by morning.

Schrödinger’s Bitcoin

The “Double Spend” is the bogeyman of digital currency world. Since Bitcoins are just data and data can be copied, what’s to protect a merchant from being paid with money already spent elsewhere?  This is the function of the blockchain, which represents the network consensus of reality.

When that consensus broke down due to the fork, it created a state best represented by the Schrödinger’s Cat thought experiment – one in which a single entity simultaneously and paradoxically exists in two states, only one of which can eventually be real.

During the disruption, one large Double Spend actually occurred and was paid out in US Dollars.  When the faulty blockchain was abandoned, the reality consensus settled on the backwards compatible ledger.  In this case, payment processor OKPAY was left without the Bitcoins they thought they received, and were also still out the nearly $10,000 they exchanged for them.

This is the most valuable instance of a successful Double Spend in the Bitcoin ecosystem, which can only occur under the Schrödinger’s Bitcoin scenario described above.

Luckily this story has a happy ending.  Both parties “did the right thing” and have refunded each other. But will things work so well next time?  Will there be a next time, is this situation repeatable?

I asked Gavin Andresen, Lead Developer on the Bitcoin project about the risks moving forward:

Question: Are “Double Spends” possible under normal circumstances?

Gavin: Double-spend risk is not a “yes or no” — like most risks, it is “more risky” or “less risky”, down to “So little risk I won’t worry about it.  One of the reasons I still tell people “bitcoin is an experiment” is because everybody is still figuring out where the edge-case risks are.

Q: So for the average user or merchant, how much risk is there in every-day use?

G: For the, average user or merchant delivering a physical product to customers:  I’d say very close to zero risk after 3 confirmations.  Exchanges or merchants that deliver “cash-like” high-value products or services have their own particular set of risks, and they need to be more careful. merchants that deliver services that have a zero marginal cost to them (e.g. “Subscribe to my e-magazine”) have even less risk than users or merchants delivering physical products.

And all of that is modified if you have some trust in the person you’re transacting with.  I recently sold a couple thousand dollars worth of bitcoin to a friend who promised to send me a check in the mail, and I was happy to do that.

Q: In your worst case scenario, where you flat out do not trust the person but want to do business with them, what would you recommend?

G: For ultimate trust transacting a huge amount of value to somebody who I think will try to rip me off– 24 hours would be the “100% safe” time.  It really is confirmations and not time, but unless something wacky is happening those two are equal after 6 confirmations or so.

I pick 24 hours because that would be 120+ confirmations.  I can’t conceive of a situation where “we” (the shared consensus of the bitcoin network) would allow a 120-block fork to happen that double-spent

Q: Now that the issue has been resolved for a day, any lessons learned or things that will be improved on?

G: actually one really stupid/simple thing ‘we’ should have done long ago:  make it easy for bitcoin services to get alerts sent to their email/SMS/etc.  We’ll be concentrating on things like improving communication during crises, because no two crises are exactly alike.

Regarding specific tasks:  implementing a -alertnotify=<run some command> is very high on my TODO list. That’s easy to do, and is the thing I’m kicking myself for not doing sooner

Q: What does that mean?

G: That means “if you get an alert, run this command to tell me about it” where “this command” is specified by the service operator — send email, send sms, whatever

Moving forward

History would have us believe Bitcoin scandal leads to big losses in the trading value of the currency, but this time was different.  Despite the currency itself seeming at risk and the impossible momentarily made real, the price immediately bounced back up to near all-time highs.

Bitcoin may be small as currencies go, but the hourly chart below speaks volumes given that every 1K Bitcoins represents about $50,000 USD.


So here we are, less than 72 hours after the event, and we’re basically back to where we started.  The system is stable, the fork abandoned and the consensus reality once again agreed upon.  And all this with a basically volunteer development team who has no real ability to make anybody do anything, outside of telling them it’s a good idea.

Compare that to the ongoing crisis in currencies controlled by central banks, who have the unilateral authority not only to make all decisions related to their respective monetary policy, but also the ability to issue more bills at will, should the need arise.

Lacking such an option, Bitcoin is forced to deal with its problems and try as quickly as possible to return to a state of normality. So does it work? For now, the answer is a definitive “yes,” but since it was just created in 2009, only time will tell.

From where I’m sitting this was just about as good an outcome as anyone could hope for.  No value was lost, even though the system prides itself on irreversible transactions (which once you’ve dealt with Paypal’s merchant Chargebacks, you begin to see the appeal of).

The development team isn’t hiding the cause of the recent issue and is instead using this as a reason to dig into the response with work being done on an internal, comprehensive post-mortem.

They’ve taken the whole thing as a learning opportunity and have pledged to implement a more agile communication-alert system so if this does happen again, users will be alerted and large users will take appropriate precautions.

I like to see people with power admit they are not omniscient, and in this world of centrally planned money it’s too rare a thing.

One thing is certain: Bitcoin isn’t suffering for lack of its own Bernanke, and that should give us all reason to hope.

Adam writes about new technology at Mind to Matter and curates The Daily Bitcoin, You can contact Adam here.

Edited by End the Lie

Did I forget anything or miss any errors? Would you like to make me aware of a story or subject to cover? Or perhaps you want to bring your writing to a wider audience? Feel free to contact me at [email protected] with your concerns, tips, questions, original writings, insults or just about anything that may strike your fancy.

Please support our work and help us start to pay contributors by doing your shopping through our Amazon link or check out some must-have products at our store.

4 Responses to Financial crisis revealed and resolved in the world of Bitcoin

  1. Anonymous March 14, 2013 at 2:37 AM

    The block fork started prior to 23:11 March 11, 2012 when the first support request on the #bitcoin-dev IRC was made. The developers realized it was a hard fork at about 00:05 March 12, 2012 when the v0.8 side was at block 225,439. i.e., merchants already had blocks with six confirmations before it was even recognized as being a hard fork.

    If the alert went out at 01:30 March 12, that’s more than two hours after the fork began and more than an hour after the first block showed six confirmations after the fork.

  2. Adam B. Levine March 14, 2013 at 4:28 AM

    anonymous: Two hours from discovery to disclosure honestly isn’t bad by any standard. One thing I took away from this article is how complicated and wrong most of the things I thought about what happened were.

  3. Ben Bernanke March 14, 2013 at 1:46 PM

    After reading this, I feel honored to own Bitcoin.

  4. Rob June 12, 2013 at 12:26 PM

    With more people searching for Bitcoin realted information and sites, this is proving to be quite the valuable resource. Thanks again!


Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>