796 Theft of 27 Jan

Bitcoin thefts come in all shapes and sizes.  Recently 796, a Chinese bitcoin exchange, has one thousand bitcoins stolen from them via a series of social and technical attacks.  This isn't a very large theft, only representing less than CN¥ 1,500,000,000, which translates to US$250,000 or the cost of a double wide trailer in silicon valley.  But the real question to be asked in this blog is this: does the evidence on the blockchain match the narrative presented in the official explanations?

The Narrative

CoinTelegraph was able to talk with 796, and in combination with a post on weibo (hope you can read Chinese) the following narrative emerged.  Being a bitcoin exchange they are always a target for theft and abuse, and three months ago they migrated their servers to a cloud host.  However somehow a hacker was able to replace one users address with another similar address.  At this point the hacker needed to simply wait for a transaction to be posted.  And they got lucky with a large 1000BTC transaction.

This is the point of the story where one of the ugly truths of cyber defense becomes manifest.  You have to be 100% in all of your defenses and secondary systems, but the hacker only has to get lucky once or twice.  This hacker had already had some luck getting the data updated on the server.  They had another stroke of luck getting the bitcoins out.  First, the weibo post indicated there was another level of approvals needed to permit the transaction because the transaction looked suspicious because of the use of different IPs.  But the transaction met the standards of approval so the transaction was processed.

About five hours later the customer who initiated the withdraw is in contact with 796 over the phone and they indicate the bitcoin had not shown up in the customer's destination address.  At this point a full scale investigation is launched, and it is revealed that the coins went to an address that was similar to the address that was intended.  This is where the hacker got lucky: by using a similar address it passed both 796 manual review and the customers manual review of the information.  Because when it comes down to it 35 characters of randomness does get tiring on the eyes after a while and if you have to commit it to memory, you probably only memorize five characters or so.

After the details are sorted out and the investigation is concluded 796 owed up to the theft, and took the losses out of company profits.  For 796.com client losses have a senior claim to shareholder losses, so there was no vote or consultation with the investors.

The Blockchain Evidence

On the 8btc.com forum (8btc is a Chinese language bitcoin forum) one post identified the theft address as 1CvGkU...2wrU.  There was one large transaction at 15:50 GMT on 27 Jul.  The whole of China is on the +8:00 time zone, so the timing of the transaction would be 23:50, consistent with the weibo post.  It was then committed to block 340748 a nearly a minute later.  So did a suspect transaction go out?  The evidence does match the narrative. 

But there is also the second half of the narrative.  796 also said that they would take responsibility for the theft.  About 13 hours later another withdraw of 999BTC is sent to the address 1CvGkM...vJwU.    Not that the first four characters of this address matches the first five characters of the theft address. It also matches up with the claim of the insertion of a similar address.  Note however, that this is 1 BTC short of making the customer whole.  If we look at the other transactions to the 1CvGkM address we see another transaction an hour earlier.

Ignore the dates on the transactions.  These are self-reported by the miners and the protocol allows for something on the order of two hours of skew. In the block prior to the 999BTC transaction a 1 BTC transaction is sent to the correct address.  I presume that this was done to test the waters of their system to make sure that the reported address can be trusted.  The next block is when the 999BTC is sent, and three blocks after that the customer then gathers the coins and spends it in two separate transactions.

These two transactions made the customer whole and the timeframe seems reasonable enough, being the next business day during business hours in China.

Technial Details

You have probably noticed that I elide the transaction identifiers and addresses in my images and in my text.  There is one principal reason for that, and it's not to protect privacy or redact data.  The reason is that 64 hex digits in a row and 35 alphanumerics in a row are hard on the eyes.  I take the six leading characters and the four trailing characters because one sixth to one third of the data is enough to reproduce my findings.  There are many occasions where a single address or a single transaction when elided will look like another address or transaction id.  However, when you assemble the network and take the elided addresses and transaction ids in context, the elided data provides enough information to reproduce the data displayed in my images.

The cognitive fatigue of looking at all of that data played a part in this hack, so it is my hope that by slashing the amount of data shown that the user might be able to process all of what is shown, rather than a part of the whole that varies between users.  I am fairly certain that it is good enough for analysis.

In fact, if anyone can show me a transaction on the blockchain where one address combined with the input and output transactions cannot be distinguished from a similar transaction based on the elided values, I'll send you a box of girl scout cookies. Of course I'll only pay up when girl scout cookies are on sale.  If you can show me one below block 340950 then I'll even let you pick the (in-stock) flavor of cookies.  This offer is good only for the first taker however, and only in jurisdictions where they are legal (they are addictive after all).

BitStamp Theft - Two Weeks Later

Before I get into the analysis I'de like to thank all of you who have found me in the past two weeks. What used to be a sleepy and infrequent technical blog resulted in a story on CoinDesk (where they took my analysis and re-told it in a form even an e-mail administrator can understand), an interview on The Bitcoin Game podcast over at Let's Talk Bitcoin, and I even got a name-drop on GigaOm.  As always your interest and feedback is appreciated.

BitStamp Theft From Old Wallet Addresses is Still Ongoing

Now that it's been over two weeks since the hack you would think that people have received the memo to not use their old BitStamp deposit addresses or at least to not put large sums of bitcoin in them.  You would be wrong.

Last Friday there was a 700 BTC deposit that was caught up in the heist.  Check out the block numbers and transaction time.  It was stolen in the block after it was confirmed in and it was also stolen in 3 minutes flat from the confirmation of the previous transaction.  Also, this wasn't an account that has been stolen from before, this the first transaction to the theft wallet for this address.

One obvious question is "was this a customer?"  I consider them a Bitstamp customer because they have shared transactions with other affected accounts before.  One such transaction is 311f9f which shared inputs with 18dsZT and 1BPezx, both of which were "doners" to the 1J2Ls theft address.

They have been at this address with Bitstamp for over a year now and have made several sizable deposits before.

This transaction is actually relevant for reasons other than its loss.  On my podcast I said if I had to make a Vegas bet about what I thought happened I would have gone for some form of server modification in the transaction signing program.  I think I would lose money if I ever make that bet since this seems to be conclusive evidence that private keys were lost in the hack, or keys that run a deterministic wallet were leaked and the thief knows what to do with them (which would explain the change address quirk with cold storage).  I was hedging my bets because I didn't think such a leakage was necessary for this theft, but now I really see little alternative to that conclusion.  Since this transaction occurred over a week since the hack occurred it is unimaginable that the old servers were still running and processing transactions, and the speed of the movement combined with the high fee indicates this isn't a normal transaction.  This will likely lead to some very uncomfortable discussions between the customer, Bitstamp, and whoever the customer is accountable to.

Stolen Coins Continue to be Spent

The last time I looked at the outputs I saw only three locations where you could confirm they were being mixed with other coins.  Now, the spending is going in earnest all over the place.  I considered doing a comprehensive analysis of the particular places they were mixed, but that would take way too many words to write about.

As of block number 339205, which cleared on or about Friday 16 Jan 2015 at 5:33 GMT, a rounded total of 19,940 BTC has been deposited in the 1L2Js theft address.  About 95% of those stolen coins have been spent out of that address.  For the most part the 5% of the coins remaining represent new balances arriving after the theft went public.  For transactions and outputs that are one step away from the theft address a rounded total of 19,274 BTC is either sitting in unspent transaction outputs (60%, or about 11,495) or has been spent in a transaction (40%, or about 7,752).  The sum on the first step is greater than the original because some of these transactions involve coins that were not involved in the theft.  The second step shows a similar unspent/spent ratio (54%/46%) but a significantly higher rate of mixing with coins outside the theft.

One interesting outbound transaction from the theft involves 1 BTC that has been sent to the Sarutobi iOS game. This is a game that rewarded players with 100 bit donations (just over 2 cents when I was writing this post) if they played the game well.  It took a long and winding road, but after 21 transactions Sarutori starts splitting it up into it's hot wallet address 3MXxfN.

All of the transactions not on the top line are 100% derived from the stolen coins, all the way to the very bottom row.  And it gets even more insane when you go down some of those chains I didn't expand after Saurtobi split it up into quarters. For these four peel chains, until it hits the users wallet for those transactions, the bitcoins have a 100% taint from the theft address.  And those peel chains down the 3MXxfN paths are insanely long, some of the longest peel chains on the blockchain.  And those have all been formed in the last three weeks.

Conclusion

Unless something else interesting happens with the Bitstamp theft coins, I don't see myself returning to report on their propagation across the blockchain.  There is just too many threads forming from the coins that have been spent and they are for the most part unconnected to each other.  I find it unlikely at this point that any of the coins will be returned "intact" to Bitstamp (sorry).  The thief has proven that he can unload 40% of them inside of two weeks, so the other 60% may just be bag holders who may or may not be aware of the true origin of their balance.

If you know of any interesting transactions on the blockchain that may benefit from a visual analysis, feel free to drop me a line at danno.ferrin@gmail.com or tweet me at @Numisight and I'll take a look at it.  I cannot guarantee blog coverage if I don't find any entertaining findings.  I am also open to paid investigations or paid consulting relating to blockchain analysis, and I can be as public or confidential as you desire.  For these inquiries please send email to danno@numisight.com.

BitStamp Theft Bitcoins Being Spent

The BitStamp Theft coins are more than on the move, they are being spent or being prepared for spending. The controller of the 1L2Js address has a problem, since the vast majority of the bitcoins that were stolen have been placed into a single address. Anyone who looks at addresses would clearly be able to see that those coins were stolen. And any regulated exchange is supposed to engage in these pattern matching practices, so if they want Dollars or Euros then they need to do some gymnastics.

Up until block number 338060 the presumed theft address 1L2JsXHPMYuAa9ugvHGLwkdstCPUDemNCf has kept it's output coins in a closed system. By closed system I mean that if you trace all input coins repeatedly every coin would pass through a 1L2Js address eventually.  The first address to break the pattern is 15wsXq5uSe2aT5BssLvQehUAQVn525RH25.  First we need to be careful of false positives, to make sure we are not just reporting someone tagging an involved address with 666 bits.  So here is the "heratige" of the 15wsX coins:

The very top transaction, 8328a2 contains 13 inputs from the 1L2Js address associated with the theft.  The gathered amounts are then peeled off until we have two 1BTC amounts in the 15wsX address.

Quick sidebar: note that even though the entire source of the BTC can be traced back to the hack, we cannot presume the controller of 15wsX is the same as the controller of the theft coins.  It is very easy for each of those peel chain steps to be the deposit to another party willing to buy hot bitcoins. There were 5 places in this chain where it could have happened, so it is a possibility that cannot be dismissed.

This part of the chain is where the external linkage happens.  For clarity I will only show the 930ae0 transaction, but the b52316 is functionally identical.

This transaction is either a simple coinjoin or it is a transaction deliberately structured to look like a coinjoin.  On the left we have one BTC in and one BTC out.  On the right we have some loose change in and 100 bits less out.  If you look at the transaction (here it is on blocktrail) you will see that 100 bits was the transaction fee paid.  Both sides of these transactions could very easily be their own transaction, so we cannot presume that the transaction is a single party transaction. This little bit of loose change washing continues to wash other amounts for quite some time (all the yellow transactions follow this pattern).

The exact same pattern is seen for the other transaction as well.  My conclusion is that the loose change address is intended to "wash" the bitcoins by spreading the taint of their source around.  However since the wash amount is less than the output amount then at least some of the BTC in the 1HRv8 address had to have come from the previous address, removing the plausible deniability.  This isn't the best washing job I've seen.

This may be a transaction only designed to test the waters for identification, but I think it is safe to say the thief intends to sell or spend the proceeds of the theft, if they haven't done so already.

Bitstamp Theft Change Addresses and Late Transactions

The coins in the stolen address are on the move, but at the moment (block 337938) they are in a closed system with no outside coins.  But there are some other interesting transactions to examine first.

When looking at the fact that BitStamp was hacked it is natural to immediately has how were they hacked.  With the information available I don't think we can give a definitive answer, and the evidence is a little unclear at the moment.  Assuming this was an outside actor there are two simple explanations: the private keys were stolen/leaked/compromised or the outside actor was inducing the BitStamp systems to pay him out.  However there are transactions that bring those two simple conclusions into doubt.

First, if this was simple key theft then why was the 1L2Js address generating change addresses that BitStamp was able to sweep into 1Jokt (that is presumed to be BitStamp cold storage).  Here's one address:

The b68738 transaction sent over 32 BTC to the theft address and created a change address 1BtGH with over 641 mBTC.  Less than an our later at transaction d1d835 a transaction placed exactly 10 BTC into the current BitStamp cold storage address.  The change address also peeled off into more cold storage transactions.  It is unlikely the thief was being nice and putting the coins back in the vault.  This happened two more times:

Transactions 2378aa and a8c199 both created separate change addresses and sent to the same theft address (which has since been swept up into another address).  However both of the change addresses were sept into BitStamp cold storage (and those change addresses also peeled into more cold storage).  Bit that's not the strangest part.  The 19C5D and 1BtGH addresses were not single use addresses.

The 19c5D address was an input to a theft transaction and there are still just over 10 mBTC sitting in an unspent output to 1BtGH.  The 1KPeo address is at the moment a single use address.  

Why on earth would the thief create change addresses that BitStamp could use to sweep the change into cold storage?  This is evidence in favor of someone inducing the BitStamp systems to pay out to a theft address.  But you would think that BitStamp would shut their old system down after a couple of days, and we wouldn't see any more large movements into the wallet except taggers and dusters. Except nearly 4BTC is not a dusting and tagging amount:

Over $1,000USD is a bit high for an address tag, and a bit high for misdirection. This coin was also generated after the thefts occurred, so it can't be a stray transaction that got lost in the network.  My best guess is that this was a deposit account for someone that didn't get the memo to stop depositing.  If this was a deposit address their last deposit was before the hack:

Then their deposit was used to tumble out some payment to another account prior to the attack as well. (BitStamp is reported to have it's own mixer/tumbler available for customer use.  I haven't used BitStamp and obviously can't open an account to verify at the moment.)

Better evidence of it being a long standing deposit can be seen with BlockTrail's summary of the address.  It is very spiky and goes to zero a lot, so it's not a long term holding address but a transactional one.

So the simple explanations are out the window.  The two leading explanations in my mind are that the theft stole the keys and the software and stood up their own instance of the hot wallet to do the theft, or that the compromised services at BitStamp are still up and running.  Either one of these could have been done by an inside agent or an outside agent.  Odds are BitStamp won't say much until the relevant law enforcement agencies has had their turn to examine the evidence.

Bitstamp Hot Wallet Theft - 2 to 5 Jan 2015

From 4 Jan to 6 Jan 2015 Bitstamp experienced a loss of nearly 19,000 Bitcoins from it's operational hot wallet (CoinDesk has a nice writeup about the issue).  A reddit thread identified what it believed to be the destination address for the stolen coins: 1L2JsXHPMYuAa9ugvHGLwkdstCPUDemNCf.  Evidence on the blockchain is consistent with this allegation.

First, this address was not seen prior to 4 Jan 2015, and within 24 hours it had amassed nearly a 18,000 BTC balance.

A graph of the transactions that involve the alleged address shows a lot of interaction between other addresses.

Older transactions tend to be to the right of this graph, and they form peel chains that in some cases are combined multiple times into one transaction.  It is interesting to note that some of the transactions form exact sums of the collected coins.  To the left of the graph we start to see some of the "dusters" putting dust amounts into the 1L2J wallet.

One of the concerns on the reddit thread was that the cold storage may be leaking.  If the address found in another comment is to be believed, then their cold storage is not leaking.

If anything coins are being moved into cold storage based on the uptick on 4 Jan 2015, so there is no evidence of a cold storage leak.

So how could this happen?  (Warning, baseless speculation follows).  There are two ways this could be done, first the private keys of the input addresses could have been leaked.  This would be consistent with their request to stop deposits.  The other possibility is that the attackers somehow are inducing their software to send all the bitcoins to an address of their choosing.

What would indicate private key compromise is continued activity and continued theft.  While we see continued activity on 6 January 2015 it appears to be of the "dust tagging" variety.  Consider this peel chain:

100 bits are peeled off four times from the same source address.  This is not consistent with the earlier transactions where the change addresses were single use interior addresses.

Second, there is evidence of deposit addresses not being cleared out after the bulk of the movements occured. Consider the address 18unRBGev1pkTo35zqCtCscSWUg4r9RNrh that looks to be a P2Pool payout address.

There were five deposits that were stolen durring the hack, but 2 addresses appear to be untouched on the 6th of January.  If the hacker had the private keys (along with bitstamp) then there would be a race to cash in those deposits.  If bitstamp was worried about a private key theft surely they would aggressively sweep it within the hour, instead of waiting nearly 8.

So why didn't BitStamp simply pull the plug the moment they were sure they were hacked?  Maybe they did and this was just the remaining transactions propagating through the system.  Or perhaps they were attempting to sweep what they could to their cold storage.  There was over 6,000 BTC of movement into cold storage near the tail end of the hack, representing $1.5 Million of value saved.

It could have been worse.

This analysis was performed when the blockchain was at height 337832, so any transactions after that block are not reflected in this post.