Following the Inside Man

Last night (19 Feb 2015) Morgan Spurlock's show "Inside Man" premiered their episode about Bitcoin on CNN.  Sure enough there were scenes of him paying for and receiving bitcoin. And since the transactions occured on the blockchain, they are available for proof.

The first bitcoins on his show is a small $0.25 gift for his brand new install of an Aegis Bitcoin Wallet he used for the whole show.

His next task was to buy a whole bitcoin at an exchange. After being inspired by Andreas Antonopoulos, he bought an entire bitcoin for $630 (this was filmed on 7 Jul 2014, so the price was good then).

Next he goes to upstate New York, but because of the magic of television editing we see him spending some of that hard bought bitcoin.  First, he buys a pizza, or two.  Thanks to the nice steady and straight on shot of the barcode we have a high level of confidence this is indeed the transaction where he bought pizza.

The blue transaction is when the pizza parlor cashes out their bitcoin (at a loss).  Next he buys some groceries.  And apparently they manage their bitcoins on the same address, because they also get cleared out in the same transaction.

Then after all that hard shopping and eating he gets a massage.  Would you believe the massage place works off of the same vendor?

Actually, I'm not terribly sure about this one.  Because in the video the transaction is stated as $20, and I cannot make out any QR codes nor is an exact BTC amount stated.  But this is the closest one I can see. (Come on CNN, what's it going to take to get Inside Man filmed in 4K?) But it makes sense if they filed on one day and had only one vendor that needed to fill out a release.  I'm sure they got a release just in case some snoopy person posts the transactions on some blog somewhere.

Next in the show we see a scene where Morgan "buys" airfare and hotel to the bitcoin mine he is about to visit.  This is a bit of editing magic to provide a nice segue to the next segment.  But the dates on the travel drop down start after the next transaction we will see.  Furthermore, the two bitcoin addresses show zero transaction (which I won't link to as I don't want to encourage dust and tags to go there).  But the point of the segment is that yes, you can book travel and hotels with bitcoin.

Now we go back in time to his visit where he worked briefly at a Bitcoin mine.  He gets paid 0.3 BTC for his labors and gets to see a snazzy animation of his transaction showing up at blockchain.info.

Next he works with the (former) FBI agent who helped bring down the original Silk Road.  Yea, we won't see him on the stand but he will do a CNN interview.  (Makes you think the prosecution had a better strategy in mind than the defense.)  Morgan then buys a "Fauxlex" from Silk Road 2.  Except I doubt he used his Aegis wallet.  You see, Morgan's wallet address seems to be 12qp1Ksm8NDxaermZpf4wbpdC2FSyupFx5, and all the outbound transactions I see coming from that address are either (a) too small or (b) go through Coinbase.  I'de like to think that the Silk Road 2.0 operators aren't so dumb as to process transactions through Coinbase, but I've been wrong before.

Now after that late night on the deep web Morgan wakes up and is a little groggy, so he needs to get some coffee.  He stumbles upon one vendor who will sell him coffee... for a $25 minimum charge.  His complaint is that he loses money on the volatility if he goes too low.  But this is really an educational opportunity because there are service providers (such as BitPay who had a banner at the Bitcoin center scene) who will immediately cash out your bitcoin to fiat at the time of your transaction.  But Morgan has a large crew so buying 10 cups of coffee is actually a reasonable course of action.

Finally, he finishes the show at the Grumpy Cafe and recruits a new vendor to accept bitcoin.  Although I don't think he has fully drank the Kool-aid on this one (being a coffee shop and all) since the payment is still sitting in the Unspent Transaction Outputs pool.

But what is interesting is what wasn't shown.  By my reckoning Morgan also bought some bitcoin from Coinbase, and there were lots of vendors who used Coinbase to clear their transactions. Most of those transactions occurred off screen, and some of them were duplicates (like me I doubt Morgan can keep from going to the grocery store every day or two).

All in all I thought it was a fairly balanced piece on Bitcoin.  It's not often you get a journalist to actually use bitcoin before reporting on it.

Ross Ulbricht Trial Day 9 - Mr. Yum's Testimony

If you are currently a juror for the Ross Ulbricht "Silk Road" trial you should close this page now, because this is precisely the kind of stuff the judge doesn't want you reading: armchair analysis of a witness on the stand.  Specifically Ilhwan Yum, who testified on Thursday which was day 9 of the trial.  I was a little underwhelmed by the testimony, but I don't think it's Mr. Yum's fault since the rules of evidence would preclude some of the more entertaining findings.  Also, the prosecution needs to tailor his testimony to a group of people who don't read /r/Bitcoin multiple times a day.

FBI Addresses

After some preliminaries where Mr Yum establishes a chain of custody for the two hard drives that were recovered and necessary legal positioning regarding MD5 and SHA1 checksums of the hard drive he confirms what has been essentially public knowledge: the coins from Ross's Laptop were swept into 1FfmbH...paPH. What wasn't explicitly stated was that the coins from the Silk Road server were swept into 1F1tAa...4xqX.  It was also made clear that the addresses held coins from the two different servers. There was some brief mention the sum of the address was higher, but the defense addressed that later.

There was some operational information in the testimony about how they claimed the silk road coins.  Apparently the sweep occured while the site was live and operational, and they even went through some steps to insure that the balances wouldn't be updated until all of the bitcoins were seized.  And they even used the bitcoin services that were already on the server.  This was important because the initial sweep off of the server took about four ours, according to the blockchain.

But what is interesting is that there appear to be two larger bumps within the next month:

Every circle on this chart represents a transaction into the 1F1tAa address.  Most of the dots on the flat part probably represent "dusters" and "taggers" as I call them, but I don't know if the big jumps are coins recovered from other servers or coins that were improperly swept up.  There's a lot of them so it's hard to tell.

Since this is a legal proceeding I think I should also be a bit more precise.  The coins were not actually on the Iceland server, what was found was the private keys to bitcoin addresses, lots of them.  Over 2 million on the Iceland server and over 11 thousand on Ross's laptop.  Clearly there weren't that many addresses holding a balance, so the sweeps only gathered those coins with balances.  These addresses were entered in as CDs and are exhibits 650 and 651, which an individual well known in bitcoin circles has said he will be obtaining.

Connecting the dots

This is the part of the testimony I found most astonishing.  The analysis was simply of one-to-one transactions.  That's not the astonishing part, the rules of evidence basically would only allow transactions to be admitted to evidence where the party conducting the transaction is known and proven.  In this case they only took effort to prove the transactions on behalf of keys held by the Iceland server to addresses whose private keys were on Ross's Laptop.  One to one, not passing go or going through a tumbler or even through any kind of a relay.  Direct from Iceland to Ross's laptop.  That was the surprising part, not the astonishing part.

The astonishing part was how many bitcoins went into Ross's Laptop.  There were 700,254 BTC worth of these one-to-one exchanges that occurred, and only 89,000 BTC received didn't come directly from silk road, 88.7% of the coins had a direct connection.  Rather than detail every transaction in excruciating detail the defense showed only one: b3561f...4984

I'm not sure why this transaction (the one in yelllow) was shown of all of the possible transactions to show.  But judging on the way legal processes can go they were likely putting something into play that may need to be referenced later.

But wait... there's more!

You didn't think they would spend all that time and effort just to get a victimless crime prosecuted?  Although they prosecution hasn't come out and made the final accusation in testimony (it's supposed to be coming Monday 2 Feb) they did have Mr. Yum enter in evidence relating to a transaction that wasn't one of the one-to-one transactions, but instead came from Ross's laptop.  Mr Yum stuck to the boring details of the transaction itself in his testimony.

He read in two transactions, and over the course of several pages makes it clear that the coins from the top three transactions were from private keys on Ross's laptop, the orange transaction came from Ross's laptop, and the coins in blue (later spent in the transactions in blue) received the coins he sent.  Something more than $489,000 USD by Coinbase's estimates (where I get my transaction rate information).  The next witness was getting to the good parts before the judge sent everyone home for the weekend.

Defense Cross Examination

The defense didn't spend nearly as much time cross examining the witness as the prosecution did directly examining him.  And there was no redirect.  But what he did do was place some seeds where a reasonable doubt could form.  Like he worked with a partner, he wrote his analysis programs in Python (actually it was Mr. Edmond who did the hands on typing, coders get left out of the spotlight again), but they chose that because Pie Wallet (which was written in Python) was already on Ross's laptop.  Those were all conversational.

The real doubt seeds come int he questions that are relating to the fact that the FBI is new to bitcoin seizures.  They initially didn't have a protocol in place to seize coins, which Mr. Yum helped establish.  Wallets can move from computer to computer, (but some other forensic data established their last access date). The government had access to the Iceland server for months before the shutdown.  And hey, maybe Silk Road was just a hot-wallet service? Some of the more blockchain related questions (and the more sensible ones) were when the defense also discussed the practice I call "dusting" and "tagging" with the witness, not grounds for reasonable doubt in my book.

He also tested Mr. Yum's knowledge of the large transactions on the blockchian.  The defense asked Mr. Yum about a large transaction on 22 Nov 2013 that summed 195,000 BTC.  He then showed him some paperwork (but the transcript doesn't have references to the evidence numbers) and claims it has three outputs.  I'de love to see what Mr. Yum was shown because I believe it was these two transactions (1746d7...8ca9 and 1c1244...d204), but the lack of precision annoys me.  It was described as

a transaction of 195,000 bitcoin that that was then quickly broken up into three smaller transactions [source]

There are two transactions close to that sum, both on the same date (22 Nov), and both are nearly 195 (194.993, with over half a BTC in the earlier one).  Also both had two outputs.  While there was three between them I think the defense was being sloppy in their questioning.  Here's the graphs (minus the 47 inbound transactions for the first tx).

And what bothers me more is that the notion of on the spot "blockchain trivia" somehow makes an expert qualified or not.  There are over 58 million transactions and you expect a witness in a trial to be able to answer about a random transaction without their proper tools and due diligence?  If I had to guess I would have gone with a Bitstamp audit (and I would have gotten lucky).  Perhaps he felt he could ask because it was from a government exhibit, but I can't tell from the transcript.

Presumption of Innocence Still Stands

Don't forget, as of this post the trial isn't over, and Ross still has the presumption of innocence.  While you may be reading the transcript and be ready to convict someone (Ross or the government) realize that the prosecution hasn't rested and the defense hasn't had a chance to present their case.  If the case seems damning, that is only because the prosecution is competent in what they do, because if they didn't get to this point in pre-trial the judge would have kept it off her docket.

But the real question is how will the defense will address their side of the case.  Spoliation of evidence?  The keys were planted?  Ross was a patsy for the real DPR (that's been teased in opening statements)?  Ross never created those transactions he just held the keys as a backup for the real DPR?  It was actually Mr. McMillan the school custodian who would have gotten away with it if it wasn't for those meddling teenagers and that talking dog?  Who knows, it's their case to make.

So please, keep the pitchforks and torches down until the jury comes back with a decision.  Then we can see which side will be mobbing and for what reason.

796 Theft of 27 Jan

Bitcoin thefts come in all shapes and sizes.  Recently 796, a Chinese bitcoin exchange, has one thousand bitcoins stolen from them via a series of social and technical attacks.  This isn't a very large theft, only representing less than CN¥ 1,500,000,000, which translates to US$250,000 or the cost of a double wide trailer in silicon valley.  But the real question to be asked in this blog is this: does the evidence on the blockchain match the narrative presented in the official explanations?

The Narrative

CoinTelegraph was able to talk with 796, and in combination with a post on weibo (hope you can read Chinese) the following narrative emerged.  Being a bitcoin exchange they are always a target for theft and abuse, and three months ago they migrated their servers to a cloud host.  However somehow a hacker was able to replace one users address with another similar address.  At this point the hacker needed to simply wait for a transaction to be posted.  And they got lucky with a large 1000BTC transaction.

This is the point of the story where one of the ugly truths of cyber defense becomes manifest.  You have to be 100% in all of your defenses and secondary systems, but the hacker only has to get lucky once or twice.  This hacker had already had some luck getting the data updated on the server.  They had another stroke of luck getting the bitcoins out.  First, the weibo post indicated there was another level of approvals needed to permit the transaction because the transaction looked suspicious because of the use of different IPs.  But the transaction met the standards of approval so the transaction was processed.

About five hours later the customer who initiated the withdraw is in contact with 796 over the phone and they indicate the bitcoin had not shown up in the customer's destination address.  At this point a full scale investigation is launched, and it is revealed that the coins went to an address that was similar to the address that was intended.  This is where the hacker got lucky: by using a similar address it passed both 796 manual review and the customers manual review of the information.  Because when it comes down to it 35 characters of randomness does get tiring on the eyes after a while and if you have to commit it to memory, you probably only memorize five characters or so.

After the details are sorted out and the investigation is concluded 796 owed up to the theft, and took the losses out of company profits.  For 796.com client losses have a senior claim to shareholder losses, so there was no vote or consultation with the investors.

The Blockchain Evidence

On the 8btc.com forum (8btc is a Chinese language bitcoin forum) one post identified the theft address as 1CvGkU...2wrU.  There was one large transaction at 15:50 GMT on 27 Jul.  The whole of China is on the +8:00 time zone, so the timing of the transaction would be 23:50, consistent with the weibo post.  It was then committed to block 340748 a nearly a minute later.  So did a suspect transaction go out?  The evidence does match the narrative. 

But there is also the second half of the narrative.  796 also said that they would take responsibility for the theft.  About 13 hours later another withdraw of 999BTC is sent to the address 1CvGkM...vJwU.    Not that the first four characters of this address matches the first five characters of the theft address. It also matches up with the claim of the insertion of a similar address.  Note however, that this is 1 BTC short of making the customer whole.  If we look at the other transactions to the 1CvGkM address we see another transaction an hour earlier.

Ignore the dates on the transactions.  These are self-reported by the miners and the protocol allows for something on the order of two hours of skew. In the block prior to the 999BTC transaction a 1 BTC transaction is sent to the correct address.  I presume that this was done to test the waters of their system to make sure that the reported address can be trusted.  The next block is when the 999BTC is sent, and three blocks after that the customer then gathers the coins and spends it in two separate transactions.

These two transactions made the customer whole and the timeframe seems reasonable enough, being the next business day during business hours in China.

Technial Details

You have probably noticed that I elide the transaction identifiers and addresses in my images and in my text.  There is one principal reason for that, and it's not to protect privacy or redact data.  The reason is that 64 hex digits in a row and 35 alphanumerics in a row are hard on the eyes.  I take the six leading characters and the four trailing characters because one sixth to one third of the data is enough to reproduce my findings.  There are many occasions where a single address or a single transaction when elided will look like another address or transaction id.  However, when you assemble the network and take the elided addresses and transaction ids in context, the elided data provides enough information to reproduce the data displayed in my images.

The cognitive fatigue of looking at all of that data played a part in this hack, so it is my hope that by slashing the amount of data shown that the user might be able to process all of what is shown, rather than a part of the whole that varies between users.  I am fairly certain that it is good enough for analysis.

In fact, if anyone can show me a transaction on the blockchain where one address combined with the input and output transactions cannot be distinguished from a similar transaction based on the elided values, I'll send you a box of girl scout cookies. Of course I'll only pay up when girl scout cookies are on sale.  If you can show me one below block 340950 then I'll even let you pick the (in-stock) flavor of cookies.  This offer is good only for the first taker however, and only in jurisdictions where they are legal (they are addictive after all).

BitStamp Theft - Two Weeks Later

Before I get into the analysis I'de like to thank all of you who have found me in the past two weeks. What used to be a sleepy and infrequent technical blog resulted in a story on CoinDesk (where they took my analysis and re-told it in a form even an e-mail administrator can understand), an interview on The Bitcoin Game podcast over at Let's Talk Bitcoin, and I even got a name-drop on GigaOm.  As always your interest and feedback is appreciated.

BitStamp Theft From Old Wallet Addresses is Still Ongoing

Now that it's been over two weeks since the hack you would think that people have received the memo to not use their old BitStamp deposit addresses or at least to not put large sums of bitcoin in them.  You would be wrong.

Last Friday there was a 700 BTC deposit that was caught up in the heist.  Check out the block numbers and transaction time.  It was stolen in the block after it was confirmed in and it was also stolen in 3 minutes flat from the confirmation of the previous transaction.  Also, this wasn't an account that has been stolen from before, this the first transaction to the theft wallet for this address.

One obvious question is "was this a customer?"  I consider them a Bitstamp customer because they have shared transactions with other affected accounts before.  One such transaction is 311f9f which shared inputs with 18dsZT and 1BPezx, both of which were "doners" to the 1J2Ls theft address.

They have been at this address with Bitstamp for over a year now and have made several sizable deposits before.

This transaction is actually relevant for reasons other than its loss.  On my podcast I said if I had to make a Vegas bet about what I thought happened I would have gone for some form of server modification in the transaction signing program.  I think I would lose money if I ever make that bet since this seems to be conclusive evidence that private keys were lost in the hack, or keys that run a deterministic wallet were leaked and the thief knows what to do with them (which would explain the change address quirk with cold storage).  I was hedging my bets because I didn't think such a leakage was necessary for this theft, but now I really see little alternative to that conclusion.  Since this transaction occurred over a week since the hack occurred it is unimaginable that the old servers were still running and processing transactions, and the speed of the movement combined with the high fee indicates this isn't a normal transaction.  This will likely lead to some very uncomfortable discussions between the customer, Bitstamp, and whoever the customer is accountable to.

Stolen Coins Continue to be Spent

The last time I looked at the outputs I saw only three locations where you could confirm they were being mixed with other coins.  Now, the spending is going in earnest all over the place.  I considered doing a comprehensive analysis of the particular places they were mixed, but that would take way too many words to write about.

As of block number 339205, which cleared on or about Friday 16 Jan 2015 at 5:33 GMT, a rounded total of 19,940 BTC has been deposited in the 1L2Js theft address.  About 95% of those stolen coins have been spent out of that address.  For the most part the 5% of the coins remaining represent new balances arriving after the theft went public.  For transactions and outputs that are one step away from the theft address a rounded total of 19,274 BTC is either sitting in unspent transaction outputs (60%, or about 11,495) or has been spent in a transaction (40%, or about 7,752).  The sum on the first step is greater than the original because some of these transactions involve coins that were not involved in the theft.  The second step shows a similar unspent/spent ratio (54%/46%) but a significantly higher rate of mixing with coins outside the theft.

One interesting outbound transaction from the theft involves 1 BTC that has been sent to the Sarutobi iOS game. This is a game that rewarded players with 100 bit donations (just over 2 cents when I was writing this post) if they played the game well.  It took a long and winding road, but after 21 transactions Sarutori starts splitting it up into it's hot wallet address 3MXxfN.

All of the transactions not on the top line are 100% derived from the stolen coins, all the way to the very bottom row.  And it gets even more insane when you go down some of those chains I didn't expand after Saurtobi split it up into quarters. For these four peel chains, until it hits the users wallet for those transactions, the bitcoins have a 100% taint from the theft address.  And those peel chains down the 3MXxfN paths are insanely long, some of the longest peel chains on the blockchain.  And those have all been formed in the last three weeks.

Conclusion

Unless something else interesting happens with the Bitstamp theft coins, I don't see myself returning to report on their propagation across the blockchain.  There is just too many threads forming from the coins that have been spent and they are for the most part unconnected to each other.  I find it unlikely at this point that any of the coins will be returned "intact" to Bitstamp (sorry).  The thief has proven that he can unload 40% of them inside of two weeks, so the other 60% may just be bag holders who may or may not be aware of the true origin of their balance.

If you know of any interesting transactions on the blockchain that may benefit from a visual analysis, feel free to drop me a line at danno.ferrin@gmail.com or tweet me at @Numisight and I'll take a look at it.  I cannot guarantee blog coverage if I don't find any entertaining findings.  I am also open to paid investigations or paid consulting relating to blockchain analysis, and I can be as public or confidential as you desire.  For these inquiries please send email to danno@numisight.com.

BitStamp Theft Bitcoins Being Spent

The BitStamp Theft coins are more than on the move, they are being spent or being prepared for spending. The controller of the 1L2Js address has a problem, since the vast majority of the bitcoins that were stolen have been placed into a single address. Anyone who looks at addresses would clearly be able to see that those coins were stolen. And any regulated exchange is supposed to engage in these pattern matching practices, so if they want Dollars or Euros then they need to do some gymnastics.

Up until block number 338060 the presumed theft address 1L2JsXHPMYuAa9ugvHGLwkdstCPUDemNCf has kept it's output coins in a closed system. By closed system I mean that if you trace all input coins repeatedly every coin would pass through a 1L2Js address eventually.  The first address to break the pattern is 15wsXq5uSe2aT5BssLvQehUAQVn525RH25.  First we need to be careful of false positives, to make sure we are not just reporting someone tagging an involved address with 666 bits.  So here is the "heratige" of the 15wsX coins:

The very top transaction, 8328a2 contains 13 inputs from the 1L2Js address associated with the theft.  The gathered amounts are then peeled off until we have two 1BTC amounts in the 15wsX address.

Quick sidebar: note that even though the entire source of the BTC can be traced back to the hack, we cannot presume the controller of 15wsX is the same as the controller of the theft coins.  It is very easy for each of those peel chain steps to be the deposit to another party willing to buy hot bitcoins. There were 5 places in this chain where it could have happened, so it is a possibility that cannot be dismissed.

This part of the chain is where the external linkage happens.  For clarity I will only show the 930ae0 transaction, but the b52316 is functionally identical.

This transaction is either a simple coinjoin or it is a transaction deliberately structured to look like a coinjoin.  On the left we have one BTC in and one BTC out.  On the right we have some loose change in and 100 bits less out.  If you look at the transaction (here it is on blocktrail) you will see that 100 bits was the transaction fee paid.  Both sides of these transactions could very easily be their own transaction, so we cannot presume that the transaction is a single party transaction. This little bit of loose change washing continues to wash other amounts for quite some time (all the yellow transactions follow this pattern).

The exact same pattern is seen for the other transaction as well.  My conclusion is that the loose change address is intended to "wash" the bitcoins by spreading the taint of their source around.  However since the wash amount is less than the output amount then at least some of the BTC in the 1HRv8 address had to have come from the previous address, removing the plausible deniability.  This isn't the best washing job I've seen.

This may be a transaction only designed to test the waters for identification, but I think it is safe to say the thief intends to sell or spend the proceeds of the theft, if they haven't done so already.

Bitstamp Theft Change Addresses and Late Transactions

The coins in the stolen address are on the move, but at the moment (block 337938) they are in a closed system with no outside coins.  But there are some other interesting transactions to examine first.

When looking at the fact that BitStamp was hacked it is natural to immediately has how were they hacked.  With the information available I don't think we can give a definitive answer, and the evidence is a little unclear at the moment.  Assuming this was an outside actor there are two simple explanations: the private keys were stolen/leaked/compromised or the outside actor was inducing the BitStamp systems to pay him out.  However there are transactions that bring those two simple conclusions into doubt.

First, if this was simple key theft then why was the 1L2Js address generating change addresses that BitStamp was able to sweep into 1Jokt (that is presumed to be BitStamp cold storage).  Here's one address:

The b68738 transaction sent over 32 BTC to the theft address and created a change address 1BtGH with over 641 mBTC.  Less than an our later at transaction d1d835 a transaction placed exactly 10 BTC into the current BitStamp cold storage address.  The change address also peeled off into more cold storage transactions.  It is unlikely the thief was being nice and putting the coins back in the vault.  This happened two more times:

Transactions 2378aa and a8c199 both created separate change addresses and sent to the same theft address (which has since been swept up into another address).  However both of the change addresses were sept into BitStamp cold storage (and those change addresses also peeled into more cold storage).  Bit that's not the strangest part.  The 19C5D and 1BtGH addresses were not single use addresses.

The 19c5D address was an input to a theft transaction and there are still just over 10 mBTC sitting in an unspent output to 1BtGH.  The 1KPeo address is at the moment a single use address.  

Why on earth would the thief create change addresses that BitStamp could use to sweep the change into cold storage?  This is evidence in favor of someone inducing the BitStamp systems to pay out to a theft address.  But you would think that BitStamp would shut their old system down after a couple of days, and we wouldn't see any more large movements into the wallet except taggers and dusters. Except nearly 4BTC is not a dusting and tagging amount:

Over $1,000USD is a bit high for an address tag, and a bit high for misdirection. This coin was also generated after the thefts occurred, so it can't be a stray transaction that got lost in the network.  My best guess is that this was a deposit account for someone that didn't get the memo to stop depositing.  If this was a deposit address their last deposit was before the hack:

Then their deposit was used to tumble out some payment to another account prior to the attack as well. (BitStamp is reported to have it's own mixer/tumbler available for customer use.  I haven't used BitStamp and obviously can't open an account to verify at the moment.)

Better evidence of it being a long standing deposit can be seen with BlockTrail's summary of the address.  It is very spiky and goes to zero a lot, so it's not a long term holding address but a transactional one.

So the simple explanations are out the window.  The two leading explanations in my mind are that the theft stole the keys and the software and stood up their own instance of the hot wallet to do the theft, or that the compromised services at BitStamp are still up and running.  Either one of these could have been done by an inside agent or an outside agent.  Odds are BitStamp won't say much until the relevant law enforcement agencies has had their turn to examine the evidence.

Bitstamp Hot Wallet Theft - 2 to 5 Jan 2015

From 4 Jan to 6 Jan 2015 Bitstamp experienced a loss of nearly 19,000 Bitcoins from it's operational hot wallet (CoinDesk has a nice writeup about the issue).  A reddit thread identified what it believed to be the destination address for the stolen coins: 1L2JsXHPMYuAa9ugvHGLwkdstCPUDemNCf.  Evidence on the blockchain is consistent with this allegation.

First, this address was not seen prior to 4 Jan 2015, and within 24 hours it had amassed nearly a 18,000 BTC balance.

A graph of the transactions that involve the alleged address shows a lot of interaction between other addresses.

Older transactions tend to be to the right of this graph, and they form peel chains that in some cases are combined multiple times into one transaction.  It is interesting to note that some of the transactions form exact sums of the collected coins.  To the left of the graph we start to see some of the "dusters" putting dust amounts into the 1L2J wallet.

One of the concerns on the reddit thread was that the cold storage may be leaking.  If the address found in another comment is to be believed, then their cold storage is not leaking.

If anything coins are being moved into cold storage based on the uptick on 4 Jan 2015, so there is no evidence of a cold storage leak.

So how could this happen?  (Warning, baseless speculation follows).  There are two ways this could be done, first the private keys of the input addresses could have been leaked.  This would be consistent with their request to stop deposits.  The other possibility is that the attackers somehow are inducing their software to send all the bitcoins to an address of their choosing.

What would indicate private key compromise is continued activity and continued theft.  While we see continued activity on 6 January 2015 it appears to be of the "dust tagging" variety.  Consider this peel chain:

100 bits are peeled off four times from the same source address.  This is not consistent with the earlier transactions where the change addresses were single use interior addresses.

Second, there is evidence of deposit addresses not being cleared out after the bulk of the movements occured. Consider the address 18unRBGev1pkTo35zqCtCscSWUg4r9RNrh that looks to be a P2Pool payout address.

There were five deposits that were stolen durring the hack, but 2 addresses appear to be untouched on the 6th of January.  If the hacker had the private keys (along with bitstamp) then there would be a race to cash in those deposits.  If bitstamp was worried about a private key theft surely they would aggressively sweep it within the hour, instead of waiting nearly 8.

So why didn't BitStamp simply pull the plug the moment they were sure they were hacked?  Maybe they did and this was just the remaining transactions propagating through the system.  Or perhaps they were attempting to sweep what they could to their cold storage.  There was over 6,000 BTC of movement into cold storage near the tail end of the hack, representing $1.5 Million of value saved.

It could have been worse.

This analysis was performed when the blockchain was at height 337832, so any transactions after that block are not reflected in this post.