03-Aug-2012 ~ The Knightmare Explained
The following theory fits all available facts.
We believe Knight accidentally released the test software they used to verify that their new market making software functioned properly, into NYSE's live system.
In the safety of Knight's test laboratory, this test software (we'll call it, the Tester) sends patterns of buy and sell orders to its new Retail Liquidity Provider (RLP) Market Making software, and the resulting mock executions are recorded. This is how they could ensure their new market making software worked properly before deploying to the NYSE live system.
When the time comes to deploy the new market making software, which is likely handled by a different group, the Tester is accidentally included in the release package and started on NYSE's live system. On the morning of August 1st, the Tester is ready to do its job: test market making software. Except this time it's no longer in the lab, it's running on NYSE's live system. And it's about to test any market making software running, not just Knights. With real orders and real dollars. And it won't tell anyone about it, because that's not its function.
For stocks where Knight is the only one running market making software as a RLP, and the Tester is the only algo trading that's crossing the bid/ask spread, then we'll see consistent buy and sell patterns of trade executions, all marked regular, and all from the NYSE, and all occurring at prices just above the bid or just below the ask. Examples include EXC and NOK and you can see these patterns in charts here. The Tester is functioning just as it did in the lab, and Knight's market making software is intercepting these orders and executing them. Knight won't lose any money on these trades, but they will be generating a lot of wash sales.
For stocks where Knight is not the only market maker, or when there are other algos actively trading (and crossing the bid/ask spread), then some, or all of the orders sent by the Tester will be executed by someone other than Knight, and Knight will now have a position in the stock. Meaning it could be making or losing money. The patterns generated for these stocks will depend greatly on the activity of the other players.
Because the Tester indiscriminately buys at the ask and sells at the bid, and because the bid/ask spreads are very wide during the open, we now understand why many stocks moved violently at that time. The Tester was simply hitting the bid or offer, and the side it hit first, determined whether the stock opened sharply up or down.
Since the Tester doesn't think it's dealing with real dollars, it doesn't have to keep track of its net position. It's job is to send buy and sell orders in test pattern waves. This explains why Knight didn't know right away that it was losing a lot of money. They didn't even know the Tester was running. When they realized they had a problem, the first likely suspect would be the new market making software. We think the two periods of time when there was a sudden drop in trading (9:48 and 9:52) are when they restarted the system. Once it came back, the Tester, being part of the package, fired up too and proceeded to continue testing. Finally, just moments before an economic news release at 10am, someone found and killed the Tester.
We can fully appreciate the nightmare their team must have experienced that morning.