Nanex Logo
07/25/2011 - CQS Was Saturated and Delayed on May 6th, 2010

Back to Main Research Page

The Consolidated Quotation System (CQS) is the Security Information Processor (SIP) for stocks listed on NYSE, AMEX, and NYSE Arca (home of many ETFs). CQS was widely believed to have been operating normally and within capacity on May 6, 2010. We have found strong evidence that this was not the case, and in fact, CQS was severely saturated and therefore delayed during the flash crash.

The reason CQS was perceived to be operating within capacity is because message traffic is measured in 5 second intervals which smooths (averages) message traffic over that interval. However, quote cancellation and update rates at that time were already below 1 millisecond (ms). In effect, the metric measuring capacity was over 5,000 times the speed of trading. It would be like measuring the speed of a car by taking the distance it covered over the course of 1 hour rather than instantaneous measurements as taken by speed radar. To understand the absurdity of this difference, imagine the following dialog between a Policeman and a Driver.

Officer: I clocked you speeding over 90mph.
Driver: But, I was going 45mph for 30 minutes before that. I assure you I'm averaging 60.

When we plot CQS traffic at 2 ms intervals, the story of what really happened, becomes clear. Compare the following 2 charts. The first shows message rates at 1 second intervals, the second chart shows message rates over 2 ms intervals. Capacity at the time of the flash crash was listed at 250,000/second and is shown by a solid horizontal line at that level. Notice how in the second chart, the peak hugs the maximum capacity line: traffic was saturated at those times.

CQS message rates over 1 second intervals. (click for high resolution chart)

CQS message rates over 2 ms intervals. (click for high resolution chart)

The next 2 charts show CQS traffic rates on each of the 12 multicast lines. The first shows message rates averaging over 1 second, the second chart shows message rates over 2 ms intervals.  Each multicast line at the time of the flash crash had a capacity of 30,000/second which is shown by a solid horizontal line at that level.

CQS message rates over 1 second intervals. (click for high resolution chart)

CQS message rates over 2 ms intervals. (click for high resolution chart)

The next chart shows a closeup of the critical 14:42:44 time when CQS set a new record for message rates. The total is divided by 10 so that it fits in the same scale. Notice how the total (black line) practically pegs the 250,000 rate (25,000), and the multicast lines swing wildly. This is CQS trying to mitigate traffic overflow -- by queuing data so that it does not exceed capacity. The fact that the total rate never significantly exceeds its capacity guidelines on a 2 ms basis, tells us that CQS must be throttling traffic.

The following 2 chart shows just how severe saturation became for multicast line #1. Notice how it oscillates wildly. The 2nd chart is a closeup of this behavior. Notice how the rate surges at the exact beginning of each new second. We have confirmed this behavior appears from all reporting exchanges; in other words, it's not from inbound data feeds to CQS. It's not related to the delay of the NYSE feed into CQS.

What all these charts are showing is that CQS itself was overwhelmed and a source of significant delays on May 6, 2010.

So what does the SEC's October 2010 report say about these significant CQS saturation and delays? It doesn't appear they noticed them. There is only acknowledgment of the delay in NYSE's system feeding into CQS which we previously discovered (and reported on extensively). The report says nothing about CQS capacity itself being overwhelmed. How could they have missed this?

Even people in the industry did not think there was a capacity issue with CQS. This post from a well respected member on a Motley Fool Message Board sums up nicely how others in the industry viewed CQS performance on that day:
... SIAC (the people who manage the CTS/CQS feeds) had no capacity problems even at the peak volume on May 6th. Trust me, these systems are designed to handle millions of messages per second. ...
So why is this important? Because it continues to happen, and is happening again. We think many other systems experienced saturation and delays on May 6th, and it is only a matter of time before someone takes the time to analyze the data and find them.


Publication Date: 07/25/2011

This report and all material shown on this website is published by Nanex, LLC and may not be reproduced, disseminated, or distributed, in part or in whole, by any means, outside of the recipient's organization without express written authorization from Nanex. It is a violation of federal copyright law to reproduce all or part of this publication or its contents by any means. This material does not constitute a solicitation for the purchase or sale of any securities or investments. The opinions expressed herein are based on publicly available information and are considered reliable. However, Nanex makes NO WARRANTIES OR REPRESENTATIONS OF ANY SORT with respect to this report. Any person using this material does so solely at their own risk and Nanex and/or its employees shall be under no liability whatsoever in any respect thereof.