Post-Trade Analysis of Yesterday’s AP Hack!

And How Gatekeeper Fared…
One concern that I’ve noticed amongst options traders is their reluctance to auto-quote. Auto-quoting is the usage of an options model to send buy and sell orders in terms of volatility rather than outright market prices. Doing this is useful for traders who want to buy and sell at specified volatility points across an entire option chain, and they often times delta hedge directly into the underlying whenever their auto-quoting orders are filled.

The fear from auto-quoting comes from slow or latent systems getting hit on multiple strikes whenever a “book sweep,” or rapid repricing of the underlying, occurs. Auto-quoting computer systems need to do up to four Black-Scholes (or insert your favorite model) calculations for every single strike that they’re auto-quoting in, and then get change orders back out on the wire to the matching engine. If you’re quoting in 10 different underlyings, each with 40 live strikes under fast market conditions, you could be having to do up to 1600 partial differential equations per repricing. This in combination with messaging overhead, automated order logic, and user-interfaces can bring even the most powerful trading computers to their knees in fast market conditions! In other words, auto-quoting systems have historically tended to fail right when they’re needed the most!

Market conditions like yesterday’s False Tweet coming from Associated Press can be incredibly expensive in the first place, and a failure or freeze of an auto-quoting system could be catastrophic. The following is a recount of my experiences using SD Gatekeeper during the crash.

First a bit of background: I’ve been at a client site for the past week testing the next generation of tools we’re building. I sit in a small office right next to the traders that are using it and I spend most of my time babysitting algorithms and communicating with the developers about any bugs or performance issues with the software. In the meantime, the traders are using Gatekeeper to make 2 sided markets. Due to the proprietary nature of my customers’ trade and my duty to protect their trade secrets, I will not be disclosing what markets they were participating in however I can share the following pieces of data:

  • At the time of the crash, they were making 2 sided markets in 6 different markets, 5 of which were severely impacted by the crash.
  • All of the auto-quoting was taking place on a client machine several milliseconds away from the CME.

Around 10:00 AM PST, we observed a massive sell off in the E-Mini S&P. Though the exact events that took place from there are both a bit hazy to me due to the activity and strictly confidential, the important piece of information is that several minutes had passed between when the crash began to take place and when we finally pulled our orders. This sounds like all of the makings of disaster, but not so.

Blue Horseshoe Loves Gatekeeper

During the entire period of heightened market activity, Gatekeeper was successful in moving its orders out of the way and NOT ONE SINGLE ORDER was filled at a bad price. While there was a brief lockup of the User Interface, the auto-quoting functionality performed flawlessly even during the lockup.

For the tech geeks who want to understand how this is possible, I’m going to go into my analysis of the log files and subsequent stress testing and code profiling that my developers did. Casual readers my want to stop at this point, and I don’t blame you.

For the tech people:

Gatekeeper is an adapter-centric front end that uses a handful of assemblies. Namely, Gatekeeper, GKAPI, Strategies, and its adapters. All of the adapters inherit from our adapter base class to make it easy for us (and third-party developers) to add exchange connectivity and additional execution venues. All prices, orders, and business logic are then normalized into our concept of instruments, orders, prices and such in the GKAPI. Gatekeeper is simply a dumb UI implementation of the GKAPI. The strategies; Formula (autoquoting), Bookstackers, and Spread Bandit are all separate assemblies that use the GKAPI.

From a threading standpoint, we have a thread for each adapter, one for strategies, one for logging, and one for the UI. There other background workers doing various other tasks but for the purpose of this article, these are the relevant ones. The only one that became overwhelmed was the UI, and as we later found out, some of that wasn’t our fault.

In the form used yesterday, there were three supported adapters active; XTAPI, CTS, and Risk Server, however only the CTS Adapter was in use for auto-quoting purposes at the time of the crash.

For load-balancing purposes, CTS places limits on how often you can add or change orders to 50 orders per second but places no limitations on how many orders you can cancel. To prevent us from having our orders rejected, our adapter keeps track of how many changes and adds we’ve sent in a given second and cancels any order that “wants to change” after we’ve begun to approach our limit. We then add new orders at the requested change prices at a metered rate, as the adapter allows so that the user basically sees a changed order when in reality they were canceling, waiting, and replacing.

Under market conditions like yesterday, this essentially means that we were only slightly participating in the market (no more than 50 of our thousands of orders were active in the market at any given time).

Had we been using a DMA solution with co-location, we would have been fast enough to keep up anyways because our formula engine, FAST decoders and FIX engine have all been designed to accommodate for rapid repricing and mass-quoting. We take advantage of things like kernel bypass, scheduler optimizations, and iLink session spinning to bypass the limitations of 3rd party API’s.