Archive for April, 2013

Post-Trade Analysis of Yesterday’s AP Hack!

And How Gatekeeper Fared…
One concern that I’ve noticed amongst options traders is their reluctance to auto-quote. Auto-quoting is the usage of an options model to send buy and sell orders in terms of volatility rather than outright market prices. Doing this is useful for traders who want to buy and sell at specified volatility points across an entire option chain, and they often times delta hedge directly into the underlying whenever their auto-quoting orders are filled.

The fear from auto-quoting comes from slow or latent systems getting hit on multiple strikes whenever a “book sweep,” or rapid repricing of the underlying, occurs. Auto-quoting computer systems need to do up to four Black-Scholes (or insert your favorite model) calculations for every single strike that they’re auto-quoting in, and then get change orders back out on the wire to the matching engine. If you’re quoting in 10 different underlyings, each with 40 live strikes under fast market conditions, you could be having to do up to 1600 partial differential equations per repricing. This in combination with messaging overhead, automated order logic, and user-interfaces can bring even the most powerful trading computers to their knees in fast market conditions! In other words, auto-quoting systems have historically tended to fail right when they’re needed the most!

Market conditions like yesterday’s False Tweet coming from Associated Press can be incredibly expensive in the first place, and a failure or freeze of an auto-quoting system could be catastrophic. The following is a recount of my experiences using SD Gatekeeper during the crash.

First a bit of background: I’ve been at a client site for the past week testing the next generation of tools we’re building. I sit in a small office right next to the traders that are using it and I spend most of my time babysitting algorithms and communicating with the developers about any bugs or performance issues with the software. In the meantime, the traders are using Gatekeeper to make 2 sided markets. Due to the proprietary nature of my customers’ trade and my duty to protect their trade secrets, I will not be disclosing what markets they were participating in however I can share the following pieces of data:

  • At the time of the crash, they were making 2 sided markets in 6 different markets, 5 of which were severely impacted by the crash.
  • All of the auto-quoting was taking place on a client machine several milliseconds away from the CME.

Around 10:00 AM PST, we observed a massive sell off in the E-Mini S&P. Though the exact events that took place from there are both a bit hazy to me due to the activity and strictly confidential, the important piece of information is that several minutes had passed between when the crash began to take place and when we finally pulled our orders. This sounds like all of the makings of disaster, but not so.

Blue Horseshoe Loves Gatekeeper

During the entire period of heightened market activity, Gatekeeper was successful in moving its orders out of the way and NOT ONE SINGLE ORDER was filled at a bad price. While there was a brief lockup of the User Interface, the auto-quoting functionality performed flawlessly even during the lockup.

For the tech geeks who want to understand how this is possible, I’m going to go into my analysis of the log files and subsequent stress testing and code profiling that my developers did. Casual readers my want to stop at this point, and I don’t blame you.

For the tech people:

Gatekeeper is an adapter-centric front end that uses a handful of assemblies. Namely, Gatekeeper, GKAPI, Strategies, and its adapters. All of the adapters inherit from our adapter base class to make it easy for us (and third-party developers) to add exchange connectivity and additional execution venues. All prices, orders, and business logic are then normalized into our concept of instruments, orders, prices and such in the GKAPI. Gatekeeper is simply a dumb UI implementation of the GKAPI. The strategies; Formula (autoquoting), Bookstackers, and Spread Bandit are all separate assemblies that use the GKAPI.

From a threading standpoint, we have a thread for each adapter, one for strategies, one for logging, and one for the UI. There other background workers doing various other tasks but for the purpose of this article, these are the relevant ones. The only one that became overwhelmed was the UI, and as we later found out, some of that wasn’t our fault.

In the form used yesterday, there were three supported adapters active; XTAPI, CTS, and Risk Server, however only the CTS Adapter was in use for auto-quoting purposes at the time of the crash.

For load-balancing purposes, CTS places limits on how often you can add or change orders to 50 orders per second but places no limitations on how many orders you can cancel. To prevent us from having our orders rejected, our adapter keeps track of how many changes and adds we’ve sent in a given second and cancels any order that “wants to change” after we’ve begun to approach our limit. We then add new orders at the requested change prices at a metered rate, as the adapter allows so that the user basically sees a changed order when in reality they were canceling, waiting, and replacing.

Under market conditions like yesterday, this essentially means that we were only slightly participating in the market (no more than 50 of our thousands of orders were active in the market at any given time).

Had we been using a DMA solution with co-location, we would have been fast enough to keep up anyways because our formula engine, FAST decoders and FIX engine have all been designed to accommodate for rapid repricing and mass-quoting. We take advantage of things like kernel bypass, scheduler optimizations, and iLink session spinning to bypass the limitations of 3rd party API’s.


Associated Press Hacked. Markets react to false tweet!

The S&P plunged within minutes early this afternoon, after somebody hacked an AP Twitter account and posted a bogus tweet saying the White House had been attacked and President Obama was injured. Bonds rallied a full handle and then bounced back just as quickly as it became obvious that the “news” was false and quickly recovered.

Update 4/23/2013 :12:50 CST:  Syrian Electronic Army have already claimed responsibility!

View an animated replay of the crash here:

Read more:

Still Using Spreadsheets to Manage Risk or Create Algos? Time to Reconsider.

I’ve been building technology for trading firms for over a decade. From proprietary desks with in-house development and security so tight that you have to check your cellphones at the door, to shared office spaces where people openly collaborate about their strategies; I’ve seen it all. I’d say that one of the most common tools that I’ve seen on trading desks is Microsoft Excel.

Excel is popular because you can combine realtime data feeds from a multitude of sources with complicated mathematical formulas to do incredibly complex things incredibly quickly from a time-to-delivery standpoint. I’ve seen excel sheets used for everything from position analytics and reporting, to even realtime execution algorithms and complex order types. As a developer, I use Excel sheets to build proofs of concept constantly. Gatekeeper’s Formula Strategy, Spread Bandit, and even the Position Manager in Gatekeeper found their genesis in spreadsheets. It’s easy to use and anyone with a basic understanding of math and logic can become a programmer, well, almost.

Like just about anything that seems too good to be true, it is. All of the things that make Excel powerful can also make it dangerous. Being able to link cells together to form incredibly complex strings of logic without any kind of compile checks or design best practice enforcement means that everyone who “knows enough to be dangerous,” is. According to a recent Marketwatch Article, 88% of excel sheets contain errors! This is not surprising considering that a research survey of senior executives conducted by Vision Critical, only one fifth of companies have control policies for spreadsheets and even in companies that have spreadsheet policies, it is not always applied one third of the time!

One thing that separates traders from programmers, however, is that programmers have bug elimination built into their trade. They use strongly typed languages, compile time checks, best practices policies built into their development methodologies, and post-development quality assurance processes before their code ever makes it into production. Software engineers have had a lifetime of hard knocks and painful experiences stemming from bugs to know that you can’t put a prototype into production, and the same should apply to spreadsheets.

So as a trader, what can you do to protect yourself from spreadsheet errors?

The first thing to do is to find places where your spreadsheets are either error-prone, rely on third party data sources, or rely on bad or changing assumptions. In those places, you need to add checks for data validity and create alerts for when it’s not.

If your spreadsheets are used to enter orders into the market and/or display data that might cause a trading decision, it’s time to contract a developer to build you a custom tool, or evaluate off-the-shelf solutions. I’ve seen position greeks be completely wrong because someone didn’t update an expiration date after a contract roll or realtime price valuations become slightly off because interest rates changed and weren’t updated (okay that hasn’t happened recently, but you get the idea).

If hiring a software company sounds too expensive, remember that it’s cheaper than trading errors. Just ask JP Morgan who lost $2 billion due to spreadsheet errors! At the very least, give this article a read about best practices, or check out the European Spreadsheet Risks Interest Groups to fully understand the risks.