This section contains standalone topics. Each is indirectly important for a quant researcher.

In building trading systems, one notices patterns in how they are designing things. After you’ve built a couple, you start to see that they generally fall into certain classes based on time frequency and usage of data. Much of the infrastructure between systems feels the same, although getting the details right is always a little bit hard to do on the first attempt.

In computer science classes, one often encounters hard problems– designing efficient algorithms, writing something as concisely as possible, or solving a problem with only simple tools like bitwise operations. However there is another class of hard problems which one may unknowingly discuss in a class like “Software Engineering” but not directly encounter. Designing architectures is another category of hard problems which appear all the time in real-world software engineering but are missed by classes because it’s infeasible to actually code a large scale system in a semester and because it’s too hard for professors to grade vaguely specified projects. Design patterns are a way of coping with complexity and avoiding the tangles that can result from myopic coding. Besides just guiding the architecture designer’s thought process, we can often construct re-usable meta-frameworks and tools to speed future development.

Generally, the developer’s wisdom/learning curve follows these four steps, where you have to do each step 3-5 times to move on to the next:

prototype → full system → intuit patterns → extract frameworks

But let’s just jump ahead to the patterns.

Typically once per day or even less frequently, you run a script to download some historical data, run another script to re-optimize a model based on that updated data, and finally output signals, typically to a file to be imported into the execution software and submitted, after a quick sanity check.

Let’s rewrite that as psuedocode to make it feel more actionable–

Batch systems are the most amenable to machine learning in terms of developer friendliness, but lower frequency systems unfortunately have less data to work with. For this kind of system, data is typically stored in a matrix where each row is a day of observations of all the interesting variables on a historical date.

I bring up schedule based systems second although they’re more challenging to design than event driven systems because they are more intuitive based on human experience and more similar to TradeStation, OpenQuant, and other retail platforms which traders are probably familiar with. People are used to being driven by clocks rather than purely by unanticipatable external events and impulses.

Let’s go straight to code here, it’s clearer–

One often designs according to this pattern when automating a strategy which is currently being traded discretionarily. It might also arise in a case where you want to trade at the ‘close’, but to implement it you have to check signals and place trades 10 minutes before, for example.

Event driven systems are the most powerful and flexible because they get closest to the raw data. Power and flexibility generally lead to more tedious code – think C++ vs. Excel. ‘High Frequency Trading’, as seen on TV, is generally event-driven.

The common patterns here are generally in seamlessly indexing by time and by message number, handling stored data and indicators, and code efficiency. Execution and position tracking also become more complex when signals can change before executed trades are even confirmed.

In some sense though, these systems are the most fun to write because they don’t impose structural constraints.

GETCO, the world’s current top high frequency group, generally follows this pattern, and was the inspiration for its inclusion. Here the parts that humans are good at are delegated to humans, and the parts that a computer is good at are delegated to the computer. Humans are good at analyzing world events, understanding what time of day things will happen and what to watch out for, and developing a general feel for a certain day’s market. The computer is good at reacting on a microsecond basis and following parameters precisely and unemotionally. Basically a human is inserted as an intermediary between ‘the world’ and the automated trading system because there are a lot of common sense things a human knows about the world which turn out to be really hard to get a computer to understand.

squishy chaotic world → human → computer → market

The human turns on or off the model whenever they feel like it. There are all kinds of models (I list some possibilities here rather than previously because this is a relatively novel design pattern): A momentum model that is turned on a few minutes before earnings announcements and then turned off a few minutes after; a mean reversion algorithm that is activated on days that seem dull and don’t have any major news scheduled; a general market making model that’s run all the time except in extreme circumstances; a triple-witching-day model; etc.

Some are designed to run for minutes, some all day. Some hold for milliseconds, some for hours. Asset class and breadth are similarly unrestricted.

Each model is unique, but they also have parameters, so they are optimizable to changing conditions.

This pattern, also called “fail-fast,” is used to make a system automatically recover from random disconnection. It is mandated by many of the main exchange’s DMA protocols (FIX, NASDAQ OUCH).

This wraps the entire strategy and makes it more robust to network errors.

FSMs are used to locally track the state of objects on a remote server. This is good for handling multi-step asyncronous request-response—error—cancel logic chains. FSMs are also easy to visualize. For example, submitting a limit order to buy and working it at the best bid until it is executed or fails is a multi-step branching process. The states are: { Submitted, Canceled, Accepted, Filled, Partially Filled }. Transitions occur when the bid changes, the exchange rejects the order, an execution report arrives, or a broker status message arrives.

There is not a one-size fits all FSM implementation. Sometimes you will want all the state transition logic centralized in a switch-case statement, sometimes you will want it in each state. Sometimes states will be as simple as an enum, sometimes they will be full-fledged objects with multiple instance variables. Sometimes you will want states to share information via global variables, sometimes via arguments. FSMs which are working orders could be nested within an overall strategy FSM.

The FSM is a powerful tool for making complicated logic consistent and simpler to understand.

The API is the code library that lets your system send commands to the broker. Point and click trading GUIs that most traders place trades through are built on top of an API. Some brokers allow you to access the API. This allows third party companies to build specialized GUIs and quants to build automated trading systems. The API code library handles opening a socket connection to the broker’s server, reading and writing to the socket, constructing and parsing messages based on the brokers protocol, and passing data up to and down from higher level API calls. The API library is language specific, usually C++, Java, or Excel.

An API generally serves two concerns, data and execution.

In each part there will be functions to login, place requests, and logout, and callbacks to receive data.

We’ll compare a couple of designs closely based on actual brokers’ APIs (the names are changed, and the code is made language agnostic although strongly typed). Let’s call these brokers A, B, and C. To connect we have the following function signatures:

Broker A requires the user to have another application logged-in and running in the background. So connectA does not require a username and password. Also, it returns void since Broker A’s API is totally asynchronous. You have to listen to callbacks to determine whether you are actually connected. It also requires a unique connection id per session. There is a special case– connection id 0 is for the included GUI and receives callbacks from all connections. The purpose and recommendations for what id number to use are not clearly explained in the documentation.

connectB is better. The function to connect and login is a good candidate to make synchronous since it is not speed-sensitive, is only called once, nothing else needs to be run while you are waiting for it, and logging in does not produce a queue of messages coming over the network. It returns an int which is non-zero if the login was sucessful, just like the UNIX convention. It also has a convenient option to try to cancel all trades, reducing risk, if there is a network outage.

connectC does not exist since broker C requires you to have a lightweight GUI/console application running in the background. Although not automated, this is ok since you typically only log in once per day or less.

connectA is the worst and connectB is the best. An API is a balance between flexibility and ease of use for customers’ typical use case. The common use case is having a fully automated system connecting once per day. C is not fully automated, A ignores the fact that you only connect once per day.

Now let’s look at requesting data:

Broker A fails again, this time miserably. As consolation, they were historically one of the first to offer an API, and APIs are hard to change without breaking client applications. They also support more types of securities, more exchanges, more account types, and more countries so they require a lot more details to unambiguously specify a contract. However they should have a more consistent contract naming scheme, eliminate special cases and conditionally required fields, and lean more heavily on the compiler’s type-checker to enforce proper usage.

connectB is great. Since broker B only offers access to US equities, it is easy to name contracts. It returns an int to tell whether the request was acknowledged by the broker’s server. A separate function is offered to request all contracts rather than creating a huge array of all the contracts’ symbols.

Broker C requires clients to be colocated in their data center at the exchange. Since the computer must be extremely special purpose, they can make it so that as soon as it connects, it automatically reads the config file and requests the data without explicitly calling any function.

Broker A is bad, B is developer, user, and prototyping friendly, and C is the most barebones but effective. Like logging in, requesting data is a good candidate to make synchronized or automatic since it typically isn’t being called often.

Typically a trading system is designed as a set of logical rules within a data callback function, which places trades if the new data point matches the right conditions. So the data callback is very important. Let’s look at a few side-by-side:

Broker A is again the worst. They split the data across multiple overlapping callbacks. Tick size, price, and string callbacks will be triggered with each new data point. The user has to match up the size to the price. The documentation does not say whether the price or size should arrive first, so it is unclear which one you should have trigger the trading logic. It also does not say whether every single data point triggers both a size and price callback. Splitting them up seems to imply they might not always be called at the same time, but as far as I know they are. stringA is redundant with the first two. genericA is used for special data as explained above. You also have to store and use the original request id number to figure out which contract the data refers to.

B and C are both good.

Now that we have looked at logging in and data, let’s look at placing orders and receiving execution reports.

All three are close to equal for sending orders.

A is again slightly worse though. Users have to furnish order id numbers, and it is not well specified what should happen when you overflow int. It also has 4 different id’s and it’s not clear which are required, which must be unique, and which must be the same. The Order object itself has a “permanent”, “client” and “order” id. The function call specifies an order id. It’s not clear why the client id can’t be inferred from the current session settings, nor whether the permanent id matters. B and C assign the order id for you and then return it.

A again suffers from trying to pack functionality for all types of securities, exchanges, and clients into a single function call.

Ideally, all APIs could be improved to use the type checker to catch errors at compile-time, rather than runtime. Broker C made a half-hearted attempt to catch errors by having an error-status return type. However multiple of these errors could be caught at compile time instead if runtime. Debugging event-driven, asynchronous systems is challenging so compile time checks are a huge benefit.

Placing orders with an API is more complicated than it looks, since after placing it a number of things can happen. The order can be filled, partially filled, rejected by the broker or exchange, sit on the book, the network can become congested, or your program can crash. You will be tempted to prematurely optimize how you track your current position. If you are just trading one contract you might have an int currentPosition variable that is negative when you are short and positive when long. This is the wrong approach. Unless your system is extremely simple and only uses market orders, you should not do this. Keep an (orderId, OpenOrder) hashmap of all orders. When one is filled or cancelled, remove it. When one is partially filled, deduct the size. Create seperate methods to calculate your total size in each contract. This is one of the best places to use encapsulation.

Finally, let’s look at execution notifications (all return void):

Clearly the most functions are dedicated to execution reporting because there are multiple outcomes to submitting an order.

Unfortunately broker A again tried to cram all the functionality into one method, statusA. It’s not entirely obvious from the signature, but the status field has the possible value “Submitted”, which supersedes openA; “Filled”, which supersedes part of executionA; “Cancelled” and “Inactive”. This is very confusing for beginning developers because they wonder why 4 methods do overlapping things. So the beginner thinks that they are using them wrong because no company would release such a poor design. Unfortunately, broker A did.

Again unfortunate is that A does not give any details about why an order might be cancelled, unlike brokers B and C do.

Broker A also does not provide detailed information about execution costs, liquidity rebates, and other fees.

Broker B has a little bit of overlapping functionality in the position changeB callback since one could reconstruct their position based on a series of execution reports. But many trading systems do not work limit orders, and simplicity is preferred to flexibility. The simple position changeB function, with just two arguments, satisfies this need without causing confusion.

B and C are again similar and both well-designed.

In summary we can see broker A’s design flaws and B and C’s successes. You can see how they accomodate users with difference levels of sophistication, running on different platforms, trading different contracts. Broker B was slightly easier to use with a synchronized login and simple position change callback. C assumed users would be colocated and had more development skill.

We have ignored many other functionality including requesting account information, other types of data, and session error handling, but the successes and failures are very similar across all parts of each brokers API. I should also mention that broker A’s documentation is far worse than B and C’s, making their flawed API even more frustrating.

“What programming language is the best for trading systems?” is a fairly common question on web forums. It’s hard to answer because it’s sort of like the question, “do you know you’re ugly?” – “yes” and “no” are implied as the only possible answers, but you don’t want to say either since then you’re admitting you’re ugly. The implied answer set is all inapplicable. In this case, if you have to ask, then you probably have no idea what you’re doing. You need to go through a formal computer science degree program at a college to have any hope of designing a trading system from scratch.

Domain specific languages such as Tradestation’s EasyLanguage, Algodeal’s Market Runner, MetaTrader’s MQL5, and NinjaTrader’s Ninjascript are the best option for people with no programming experience. For simple, technical-indicator strategies that trade single assets independently and don’t use any regression and aren’t adaptive, a DSL is definitely the way to go even for experienced programmers. They typically handle all the details of managing data, writing a backtester, calculating indicators, running things at the right time of day, and so on.

One step up from this are Excel-based systems. Many brokers have an Excel API. Excel is easy to debug and tweak since all the numbers are always editable and visible. It is also the most sophisticated programmable environment that non-programmers are comfortable using. Computer scientists who are bothered by this should take care not to think of everything as a nail just because you are skilled in the use of hammers. Many good proprietary trading shops use Excel-based models for slower strategies such as fixed income trading. For strategies that are simple but don’t fit in the framework of the available DSLs or strategies which benefit from human supervision and intervention, Excel is a good choice.

Moving beyond Excel, typically the choice of which langauge to use is most constrained by the APIs your broker offers. In practice all mainstream languages are pretty much equivalent (Java, C++, .NET, maybe Python or a LISP). Most skilled programmers are indifferent between languages (except hipster programmers who program just for its own sake). By skilled programmer I mean you are either majoring in computer science and have a good GPA or you work as a developer. If this is not you, be honest with yourself and don’t waste time in this section because the complexity of a trading system will probably end up being too much. However if you are a programmer, the best language to pick is the native one for your broker’s API.

The final step up is a developer working at a trading firm. You can use any language since your firm has the resources to develop to an exchange’s protocol rather than using an API. Most high frequency firms use C++ or Java.

In summary:

Unfortunately this section has to be negative like the previous one. Choice of operating system and hardware platform seems to be the most tempting place for premature optimization. It’s to the point that people won’t even create a single trading system because they are so busy debating the benefits of Linux and cloud hosting. Just go write version 1 and run it every day for a full week on whatever desktop or laptop you have if this describes you. So much has been written about the topic of trader psychology, I think someone should write about quant trader psychology.

Use the operating system that your broker’s API targets – if they have Visual C++ project files or a .NET API then use Windows and if they use Linux system calls, use Linux. Don’t spend time trying to port it because then you will have to port every new version as well. The operating system imposes almost no new constraints on what you can do. There are high frequency trading firms using Windows.

There are three options for hardware platform: home computer, cloud host, or colocation.

As I write this in Mar 2011, cloud hosting is getting to be about as easy to run a trading system on as a computer at home. It has the advantages of 24/7 uptime, low costs, typically excellent network connections, choice of the operating system, and complete dedication to trading rather than shared with other programs. However if you have a good internet connection at home and an extra computer, then either one is fine.

Colocation has two benefits and a lot of costs. The benefits are low latency, which can make or break certain types of strategies, and access to tick data. Most data providers won’t deliver high-bandwidth tick data over the internet. If you have a strategy idea that, say, requires counting the number of trades that were buyer initiated vs seller initiated over the past minute, you will need tick data. The cheapest colocation provider I’m aware of (Lime Brokerage) costs 400 dollars per month per server and you need to meet high but not unreasonable trading volume minimums. Another cost is the hassle and experience required to buy a server, mail it to the colocation facility, log in remotely, etc. Colocation facilities do have on-site staff so you can call and have them make hardware changes remotely.

There are two options within colocation – placing trades through a broker and connecting directly to an exchange. The latter option costs tens of thousands of dollars per month or more and requires registering with the SEC as a broker-dealer.

Here are rough latency numbers and a summary: