Revisiting the Two Phase Commit (2PC)

I am working on a “Trade Book” type algorithm that needs to be very robust, yet work within the constraints of commodity hardware. Let’s see what we can do with these specs:

  • Memory: Somewhere there is an optimal setting for how much of the order book is in memory. Should it all be there? What about orders way outside the best bid/offer?
  • Fault tolerance: With commodity hardware and hosted systems, an unexpected failure is a real possibility. While it doesn’t necessarily need to be automatic, there should be something that assures the book can be recreated to its point before the crash (i.e. ACID)
  • Fast and Efficient: This should be built with testing in mind. While we want to avoid premature optimization, the tooling should be integrated but get out of the way once the code rolls into production (#DEFINEs?)
  • STL vs Boost vs ??: Algorithms around the order book should allow for replacement. While this will slow things down a bit during testing, it should be written in such a way to be easily cleaned once the best (that can be found) algorithm is chosen.
  • Threading: The order book should be single threaded (yep, that’s right!). Queueing will be handled outside the algorithm. Wow, a bullet item that will hopefully make things easier!

These are tall orders to be sure. The first orders of business will be around selecting the tooling, testing frameworks, and gathering some metrics.

It’s been ages since I messed with (or even thought about) 2PC logic. Nostalgia and cobwebs.

Considerations for Automated Trading Systems

On the surface, building an automated trading system seems like a great idea. Who wouldn’t want a machine that you turn on and generate profits?

Such systems do exist. And often the actual buy/sell/hold algorithm is not very complicated. But before you buy or build your personal ATM, there is much work to be done.

Your Strategy

You have what you believe to be a winning strategy. You have back tested and even forward tested it. Now you want to put it into production. Congratulations, you have done more work than many. But there is still work to be done on your strategy.

The simple fact is that you, your strategy, and your broker perform differently when real money is involved. Here are some things that you should consider beyond your strategy.


Who will be responsible for placing the orders? Do you plan to use your own PC, or is the strategy hosted by someone else? Perhaps your broker hosts the strategy. Who is culpable when things go awry? What is the procedure when such events inevitably happen?


Did you consider the costs of your strategy? Are you assuming that a “market order” is going to give you something between the bid and ask? This is not always the case.

Does your strategy consider rebates or costs savings by switching venues?

Order Issues

Even if the network is alive and your order is accepted, that doesn’t mean things continue to go smoothly. Did you plan for what to do with partial fills? What should be done with orders canceled by your broker, or by the exchange itself?

Exit Issues

Your order is filled. Now what?

Normally, you let your system continue to function. Perhaps the stop loss and take profit orders are in place waiting for one of your limits. Will they be there if the network goes down? What happens if the exchange has one of their famous glitches?

And then there is your emotions. What if the trade goes beyond what you feel comfortable with? What if new information comes to light, and you decide to close the position? Your system must respond accordingly.

Buying Power Issues

Brokers have risk systems that protect them from you. And often, their “paper trading” or test system will not include this. Will your system cause you to bump into their risk rules? Additionally, shouldn’t your system include code to measure risk?

The fruits of our non-labor

We have just moved to a rented home near David, Panama. Our home has a yard with a variety of trees, many of which are fruit-bearing. An interesting one is “Mamon Chino”. In Asian countries it is called Rambutan.

You can learn more about this fruit at:

This picture is of fruit we just picked a few minutes ago. Mamon Chino is the red spiny-looking fruit on the left. The pit in the middle is covered in a gummy coating that is eaten.

Also on the plate is oranges and limes.MamonChino

Multithreaded Design Patterns – Freezable

The previous post talked about the Immutable pattern. This one is a slight variant.

A Freezable object can be adjusted until it is frozen. Once frozen, it behaves like an Immutable object. Such a pattern is handy for setting up objects that should not be modified once a certain point is reached, but that point cannot occur during initialization (i.e. within the constructor).

Some of the same headaches of the Immutable pattern exist with this pattern as well. I mention it because it works around one problem of the Immutable pattern.

While care needs to be taken before freezing, locking is not necessary afterward, as the data will not (or at least should not) be changing.

Multithreaded Design Patterns – Immutable

I wanted to create a series of posts on multithreaded design patterns. I will attempt to link them together so you can flip between them.

Click here for the Freezable pattern.

The Immutable Pattern

This is one of the easiest to implement. This simply prevents changes to the object once the object has been created. Methods and properties of this object are read-only, and no method calls can modify internal data.

If changes to values are required, a new object must be made.

For lightweight, uncomplicated objects, this may work well. This also works for objects that are created once and are needed to be read from many times.

The only risk here is multiple threads attempting to create a “singleton” that should be used by multiple threads. Think of multiple threads attempting to instantiate a “lazy loaded” configuration file. Both threads see it does not yet exist, both attempt to create it, and both then attempt to make it global (setting it to the same global variable).

Believe it or not, until recent changes, there was no portable way to fix this. The original problem is discussed >here<.

The answer relies on C++ 11 features. A discussion of options can be found >here<.

Comprehensible Code

The largest percentage (by a large margin) of time spent by a software developer is maintaining existing code. A significant (although smaller) percentage will be in code that the same person recently wrote.

As the code base matures, there will be sections of code that the developer does not see for a long enough period of time that details will be forgotten. How quickly will he get up-to-speed on what that code does? More importantly, what is the best way to modify that code to match the new requirements?

This is where a strong programmer pays dividends. While many still do not believe this, clean code, documentation, and tests that exercise the code pay for themselves. Often, by the time a small section of code is complete, the extra time spent doing it right has already been paid back.

Hacking code together works. Placing code where it does not belong will get the task complete. All developers know this. And when you’re only doing a small contract, you’ll be gone before anyone notices the stench floating out of the code base.

Mature developers do not do such things. Not that they refuse to do things quickly. They simply know where to put things, how to make it clean, and make tests to satisfy themselves that the code works the way they want it to. The cost: $0

Developers: Learn to do things the right way. Your desire to do so will be repaid. Strive for quality code that is comprehensible (easy to read) and predictable (easily tested).

Clients: What you see as the final product may work, but modifications may prove costly. Spend a little time up-front thinking of how to find the right person for the job.

Why aren’t more people computer programmers?

Yesterday brought to my attention an interesting article. Well written and I agree with many of the conclusions of the author.

  1. Programming is hard to do correctly. It takes effort to get something done right.
  2. Programming is hard to learn correctly. It takes the right environment, the right student, and the right teacher.
  3. Schools do not teach how to program correctly. They only teach theory. Theory is important, but paid programmers (often) do not rely on theory to complete their work.
  4. Programming is easy to do incorrectly. Plenty of people are trained and continue to work in this industry. Sweat shops continue to crank out code and projects that do not completely work and are impossible to fix.
  5. Sweat shops continue to survive because they are making money. They will re-brand themselves when their poor reputation becomes too wide-spread. NOTE: Some sweat shops are not small and not obscure. If you have been in any part of IT for more than a year, I bet you know at least one.

Here’s the link…


Getting an image size in C++

I was in need of some quick code to get the dimensions of a JPEG. I happened upon this post that inspired me to write my own. I’ll post the results on GitHub (now posted here) when I get a chance, but some fun things came out of such research.

  1. comparing a char to its value in hex requires casting the hex value to char.
  2. Bit manipulation on the low order bit may be required. I will be doing more research on the whys.

Continue reading Getting an image size in C++

Developing Multi Platform Code

My current project includes writing for two different mobile operating systems. I had the luxury of enough time to explore the ins and outs of developing a mobile application that runs on multiple platforms.

At the start, I decided to write an example application for each platform. It was something complicated enough to warrant planning efforts.

I wrote the user interface with Cordova (PhoneGap). But had I not been able to use it, the user interface would have had to be coded for each OS individually. This is a big reason for keeping your business logic separate from the UI.

The real business logic was written in Objective C for iPhone, and Java for Android. It was a true port. I figured out ways to get the same thing done on different operating systems.

Such a project demonstrated to me that much time can be spent developing the business logic. So if you are going to support multiple platforms, you will want to do it in a platform-neutral language.

Should you write it in Javascript? The answer in my case was no. Many of the things the software needed to do did not fit within the abilities of Javascript. Cordova plugins could have been built, but that was (in my mind) too big of a chunk to bite off. I am not as comfortable with Javascript, especially in the debugging and toolset areas.

Cordova plugins were still built. But rather than doing a small task natively, and the rest in Javascript, these plugins handled a large portion of the “engine” that ran the application.

Such pieces needed access to the file system, XML parsing, and file compression.

Now with the project working on both devices, I can see how the trouble of talking to C/C++ routines should be weighed with the trouble of rewriting Objective C and Java. There is not one answer. This should be decided on a per-project basis.

For this project, writing the business logic in C++ would have been best. Once the link is set between the preferred language and C++, more complicated business logic can be done in one language, and easily ported to both platforms. Ongoing maintenance will be made easier as platform agnostic changes will be done in one place.

Yes, it is a pain to get the links set up for the Android NDK. It is a bit easier for iOS. Windows phone may be on the horizon as well.

Even with C/C++, you are forced to think about platform specific and platform agnostic code. The problem does not go away completely. But even with small projects such as this one, a “toolbox” starts to form. Some of the platform-specific code gets pushed down and hidden in some libraries that could be used on other projects.

In all, it was a great learning experience, and one with assets that I will carry to future projects.

Resource: Click here for a good article to read about writing for multiple platforms.