Critical Points

This post is part of a 3-subseries in which we design the high-level architecture for Cryptomate.

1. Modules and components.
2. Exploring patterns.
3. Critical points (this post).

In this post, we detail critical parts of the architecture: relationships between main components, strategy life-cycle and integration examples for both testing and production.

Component relations

Market – strategy

In the modules post, we determined that:

• Multiple market adapters could coexist.
• The exact set of active adapters depended on running strategies.

In order to encapsulate the complexity of dealing with multiple market adapters, we will break it down into several components:

• First, we rename the market adapter to market feed adapter, to make its purpose clearer.
• We introduce a market feed factory to keep track of loaded market adapter modules. It can be queried with a feed description, and instantiates a corresponding market adapter.
• We introduce a market engine component that will act as a façade to the whole market feed subsystem.

We can now refine the relationship between the strategy and market modules, as a port/adapter-relationship using a simplified interface that the Market Engine implements. This clean separation will make it easy to replace the component with a proxy when the market module is deployed as a micro-service.

Other noteworthy points:

• I split the market history adapter in two related but separable adapters, because there are used by different components. With this split, the strategy evaluator module no longer depends on history writing functionality. It also makes it possible to feed data into the strategy evaluator module without implementing write access, which will come in handy for testing.
• The market engine manages the life-cycle of adapters according to subscriptions it receives from other modules, starting and stopping market feed adapters as needed.
• From market engine's point of view, history adapter is a single component, which may provide either or both of reading and writing features.

Trading – strategy

As for the market adapters, multiple trading adapters can coexist. Unlike market adapters however, multiple uses of a trading adapter cannot be merged, as they are likely to use different accounts on the trading platform.

• We introduce the concept of a trading account, tracking an open, in-use account. It is an abstract concept, the details of which are supplied by their owning trading adapter.
• We introduce a trading factory to keep track of loaded trading adapter modules. It can be queried with a trading platform description to instantiate the matching trading adapter module.
• We introduce a trading engine component, that will act as a façade to the whole trading subsystem.

Noteworthy points:

• The new trading engine component is the root owner of other trading-related components, and manages their whole life-cycle.
• Account components are managed by their respective adapter (probably using some kind of abstract factory pattern).

Strategy life-cycle

The strategy evaluator module is at the core of the system. It revolves around running user-supplied business logic in a controlled environment. In turn, that user-supplied business logic and its configuration determine what components should be instantiated and destroyed in other parts of the system.

In order to abstract out the layout and complexity of Cryptomate, and control strategy's environment, we introduce a strategy host module as the single interlocutor for the strategy.

Also, we identified the necessity of a strategy builder module to encapsulate the complexity of setting up a trading strategy. Here is an overview of the required steps:

• Strategy evaluator instantiates a strategy builder, passing it the strategy description.

• The strategy builder then:

  • locates the strategy evaluator plugin matching the description.
  • instantiates a strategy host.
  • delegates actual strategy construction to the strategy evaluator plugin.
  • hands back built strategy to strategy evaluator.
• The strategy evaluator takes ownership of the build strategy and its hots, and destroys the builder.

Lastly, as the unique interlocutor of a strategy, its host can track allocated components and ensure they are disposed of when the strategy is shut down.

Composition roots

In all software, components form a graph of interconnections as they are composed to build the system. Most often, this graph has a single component that stands at the root of the graph. Hence, “composition root”.

As Cryptomate is modular and flexible composition was part of our requirements, we have multiple composition graphs, with different roots.

To ensure we do not forget important use cases, we will define a few reference scenarios to check against while we implement the system.

Custom use as API

In this scenario, Cryptomate is loaded as a library by a third-party program. That is, the composition root is external. In that case, it offers a set of APIs for the third-party developer to use, namely:

• The Strategy API allows instantiating and running strategies.
• The Market API allows subscribing to real-time market data, as well as querying market history.
• The Trading API allows placing trading orders and querying basic accounting data.

In addition, all ports are exposed to the third party, so it can provide custom adapters. For adapters that are loaded automatically by Cryptomate, an adapter registry is exposed so custom adapters can be added.

Integrated service

In this scenario, Cryptomate runs as a standalone, integrated service. This is the typical setup for an end-user that runs it on his desktop, or a server dedicated to Cryptomate, for a single user.

The (external) user interface can connect and disconnect to the control endpoint to query the state and start/stop strategies.

All components run within Cryptomate, making deployment and administration easy. This means, however, that market history recording, if enabled, stops when the system is shut down.

Split market recorder

In this scenario, we split market recording, so it can keep recording while the core of the system is updated or restarted. This is a typical setup for a power user that needs consistent history to run back-tests on detailed historical data (ticks). Such a setup would normally deploy Cryptomate alongside the market service.

Since such a setup puts a special focus on recording market history, it would probably use specialized high-volume NoSQL database such as HBase™.

Full cluster deployment

In this scenario, Cryptomate is deployed on a larger scale, handling hundreds or thousands of concurrent strategies for many users. The typical use is to provide a hosted trading automation platform.

The user interface is replaced with a full-blown web application, and most adapters are connected to other infrastructure. For instance, notifications are forwarded to third party notification systems, possibly sending text messages or emails to users, or simply updating their web-page through websockets.

The key point here is horizontal scaling and redundancy at those levels:

• Several instances of the Cryptomate engine can coexist and share external services.
• Several instances of the market service can coexist, possibly fetching duplicate data to allow for overlap during an upgrade.
• Distributed strategy persistence servers allows recovering strategies in case a Cryptomate node goes down.

Note that Cryptomate itself does not cover service supervision. Still, it must be supervisable, so we will be mindful of how it can be run under standard supervision.

Conclusion

This concludes our upfront architecture series. We will do a short recap in the next post.