Algorithmic trading

Overview

Algorithmic trading is a method of executing orders using automated pre-programmed trading instructions accounting for variables such as time, price, and volume. This type of trading attempts to leverage the speed and computational resources of computers relative to human traders. In the twenty-first century, algorithmic trading has been gaining traction with both retail and institutional traders. A study in 2019 showed that around 92% of trading in the Forex market was performed by trading algorithms rather than humans. It is widely used by investment banks, pension funds, mutual funds, and hedge funds, which may need to spread out the execution of large orders or carry out trades too quickly for human traders to react. However, it is also available to private traders using simple retail tools. Algorithmic trading is widely used in equities, futures, crypto, and foreign exchange markets. The term algorithmic trading is often used synonymously with automated trading system. These encompass a variety of trading strategies, some of which are based on formulas and results from mathematical finance, and often rely on specialized software. Examples of strategies used in algorithmic trading include systematic trading, market making, inter-market spreading, arbitrage, or pure speculation, such as trend following. Many fall into the category of high-frequency trading (HFT), which is characterized by high turnover and high order-to-trade ratios. HFT strategies utilize computers that make elaborate decisions to initiate orders based on information that is received electronically, before human traders are capable of processing the information they observe. As a result, in February 2013, the Commodity Futures Trading Commission (CFTC) formed a special working group that included academics and industry experts to advise the CFTC on how best to define HFT. Algorithmic trading and HFT have resulted in a dramatic change of the market microstructure and in the complexity and uncertainty of the market macrodynamic, particularly in the way liquidity is provided.

Machine Learning Integration

Before machine learning, the early stage of algorithmic trading consisted of pre-programmed rules designed to respond to that market's specific condition. Traders and developers coded instructions based on technical indicators - such as relative strength index, moving averages - to automate long or short orders. There was a significant pivotal shift in algorithmic trading as machine learning was adopted. Specifically deep reinforcement learning (DRL) which allows systems to dynamically adapt to its current market conditions. Unlike previous models, DRL uses simulations to train algorithms, enabling them to learn and optimize its algorithm iteratively. A 2022 study by Ansari et al., showed that DRL framework "learns adaptive policies by balancing risks and reward, excelling in volatile conditions where static systems falter". This self-adapting capability allows algorithms to market shifts, offering a significant edge over traditional algorithmic trading. Complementing DRL, directional change (DC) algorithms represent another advancement on core market events rather than fixed time intervals. A 2023 study by Adegboye, Kampouridis, and Otero explains that "DC algorithms detect subtle trend transitions, improving trade timing and profitability in turbulent markets". DC algorithms detect subtle trend transitions such as uptrend, reversals, improving trade timing and profitability in volatile markets. This approach specifically captures the natural flow of market movement from higher high to lows. In practice, the DC algorithm works by defining two trends: upwards or downwards, which are triggered when a price moves beyond a certain threshold followed by a confirmation period(overshoot). This algorithm structure allows traders to pinpoint the stabilization of trends with higher accuracy. DC aligns trades with volatile, unstable market rhythms. By aligning trades with basic market rhythms, DC enhances precision, especially in volatile markets where traditional algorithms tend to misjudge their momentum due to fixed-interval data.

Ethical Implications and Fairness

The technical advancement of algorithmic trading comes with profound ethical challenges concerning fairness and market equity. The key concern is the unequal access to this technology. High-frequency trading, one of the leading forms of algorithmic trading, reliant on ultra-fast networks, co-located servers and live data feeds which is only available to large institutions such as hedge funds, investment banks and other financial institutions. This access creates a gap amongst the participants in the market, where retail traders are unable to match the speed and the precision of these systems. Aside from the inequality this system brings, another issue revolves around the potential of market manipulation. These algorithms can execute trades such as placing and cancelling orders rapidly to mislead other participants. An event to demonstrate such effects is the 2010 flash crash. This crash had occurred due to algorithmic activity before partially recovering. Executing at such high speeds beyond human oversight and thinking, these systems blur the lines of accountability. When these crashes occur, it is unclear who bears the responsibility: the developers, institutes using them or the regulators. With these systems in place, it can increase market volatility, often leaving retail traders vulnerable to sudden price swings where they lack the certain tools to navigate. Some argue this concentrates wealth among a handful of powerful firms, potentially widening the economic gaps. An example would be individuals or firms with the necessary resources gain profits by executing rapid trades sidelining smaller traders. On the contrary, it has its own benefits as well which are claimed to boost market liquidity and cut transaction costs. This creates an ethical tug of war: does the pursuit of an efficient market outweigh the risk of entrenching inequality? European Union efforts to address these concerns lead to regulatory action. These rules mandate rigorous testing of algorithmic trading and require firms to report significant disruptions..This approach aims to minimize the manipulation and enhance oversight, but enforcement is a challenge. As time goes on, algorithmic trading evolves, whereas the ethical stakes grow higher.

Early developments

Computerization of the order flow in financial markets began in the early 1970s, when the New York Stock Exchange introduced the "designated order turnaround" system (DOT). SuperDOT was introduced in 1984 as an upgraded version of DOT. Both systems allowed for the routing of orders electronically to the proper trading post. The "opening automated reporting system" (OARS) aided the specialist in determining the market clearing opening price (SOR; Smart Order Routing). With the rise of fully electronic markets came the introduction of program trading, which is defined by the New York Stock Exchange as an order to buy or sell 15 or more stocks valued at over US$1 million total. In practice, program trades were pre-programmed to automatically enter or exit trades based on various factors. In the 1980s, program trading became widely used in trading between the S&P 500 equity and futures markets in a strategy known as index arbitrage. At about the same time, portfolio insurance was designed to create a synthetic put option on a stock portfolio by dynamically trading stock index futures according to a computer model based on the Black–Scholes option pricing model. Both strategies, often simply lumped together as "program trading", were blamed by many people (for example by the Brady report) for exacerbating or even starting the 1987 stock market crash. Yet the impact of computer driven trading on stock market crashes is unclear and widely discussed in the academic community.

Refinement and growth

The financial landscape was changed again with the emergence of electronic communication networks (ECNs) in the 1990s, which allowed for trading of stock and currencies outside of traditional exchanges. In the U.S., decimalization changed the minimum tick size from 1/16 of a dollar (US$0.0625) to US$0.01 per share in 2001, and may have encouraged algorithmic trading as it changed the market microstructure by permitting smaller differences between the bid and offer prices, decreasing the market-makers' trading advantage, thus increasing market liquidity. This increased market liquidity led to institutional traders splitting up orders according to computer algorithms so they could execute orders at a better average price. These average price benchmarks are measured and calculated by computers by applying the time-weighted average price or more usually by the volume-weighted average price.

A further encouragement for the adoption of algorithmic trading in the financial markets came in 2001 when a team of IBM researchers published a paper at the International Joint Conference on Artificial Intelligence where they showed that in experimental laboratory versions of the electronic auctions used in the financial markets, two algorithmic strategies (IBM's own MGD, and Hewlett-Packard's ZIP) could consistently out-perform human traders. MGD was a modified version of the "GD" algorithm invented by Steven Gjerstad & John Dickhaut in 1996/7; the ZIP algorithm had been invented at HP by Dave Cliff (professor) in 1996. In their paper, the IBM team wrote that the financial impact of their results showing MGD and ZIP outperforming human traders "...might be measured in billions of dollars annually"; the IBM paper generated international media coverage. In 2005, the Regulation National Market System was put in place by the SEC to strengthen the equity market. This changed the way firms traded with rules such as the Trade Through Rule, which mandates that market orders must be posted and executed electronically at the best available price, thus preventing brokerages from profiting from the price differences when matching buy and sell orders. As more electronic markets opened, other algorithmic trading strategies were introduced. These strategies are more easily implemented by computers, as they can react rapidly to price changes and observe several markets simultaneously. Many broker-dealers offered algorithmic trading strategies to their clients – differentiating them by behavior, options and branding. Examples include Chameleon (developed by BNP Paribas), Stealth (developed by the Deutsche Bank), Sniper and Guerilla (developed by Credit Suisse). These implementations adopted practices from the investing approaches of arbitrage, statistical arbitrage, trend following, and mean reversion. In modern global financial markets, algorithmic trading plays a crucial role in achieving financial objectives. For nearly 30 years, traders, investment banks, investment funds, and other financial entities have utilized algorithms to refine and implement trading strategies. The use of algorithms in financial markets has grown substantially since the mid-1990s, although the exact contribution to daily trading volumes remains imprecise. Technological advancements and algorithmic trading have facilitated increased transaction volumes, reduced costs, improved portfolio performance, and enhanced transparency in financial markets. According to the Foreign Exchange Activity in April 2019 report, foreign exchange markets had a daily turnover of US$6.6 trillion, a significant increase from US$5.1 trillion in 2016.

Case studies

Profitability projections by the TABB Group, a financial services industry research firm, for the US equities HFT industry were US$1.3 billion before expenses for 2014, significantly down on the maximum of US$21 billion that the 300 securities firms and hedge funds that then specialized in this type of trading took in profits in 2008, which the authors had then called "relatively small" and "surprisingly modest" when compared to the market's overall trading volume. In March 2014, Virtu Financial, a high-frequency trading firm, reported that during five years the firm as a whole was profitable on 1,277 out of 1,278 trading days, losing money just one day, demonstrating the benefits of trading millions of times, across a diverse set of instruments every trading day.

A third of all European Union and United States stock trades in 2006 were driven by automatic programs, or algorithms. As of 2009, studies suggested HFT firms accounted for 60–73% of all US equity trading volume, with that number falling to approximately 50% in 2012. In 2006, at the London Stock Exchange, over 40% of all orders were entered by algorithmic traders, with 60% predicted for 2007. American markets and European markets generally have a higher proportion of algorithmic trades than other markets, and estimates for 2008 range as high as an 80% proportion in some markets. Foreign exchange markets also have active algorithmic trading, measured at about 80% of orders in 2016 (up from about 25% of orders in 2006). Futures markets are considered fairly easy to integrate into algorithmic trading, with about 40% of options trading done via trading algorithms in 2016. Bond markets are moving toward more access to algorithmic traders. Algorithmic trading and HFT have been the subject of much public debate since the U.S. Securities and Exchange Commission and the Commodity Futures Trading Commission said in reports that an algorithmic trade entered by a mutual fund company triggered a wave of selling that led to the 2010 Flash Crash. The same reports found HFT strategies may have contributed to subsequent volatility by rapidly pulling liquidity from the market. As a result of these events, the Dow Jones Industrial Average suffered its second largest intraday point swing ever to that date, though prices quickly recovered. (See List of largest daily changes in the Dow Jones Industrial Average.) A July 2011 report by the International Organization of Securities Commissions (IOSCO), an international body of securities regulators, concluded that while "algorithms and HFT technology have been used by market participants to manage their trading and risk, their usage was also clearly a contributing factor in the flash crash event of May 6, 2010." However, other researchers have reached a different conclusion. One 2010 study found that HFT did not significantly alter trading inventory during the Flash Crash. Some algorithmic trading ahead of index fund rebalancing transfers profits from investors.

Trading ahead of index fund rebalancing

Most retirement savings, such as private pension funds or 401(k) and individual retirement accounts in the US, are invested in mutual funds, the most popular of which are index funds which must periodically "rebalance" or adjust their portfolio to match the new prices and market capitalization of the underlying securities in the stock or other index that they track. Profits are transferred from passive index investors to active investors, some of whom are algorithmic traders specifically exploiting the index rebalance effect. The magnitude of these losses incurred by passive investors has been estimated at 21–28bp per year for the S&P 500 and 38–77bp per year for the Russell 2000. John Montgomery of Bridgeway Capital Management says that the resulting "poor investor returns" from trading ahead of mutual funds is "the elephant in the room" that "shockingly, people are not talking about".

Pairs trading

Pairs trading or pair trading is a long-short, ideally market-neutral strategy enabling traders to profit from transient discrepancies in relative value of close substitutes. Unlike in the case of classic arbitrage, in case of pairs trading, the law of one price cannot guarantee convergence of prices. This is especially true when the strategy is applied to individual stocks – these imperfect substitutes can in fact diverge indefinitely. In theory, the long-short nature of the strategy should make it work regardless of the stock market direction. In practice, execution risk, persistent and large divergences, as well as a decline in volatility can make this strategy unprofitable for long periods of time (e.g. 2004-2007). It belongs to wider categories of statistical arbitrage, convergence trading, and relative value strategies.

Delta-neutral strategies

In finance, delta-neutral describes a portfolio of related financial securities, in which the portfolio value remains unchanged due to small changes in the value of the underlying security. Such a portfolio typically contains options and their corresponding underlying securities such that positive and negative delta components offset, resulting in the portfolio's value being relatively insensitive to changes in the value of the underlying security.

Arbitrage

In economics and finance, arbitrage is the practice of taking advantage of a price difference between two or more markets: striking a combination of matching deals that capitalize upon the imbalance, the profit being the difference between the market prices. When used by academics, an arbitrage is a transaction that involves no negative cash flow at any probabilistic or temporal state and a positive cash flow in at least one state; in simple terms, it is the possibility of a risk-free profit at zero cost. Example: One of the most popular arbitrage trading opportunities is played with the S&P futures and the S&P 500 stocks. During most trading days, these two will develop disparity in the pricing between the two of them. This happens when the price of the stocks which are mostly traded on the NYSE and NASDAQ markets either get ahead or behind the S&P Futures which are traded in the CME market.

Conditions for arbitrage

Arbitrage is possible when one of three conditions is met:

The same asset does not trade at the same price on all markets (the "law of one price" is temporarily violated). Two assets with identical cash flows do not trade at the same price. An asset with a known price in the future does not today trade at its future price discounted at the risk-free interest rate (or, the asset does not have negligible costs of storage; as such, for example, this condition holds for grain but not for securities). Arbitrage is not simply the act of buying a product in one market and selling it in another for a higher price at some later time. The long and short transactions should ideally occur simultaneously to minimize the exposure to market risk, or the risk that prices may change on one market before both transactions are complete. In practical terms, this is generally only possible with securities and financial products which can be traded electronically, and even then, when first leg(s) of the trade is executed, the prices in the other legs may have worsened, locking in a guaranteed loss. Missing one of the legs of the trade (and subsequently having to open it at a worse price) is called 'execution risk' or more specifically 'leg-in and leg-out risk'. In the simplest example, any good sold in one market should sell for the same price in another. Traders may, for example, find that the price of wheat is lower in agricultural regions than in cities, purchase the good, and transport it to another region to sell at a higher price. This type of price arbitrage is the most common, but this simple example ignores the cost of transport, storage, risk, and other factors. "True" arbitrage requires that there be no market risk involved. Where securities are traded on more than one exchange, arbitrage occurs by simultaneously buying in one and selling on the other. Such simultaneous execution, if perfect substitutes are involved, minimizes capital requirements, but in practice never creates a "self-financing" (free) position, as many sources incorrectly assume following the theory. As long as there is some difference in the market value and riskiness of the two legs, capital would have to be put up in order to carry the long-short arbitrage position.

Mean reversion

Mean reversion is a mathematical methodology sometimes used for stock investing, but it can be applied to other processes. In general terms the idea is that both a stock's high and low prices are temporary, and that a stock's price tends to have an average price over time. An example of a mean-reverting process is the Ornstein-Uhlenbeck stochastic equation. Mean reversion involves first identifying the trading range for a stock, and then computing the average price using analytical techniques as it relates to assets, earnings, etc. When the current market price is less than the average price, the stock is considered attractive for purchase, with the expectation that the price will rise. When the current market price is above the average price, the market price is expected to fall. In other words, deviations from the average price are expected to revert to the average. The standard deviation of the most recent prices (e.g., the last 20) is often used as a buy or sell indicator. Stock reporting services (such as Yahoo! Finance, MS Investor, Morningstar, etc.), commonly offer moving averages for periods such as 50 and 100 days. While reporting services provide the averages, identifying the high and low prices for the study period is still necessary.