Reinforcement Learning Trading

The three parts, plainly

• The agent is the decision rule. At each step it looks at the state (some numbers describing the market and its own position) and chooses an action.
• The environment is everything the agent acts on. In trading it is usually a replay of past price data plus a simple model of fills and costs.
• The reward is a number the agent receives after acting. It is the only thing the agent is trying to increase.

Training repeats a loop: observe the state, act, receive the reward, adjust the decision rule so that actions followed by high reward become more likely. After many passes through the data the agent has a policy, which is a map from state to action. Some versions use a table of values, others a neural network. The idea is the same.

Why trading is a hard fit

• Little data. Games let an agent play millions of times. A price history is one path. Ten years of daily bars is about 2,500 steps, and nearly all of it is noise.
• No push back. In a replay your orders do not move the price, so the agent never learns about its own market impact or about partial fills.
• The rules change. The environment that produced the data is not the one that will produce tomorrow's.
• Delayed, noisy reward. A trade's result is mostly luck, so credit goes to actions that were not the cause.

Why reward design breaks it

The agent maximises the reward you wrote, not the result you meant. Raw profit ignores risk, so an agent can learn a policy that earns steadily and then takes one huge loss that the training data never contained. A reward with no trading cost teaches it to trade on every wiggle. A reward on closed trades teaches it to hold losers open. Adding penalties (for costs, for drawdown, for size) helps, but each penalty is a number you chose, and the agent will find the gap in whichever one you forgot. See drawdown and slippage.

A worked example: a toy agent on random prices

Everything here is made up. We generated two price series of 300 bars each, series A and series B, both random with no pattern. The state is whether each of the last two moves was up or down, so there are 4 states. In each state the policy chooses short, flat or long, which gives 3 x 3 x 3 x 3 = 81 possible policies. "Training" is the simplest possible form: score all 81 on series A and keep the best. The cost is 0.05 points for each unit of position change.

We did it twice, to show reward design. In the first the reward ignores cost. In the second the reward subtracts cost. Then both winners were played, with cost, on the new series B.

Calculated by us on 29 Oct 2026, exact for these seeds. The first agent made 89 position changes in training, because its reward never counted cost. The second made 1, so once trading was charged for, it settled into holding a position. Both look good on A, which is the best of 81 tries on one random path. On B, where there is nothing to find, they scored -3.96 and -6.22. Across all 81 policies the average on B was -10.52 and only 3 finished above zero. Over the whole of series A the price rose 3.75 points and over series B it fell 6.06 (calculated by us, exact), so a policy that learned to hold a long position on A met a falling path on B. The training score said nothing about it. That is a toy with 81 policies. A neural network agent has far more room to memorise.

How it hits bots

• Sold as "AI that learns the market". Ask what the reward was, what data it trained on, how many versions were tried and what it earned on data it never saw.
• Grid and martingale behaviour. An agent rewarded on profit per step can learn to add to losers, because most steps in the training data recovered. That is a martingale in disguise, with the same tail.
• Constant retraining. Each retrain on recent data can fit a new noise pattern, so past live results say little about the next version.

How to check an agent before any money

• Train on early data and test once on a later period it never saw. See overfitting.
• Include spread, fees and slippage in the environment and in the reward.
• Run it on random prices. If it earns there, the environment has a leak. See look-ahead bias.
• Check the worst fall and the largest single loss, not only the total reward.
• Compare with a plain fixed rule. If the agent cannot beat it after costs, it adds complexity and nothing else.
• Forward test and paper trade before any real size.

What this page does not tell you

It does not say reinforcement learning cannot work in some settings. It says that in trading the data is thin, the reward is easy to get wrong and the agent is free to overfit. The toy numbers are made up and nothing here is a method to trade.