Reinforcement learning the exit decision

The LBO post ends on the one decision the model leaves open: when to sell. IRR carries time in its exponent, the market multiple you exit into is a moving target, and deleveraging keeps adding value while you wait — three forces pulling the answer in different directions. That trade-off has the exact shape of an optimal stopping problem, and optimal stopping is where reinforcement learning is at its most honest: small state, two actions, a reward you cannot argue with.

This page trains the agent in your browser. Nothing is pre-computed: press Train and a Q-table is filled from zero by ten thousand simulated deals; press Reset and it is gone.

The environment

The deterministic toy LBO is made stochastic and quarterly:

• EBITDA follows lognormal growth: drift 7% a year, volatility 12% — a business that usually grows and sometimes doesn't.
• The multiple mean-reverts: each quarter it is pulled a little toward 8× and shocked, clamped to [4×, 14×]. Cycles emerge; booms decay.
• Debt amortizes by the same cash sweep as the deterministic model, one quarter at a time.
• The fund horizon forces a sale at seven years, so "never decide" is not an available policy.

Each quarter the agent observes (t, multiple, leverage) and chooses HOLD or SELL (selling is unavailable in the first year — deals are not flipped in a quarter). Selling ends the episode with one terminal reward; holding earns nothing today and everything later. That asymmetry — one terminal reward, credit assigned backwards through time — is precisely what Q-learning's bootstrap handles.

The reward needed one iteration, and the failure is the best part. The first version rewarded annualized IRR, and within thirty seconds of training the agent had found the exploit every finance person will recognize: sell after one lucky quarter and let annualization do the talking — a +11% quarter reads as a 53% "IRR". Nothing was wrong with the learner; the objective was lying. The fix is to reward log money-multiple and put time preference where it belongs, in the discount: γ = 0.975 per quarter, a hurdle of roughly 10% a year. The dashboards still display IRR, because that is the unit PE thinks in — but the agent optimizes discounted log-wealth, which has no annualization seam to pick at.

The learner

Tabular Q-learning, on purpose. The continuous state is bucketed onto a grid of 28 quarters × 10 multiple bands × 8 leverage bands, so the value function is a table you can watch being written:

Q(s,a) ← Q(s,a) + α · [ r + γ·max_a′ Q(s′,a′) − Q(s,a) ]

with γ = 0.975 per quarter, ε-greedy exploration decaying from 1.0 to 0.02, and α = 0.08. No neural network — the demo's point is that the Bellman update is the whole mechanism, and the sidebar below shows each update with its actual numbers as it happens.

Train it

What to look for as it converges:

• The heatmap grows a shoreline. High multiple + late quarter turns solid SELL; low multiple + early quarter stays HOLD. The boundary between them is the learned exit policy — sell into strength, but not before the sweep has retired the debt that makes the equity move.
• Mean exit settles around two years — and that is the finding. The trained agent harvests the multiple's first spike when one comes and holds on when it doesn't (exits range out to year five), averaging ≈2 years. In this simulator patience is a cost: mean reversion means a boom decays, so the option is worth most early.
• Evaluate beats the best fixed rule, not a strawman. The comparison chips try every "always sell at year k" policy and show the winner. State-dependence is worth roughly ten IRR points over the best calendar rule here (≈30% vs ≈19% on converged runs) — the entire gap is knowing what the multiple is doing when you decide.

Honest limitations

The agent exploits a known simulator — in reality nobody hands you the multiple's true mean-reversion speed, and a policy fitted to a misspecified process inherits the misspecification (the RL version of model risk). The state ignores everything soft that actually drives exits: management readiness, buyer appetite, portfolio context. And a table over 2,240 states works precisely because the toy is small; the moment the state grows, you trade the visible Bellman update for function approximation and a new set of failure modes. There are also no transaction costs or sale processes here — a real exit takes two quarters and a banker, which alone would push the learned policy later.

None of that is the point. The point is that "when should we sell?" — a question answered in practice by pattern-matched experience — has a formal skeleton, the skeleton is an MDP, and the cheapest algorithm in the RL toolbox finds its optimal policy in your browser tab in under a minute. The interesting conversations start where the simulator stops being true.