One decoder. Every card’s full history. Many risk models.
Last month I rebuilt Netflix’s foundation model for recommendation. This time the same idea goes after money: train one decoder on unlabeled transaction histories, then let fraud, credit and churn models build on what it learns.
Payments companies are already doing this. Stripe, Nubank, Visa, Mastercard, Revolut and Plaid have all described large models trained on transaction data. NVIDIA published an open recipe for it, and I ported that recipe into ai-cfo with portable libraries: pandas, PyTorch, Hugging Face and XGBoost.
This post walks through how I built each piece, what the code does, and what is still left to do.
At a glance
- 13 tokens per transaction: one marker plus one token per field, instead of the ~39 a GPT-2 BPE tokenizer spends on commas and dollar signs.
- A small vocabulary: 3,171 tokens on the synthetic data, against 50,257 for GPT-2.
- A compact Llama-style decoder, pretrained with plain next-token prediction. No labels.
- One embedding per transaction, which only sees the history up to that transaction.
- A fraud test: XGBoost on hand-built features, on embeddings, and on both, all scored on the same time-ordered test set.
- Runs on a laptop CPU, a single GPU, or eight GPUs with
torchrun.
The language of money
Most fraud and credit models still run on hand-engineered features: amount, hour, merchant category, time since the last purchase. Each team builds its own set. Each set looks at one row, or a few rolling averages, and misses the order in which things happened.
But order is the signal. A $900 online purchase in another state at 3 a.m. is normal for one cardholder and alarming for another. You can only tell which by reading that person’s history.
A language model learns meaning from the order of words. A transaction foundation model learns behaviour from the order of payments. Train it once on everyone’s history, and every downstream team gets a vector that already knows what “normal” looks like for this card.
A tokenizer built for transactions
The obvious shortcut is to write each transaction as a CSV line and feed it to a text tokenizer. GPT-2’s tokenizer spends about 39 tokens per transaction that way, most of them on punctuation and digits. At a 4,096-token budget, that is roughly 100 transactions of history.
The domain tokenizer emits one token per field instead. Every transaction becomes:
[TXN] AMT_b10 MERCH_h1247 MCC_5311 HOUR_19 DOW_3 MON_1 CHIP_Online Transaction ZIP3_866 STATE_AZ ERR_0 DT_UNK CARD_0That is 13 tokens, so the same budget now holds about three times as much history. Each field gets the treatment it needs:
- Amounts go into signed log-scale bins, so a $12 coffee and a $1,200 laptop land far apart, and refunds get bins of their own.
- Merchants are hashed into 2,048 buckets. New merchants never grow the vocabulary.
- Categories such as MCC, hour, ZIP3 and state keep their most frequent values. Rare and unseen values map to that field’s own UNK token.
- Time since the previous transaction gets log bins too. Bursts of spending show up as short gaps.
Every transaction starts with a [TXN] marker, so its last token always sits at a fixed offset. That matters later, when we pool embeddings.
Each field is a small tokenizer class. Adding a new field, say a device ID or a beneficiary country, is a few lines in default_steps().
Code: tokenizer.py.
No leakage, by design
Fraud models are easy to fool yourself with. Three rules keep this pipeline honest:
- Split by time, not at random. The first 80% of transactions train the model, the next 10% validate it, and the last 10% test it. The model never sees the future.
- Fit everything on the training period. The tokenizer vocabulary and the pretraining data both come from the first 80% only.
- Never tokenize the label. The fraud flag is not a field. The foundation model learns behaviour, not answers.
Code: data.py.
Built like a language model

Under the hood, the model is a standard Hugging Face LlamaForCausalLM, sized to match NVIDIA’s recipe: hidden size 512, 8 layers, grouped-query attention with 8 query heads and 2 key-value heads, rotary positions up to 8,192 tokens, SwiGLU and RMSNorm. With the TabFormer vocabulary, that is about 29M parameters. Tiny by LLM standards, and enough for a vocabulary of a few thousand tokens.
Windows that respect transactions. Training samples random windows from each cardholder’s history. Windows always start on a transaction boundary, so the model never sees half a payment. Busy cardholders are sampled more often, in proportion to their history length.
Plain next-token prediction. No special heads and no labels. Predicting the next merchant bucket, amount bin or time gap forces the model to learn each cardholder’s routine.
A checkpoint anyone can load. Training is a plain PyTorch loop with AdamW, warmup and cosine decay, and bf16 on GPU. The result saves as standard safetensors and loads back with AutoModelForCausalLM.from_pretrained. With torchrun, data-parallel training across GPUs switches on automatically.
Code: model.py.
One vector per transaction
With causal attention, only the last position has seen everything before it. So the hidden state at the last token of transaction t is an embedding of the cardholder’s history up to and including t. Never the future, and never the label.
That gives a useful property: one forward pass over a window produces an embedding for every transaction in it. Long histories are covered with overlapping windows, so each transaction keeps at least half a window of prior context.
The 512-dimensional vectors are compressed to 64 with PCA, fitted on the training period. On the synthetic run, 64 dimensions kept 98% of the variance.
Code: embeddings.py.
Does it move fraud detection?
A foundation model is only worth it if a downstream metric moves. The test: three XGBoost classifiers, scored on the same held-out time period.
- Raw: 13 hand-engineered features. The baseline.
- Embeddings only: the 64 PCA dimensions.
- Combined: both, 77 features.
Judge by average precision (AP) first, not ROC-AUC. At around 0.1% fraud, ROC-AUC saturates near 1 and hides what happens at the top of the score list. That top is where a review team with fixed daily capacity actually works.
On IBM’s TabFormer dataset (24.4M synthetic card transactions), NVIDIA reports that the combined model lifts AP by about 42% over the raw baseline, from 0.124 to 0.176. ROC-AUC barely moves. Embeddings on their own do worse than the baseline. They complement hand-built features; they don’t replace them.
Swap the fraud label for churn or default, and the same pattern applies.
Code: downstream.py.
Fraud that only history reveals
I wanted the pipeline to run with zero setup, so it ships with a synthetic generator. The generator is built so that fraud is hard to see from a single row.
Each synthetic cardholder has a home state, favourite merchants, a typical spend and their own active hours. Some are night owls, so the hour alone proves nothing. Fraud arrives as a burst of spending in another state at new merchants. But so does ordinary travel.
What separates them is the cardholder’s own history. Fraud bursts run several times above their usual spend, come through a channel they rarely use, and land far from their normal hours. A single row is ambiguous. The sequence is not. That is exactly the case a foundation model should help with.
An agent for the CFO stack
The pipeline is also wrapped as an agent, with the same contract as every other agent in ai-cfo. It trains a small model, reports baseline and combined AP and ROC-AUC, and returns the 25 highest-risk test transactions as a review queue. An LLM then explains in four sentences what the lift means for a fraud team with fixed daily capacity.
Code: agent.py, pipeline.py.
Try it
The whole pipeline runs on a CPU with synthetic data. It generates card histories, tokenizes them, pretrains, extracts embeddings and runs the fraud comparison.
git clone https://github.com/avinashbarnwal/ai-cfo.git
cd ai-cfo
python3 -m venv .venv && .venv/bin/pip install -r transaction_foundation_model/requirements.txt
.venv/bin/python -m transaction_foundation_model.pipeline --synthetic --preset small --max-steps 300For the full setup, download TabFormer’s card_transaction.v1.csv and run on a GPU:
python -m transaction_foundation_model.pipeline --data data/card_transaction.v1.csv \
--preset paper --max-steps 3000 --seq-len 4096 --batch-size 16Set TFM_CUDF=1 to run the pandas steps on the GPU with cuDF, and use torchrun --nproc-per-node 8 to train across eight GPUs.
With the smoke-test settings:
- 201,624 synthetic transactions from 400 cardholders, 0.55% of them fraud
- 1.2M parameters, with 78-transaction windows
- pretraining loss fell from 8.09, the random-guess level, to 3.31 in 300 steps
- about 19 minutes end to end on a 2-core cloud CPU
A note on results: on this small synthetic run, the combined model lifted ROC-AUC from 0.959 to 0.972, but AP stayed flat (0.288 against 0.284). Embeddings alone scored far lower. A 1.2M-parameter model trained for 300 CPU steps hasn’t learned much yet. These numbers show the pipeline works end to end, not that it catches more fraud. The real test is the full-size model on TabFormer.
And I’m not done
This is version one. Next up:
- The full run: the 29M-parameter preset on all 24.4M TabFormer transactions, to see whether I reproduce NVIDIA’s AP lift.
- Beyond fraud: churn, credit default and cash-flow anomalies from the same embeddings.
- Business payments: bank feeds and AP/AR flows instead of consumer card swipes, which is where a CFO stack lives.
- Scaling experiments: does AP keep improving as the model and the data grow?
If you work on fraud or credit models, I’d love to hear what you would add.
References
- NVIDIA Technical Blog, Build Your Own Transaction Foundation Model for Financial Intelligence, June 2026. The recipe this project follows.
- NVIDIA, Build Your Own Transaction Foundation Model developer example.
- Code: github.com/avinashbarnwal/ai-cfo, transaction_foundation_model/ folder.