Two lineages of feature computation
Feature denotes two distinct computational objects, and the distinction determines the architecture around it. The Feature Store implements the second.
Entity-keyed features: the dual-store architecture
In the ML-platform lineage, a feature is an attribute of an entity – a user's order count over the last 30 days, a listing's cancellation rate. The entity carries an id, and at inference the model requests the vector bound to that id.
Uber described this shape in 2017 with Michelangelo, whose feature store – Palette – is a database of curated, internally crowd-sourced features any project can reuse. Airbnb worked the same problem for seven years and open-sourced Chronon in 2024. Feast, Tecton and Hopsworks sell variations of it.
The architecture follows from the access pattern: an offline store for training, a low-latency key-value store for serving, materialisation pipelines keeping the second consistent with the first, and a registry holding the definitions. Time enters only as a window specification; the query axis is the entity.
Time-keyed features: expression engines
In the quant lineage, a feature is not an attribute of an entity but a function of market data over time, and the query axis is time itself. There is no entity lookup; the query is the value of a quantity as of a given instant.
This lineage has its own public record. WorldQuant published 101 Formulaic Alphas in 2015 – explicit formulas that are simultaneously code, which established much of the operator vocabulary the field still uses. Microsoft's Qlib ships an expression engine: you write something like Ref($close, 1), the engine parses it into a syntax tree, and results of the tree's nodes are cached so repeated work isn't repeated.
That pattern – a DSL parsed to an AST, mapped onto operators and executed as a computation graph – is well established in quantitative finance. It shows up in WorldQuant's alpha compiler, in DolphinDB's stream engine parser, and in the dependency graph behind Zipline's Pipeline API. Our FQL, which stands for Feature Query Language, sits squarely in this lineage, and the AST-plus-hash machinery described below will look familiar to anyone who has used those systems.
The naming is therefore slightly misleading: BHFT's Feature Store is not a materialised store of entity-keyed vectors. It's an expression engine with a compilation and transport layer on top.
Two access patterns. On the left the query carries an entity id; on the right it carries an instant.
Constraints neither architecture addresses
Two constraints fall outside both designs, and both are load-bearing here.
Reuse without visibility
In both lineages, reuse is achieved by making definitions visible. The ML platform publishes them to a central registry; the quant literature publishes the formula. That is the mechanism – sharing happens because the definition is visible.
In a proprietary trading firm, the definition is the product. A researcher who publishes their implementation somewhere colleagues can read it has published the thing that makes them valuable, and no amount of goodwill makes that a comfortable trade. The result, before the Feature Store, was predictable: good work stayed local, got rebuilt by the next person, and disappeared when its author moved on.
Two researchers both need a normalised spread. One normalises against a rolling median, the other against a session average. Both choices are defensible, neither knows the other is working on it, and any model that later combines their outputs quietly inherits two different definitions of the same idea.
What we required was reuse without visibility, and no published system provides it. Every design available to learn from assumes the organisation is willing to expose the definition to its own users.
Arrival time as a first-class quantity
Both lineages treat time as a window parameter, on the assumption that whatever carries an earlier timestamp was available earlier. On daily bars that assumption is harmless, at nanosecond resolution it's false: the timestamp a venue writes and the moment the data reaches us are two different quantities. The second one isn't recorded anywhere in the data.
That's a requirement on the language, not on the infrastructure. A feature has to be able to say how long its inputs took to arrive, otherwise every computation silently assumes zero. So the delay is declared alongside the data, and evaluation happens against the moment a value became available rather than against the venue clock.
Point-in-time correctness therefore denotes something different here. In the ML-platform lineage it guards against label leakage during training-set assembly. For us it's a question about physical availability: what had actually reached us at the instant the feature was computed.
Why caching came second
One inversion of priorities is worth noting. In Qlib the expression engine and the cache are close to inseparable – parsing to a tree exists largely so that node results can be persisted and reused.
What the Feature Store offers instead
Before it existed, every researcher at BHFT worked the same way. You pulled the datasets you needed out of the data lake, wrote your own code to compute whatever you were after, and trained on the result. Everyone did this independently, and everyone did it from the beginning. Research still got done. What it didn't do was accumulate.
Now a researcher publishes a feature into the Feature Store. Other researchers can then use it in their own work and their own models – and the author doesn't have to worry about their code being taken, because the code isn't accessible in the first place.
What you get when you depend on someone's feature is its interface: the name, the config type, the type of the output, the declared latency. The formula behind it stays with the person who wrote it. You depend on the output, never on the source.
Pushing features into the Feature Store carries bonuses for the author, so contributing is rewarded rather than absorbed. In the process, the system centralises two things each researcher previously solved alone: describing features, and describing the computation that turns them into signals.
A researcher joining today doesn't start from raw market data. They start from everything the firm has already built.
A shared base has to be trustworthy
There's a condition attached. If you're going to build on a feature you didn't write and can't read, its correctness can't rest on the author's personal diligence – it has to be a property of the platform. And the correctness that matters most is point-in-time correctness: the guarantee that a computation never sees data that hasn't arrived yet.
Point-in-time correctness is normally achieved with a streaming API rather than a vectorized one, and that carries a performance cost – because with a streaming API you cannot perform operations in vectorized form.
Streaming: honest about time, slow to replay
Streaming describes a computation as though it were running in production. Your robot is live, updates arrive from the market, and on each update it wakes up – that's a callback. On that callback you perform some computation.
An update arrives from the exchange, your handler fires, it recomputes a moving average and decides whether to quote. One update at a time, in order – exactly how production behaves.
This is honest about time by construction: at the moment the callback runs, later data doesn't exist yet, so nothing can reach for it. It isn’t fast – replaying years of history one callback at a time is slow enough to change how research gets done.
Vectorized: fast, but unguarded
Vectorized is when your data is laid out in a vector, spread across different points in time. With data in a vector you can compute far more efficiently.
Instead of a loop that wakes up ten million times, you hand the engine a single column of ten million prices and ask for the mean over every five-second window. One operation, distributed across all cores and able to make use of SIMD instructions on the CPU.
The catch is that you now have to track point-in-time correctness yourself, by hand, on every window and every alignment – and that kind of mistake doesn't announce itself. It just makes the backtest look slightly better than reality.
FQL: describe it one way, execute it the other
The Feature Store gives both at once. It has a streaming API and still computes in vectorized form, because its style of describing computation is declarative rather than imperative – closer to the approach of functional programming languages than to a chain of callbacks.
All computation is built on FQL, Feature Query Language, which describes the computation declaratively. Because you state what a feature is rather than the procedure that produces it, the engine is free to choose how to execute it.
How the guarantee works
Availability timestamps
Every value in the system carries an availability timestamp – the timestamp of the event plus all the latencies it has accumulated on the way to us.
The logic applied on top of it is the last observation carried forward, evaluated against availability TS. At any point where a feature is computed, it sees the most recent value that had actually arrived by then. Where nothing new has arrived, the previous value is carried forward.
An event stamped 12:00:00.000 by the exchange crossed a network and was only in our hands at 12:00:00.008. A feature evaluated at 12:00:00.005 doesn't see it – it carries forward the last genuinely available value, and picks the new one up at the next evaluation after it landed.
The availability timestamp is what encodes that eight-millisecond difference, and it's what makes the guarantee structural.
Last observation carried forward, evaluated against availability TS rather than exchange time.
Latency is a feature, not a setting
Latency is a first-class construct. You can declare it both on data sources and on the output of a feature. And latency is itself a feature – a feature of type Duration.
Because latency is a feature rather than a setting, it can vary over time like any other value. You can simulate a latency profile, or reconstruct a real one from latency logs and feed that in. And because it's expressed separately, that logic stays out of both the feature definition and the data source – swapping the latency model doesn't touch either.
Resampling timelines
Each feature has a resampling timeline – the grid on which the feature is evaluated. Between calls there are two modes: take the most recent available value, or accumulate all the updates that occurred between evaluations.
Which mode you want depends on the question. 'What is the price right now' wants the latest value. 'How much traded since I last looked' wants the accumulation.
This is also what makes cross-asset and cross-venue features tractable, where data arrives with different latency and the timestamps of different exchanges are not perfectly synchronised. You aren't hand-aligning clocks; you declare a grid and a policy, and the availability machinery does the rest.
The two modes available between evaluations of a feature.
Where the researcher's job ends
At the FQL level, a researcher doesn't have to track point-in-time correctness at all – unless they're implementing a new FQL operator, which is sometimes required.
If FQL doesn't have enough expressive power for what you need to compute, you can write the operator's code in Rust, and at that level it is possible to leak data from the future.
So the project is tightly integrated with LLMs. Dedicated agents, wired into the code itself, check the point-in-time correctness of a new FQL operator, and do so quite reliably. The guarantee at the language level is structural; at the operator level, it's actively verified.
The AST is the dependency graph
The computation graph is an AST tree of FQL operators, and that tree is the dependency graph. There is no separate structure describing what depends on what.
A feature built on two others isn't registered anywhere as depends on A and B. The dependency is simply the shape of the expression.
The graph and the definitions can never drift apart, and changing the order of recomputation changes the hash along with it.
One edit propagates upward: the same hashes drive dependency order, version identity and the cache key.
The hardest thing to design
Two requirements that pull apart
In your own computations you can reference features or operators whose code is not available to you, and you still want your graph computed. Meanwhile, a feature you're still developing lives locally, on your own machine – and you can't simply push it to the place where those private features execute. Yet you want the whole thing computed within a single graph.
A single graph matters because the execution engine underneath works best when handed one unified description: it can then distribute the computation across threads optimally, since it runs on a single thread pool, and optimise the plan as a whole. Split the computation in two and you lose both.
The hardest part to design was the transport between client and server, because it requires that everything computes identically on both sides. A disagreement between the two means a wrong number in somebody's backtest.
Bytecode across the wire
The resolution: the feature executes on the server. That's where the private code it references already sits. Your feature's code compiles to bytecode, the bytecode goes to the server, and the server interprets it there — alongside the private implementations it references.
One graph, evaluated in one place, with neither side's source exposed to the other.
The transport between client and server: bytecode crosses, source does not.
Knowing when to send
The second half of the problem is the logic that decides whether the code needs sending at all. If a quant has changed a feature locally, its bytecode has changed – so the system computes a hash of the AST (of the bytecode, in effect, not of the data) and ships only when the hash has moved.
Your machine and the server hold the same computation in two forms, and the hash is how they agree on whether they're still in sync.
Versions and reproducibility
Feature definitions have a version number, plus an expected-hash mechanism: the engine checks that the declared operator hash matches the computed one. A mismatch is a sign that the feature has been changed, and a prompt to decide what kind of change it was.
If it's backward-compatible, the version stays and only the expected hash is updated. If it's incompatible, both the version and the hash change.
Adding an optional parameter that leaves existing behaviour identical is the first kind. Changing a window from a mean to a median is the second – same feature name, different numbers, and every backtest that used it now means something else.
The payoff is that reproducing the feature set of a backtest run months ago is straightforward. In a shared base, where the features underneath your model are maintained by other people, that's what keeps old results meaningful.
Order books
In the data lake, order books are stored as ground-truth L2 snapshots together with the full history of increments between them. A dedicated Rust operator reconstructs the state of the book at any point in time you ask for: it starts at the nearest snapshot and replays the increments forward until it reaches that instant.
It's fast enough to run inside feature computation rather than as a separate preparation pass – which means a feature can ask for book state on its own grid, instead of inheriting whatever grid some earlier job happened to choose.
Book state is reconstructed on demand from the nearest snapshot plus the increments after it.
What's being built now
Caching
Caching is not yet implemented. It sits in the immediate backlog, behind the security work, which has to be solved first.
The design is a special FQL operator called cache, embedded directly into the AST, caching intermediate – or final – results. The cache key is the hash of the upstream AST plus the hash of the data that AST is applied to, an md5 sum, with all the parameters folded in. If any part of that changed, the cache recomputes; if not, the cached result is taken.
The bottleneck, and what happens at ten times the load
The main performance bottleneck is loading data from the data lake over the network. Everything else computes fast, and does so in practice.
If load grew tenfold, the change would be raising Slurm nodes and bringing up workers on demand, rather than computing on the server. That too requires certain security questions to be resolved first.
Offline and online
The same code can run in production – but for that, the AST has to be translated by an LLM onto the Rust-based Robot SDK API. That the two compute the same thing is verified in two ways: by analysis of the code, and by a detailed platform backtest.
There's an alternative approach: every FQL operator would carry a second definition expressed through the robot SDK, compiled into Rust templates. It's a heavier internal task, so translation came first.
Why the security model is the critical path for everything else on the roadmap.
Where this is going
The Feature Store isn't a finished platform in maintenance. The DSL exists, the transport works, features are computed through it, and the path into production trading is being cut now. Caching, the security model, elastic execution and the offline-to-online translation are all live problems with people on them.
If you do research, it's a place where the features you write become infrastructure other people build on – and where keeping your model honest about time is the platform's job, not yours.
Sources and further reading
The quant lineage
- 101 Formulaic Alphas – Zura Kakushadze, 2015. Explicit alpha formulas that are also code; the origin of much of the operator vocabulary used across the field.
- Qlib: An AI-oriented Quantitative Investment Platform – Microsoft Research. Expression engine that parses features into a syntax tree, with expression and dataset caching layered on top.
- Qlib data layer documentation – How the expression and dataset caches are keyed and shared.
- Factor engine design: DSL, AST and operator mapping – A practitioner overview of the DSL-to-computation-graph pattern across WorldQuant BRAIN, DolphinDB and Zipline Pipeline.
The ML-platform lineage
- Meet Michelangelo: Uber's Machine Learning Platform – Uber Engineering, 2017. The canonical description of a production ML platform with a feature store at its centre.
- Palette Meta Store Journey – Uber Engineering. How Palette organises curated and internally crowd-sourced features.
- Chronon, Airbnb's ML feature platform, is now open source – Varant Zanoyan, Airbnb Tech Blog, 2024. Point-in-time correct backfills, and online-offline consistency measured rather than assumed.
- Feature Store Benchmark Comparison: Hopsworks and Feast – Hopsworks. Useful for the shape of the conventional dual-store architecture and its serving latencies.
- Hidden Technical Debt in Machine Learning Systems – Sculley et al., NeurIPS 2015. Data dependencies as the costliest and least visible debt in ML systems.