Oracle performance isn’t binary. Feeds don’t simply “work” or “fail.” They exhibit complex, context-dependent behaviors that documentation struggles to capture and theoretical analysis can’t always predict. Update latencies that vary with block builder competition, price dislocations that amplify when arbitrage breaks down, and update cadences that shift unpredictably during the exact moments when accuracy matters most. When evaluating collateral assets for lending protocols such as FiRM, most risk assessments focus on liquidity depth, smart contract audits, economic incentives, and historical price volatility. Oracle reliability, despite being the critical infrastructure that enables every liquidation, often receives lighter treatment: check that a reputable feed exists, verify it’s been operational for some time, confirm the heartbeat interval and deviation thresholds meet baseline standards, then move on. But when oracle systems fail during market stress, lending protocols face catastrophic risk. A price feed that freezes for minutes while an asset crashes can turn healthy positions into bad debt faster than any liquidator can react. Traditional approaches to oracle evaluation provide theoretical baselines but reveal nothing about actual behavior when markets enter crisis mode.
The broader market stress test of October 10th revealed systematic oracle performance issues across multiple providers and virtually all assets. From this, the RWG recognized the need for infrastructure to conduct that forensic analysis in a repeatable, comparable way across different assets, different time windows, and different oracle configurations. We need the ability to answer questions such as: How did this specific oracle feed actually perform during the most volatile trading period in the asset’s recent history? How do different oracle architectures respond to identical market conditions? Can we identify on-chain signals that predict when oracle reliability is degrading before failures materialize?
In this installment of RWG: Behind the Scenes, we cover the RWG’s approach to understanding oracle behavior during market stress: why price feed analysis matters beyond what’s provided in documentation, what we’ve learned from analyzing recent crashes and how we’re developing infrastructure to benchmark oracle reliability empirically rather than theoretically.
The Problem: Oracle Opacity During Stress
A feed configured with a 0.5% deviation threshold should trigger updates whenever price moves 0.5% from the last reported value. But when this feed makes sequential updates exceeding 10% deviation repeatedly, we need to stop in our tracks and understand why. Did the feed malfunction in the sense of breaking its technical implementation? Or were the conditions that trigger updates (node consensus, transaction inclusion, block builder competition) under such pressure that the designed update cadence was overwhelmed? Understanding this distinction requires understanding the layers between “market price changes” and “oracle updates on-chain.”
When an asset’s price moves on a DEX, this creates a state change in the pool contract that’s immediately visible on-chain. Oracle nodes monitoring this pool observe the change, but must independently verify the new price across multiple data sources before voting to update. Consensus-based oracle protocols require threshold agreement across node operators before constructing an update transaction. This transaction enters the mempool where it competes with thousands of other transactions for block inclusion. Block builders prioritize transactions offering highest fees, and during volatility when liquidation bots, arbitrageurs, and normal users all simultaneously demand block space, oracle update transactions must compete for inclusion against transactions offering premium fees because they enable profitable operations. If the oracle update transaction offers insufficient fees relative to competing demand, it waits in the mempool. If it waits long enough, the next round begins with an even more stale price, creating a backlog that compounds latency. This reality means oracle update timing and by extension FiRM solvency depends on block builder economics; a dependency that becomes hard to predict during stress.
Along with block builder competition, FiRM solvency depends on arbitrage functioning across venues. If an asset trades at different prices on Binance, Coinbase, Bybit, and various DEXs, arbitrage should rapidly equalize these prices, giving oracle nodes consistent data to aggregate. But during market stress, price divergence between major venues can persist for extended periods because the cost to execute arbitrage (gas fees, block builder fees, slippage) exceeds the profit available from the price differential. When arbitrage fails, “true price” becomes ambiguous. Different oracle nodes observing different venues might report legitimately different values, and consensus mechanisms must somehow aggregate these into a single canonical price while time-sensitive liquidations wait for an answer.
All of this compounds into a problem space that’s fundamentally difficult: providing fast, accurate, reliable pricing during precisely the conditions when fast, accurate, and reliable become mutually challenging to achieve. Block builders demand higher fees during volatility. Arbitrage breaks down during stress. Price discovery fragments across venues. Node consensus takes longer when data sources disagree. Transaction inclusion becomes unpredictable when competition spikes. Each layer introduces latency or uncertainty, and the layers interact in complex ways that testing during calm markets won’t reveal. This is why theoretical analysis of oracle feeds alone is an insufficient evaluation. We need empirical data about how feeds actually behave during the specific market conditions where lending protocols face maximum risk.
The Need: Price Feed Analysis as Risk Infrastructure
Lending protocols operate on the assumption that price feeds accurately reflect market reality with sufficient speed and reliability to enable safe liquidations before positions accumulate bad debt. For most collateral assets most of the time, this assumption holds; oracles update regularly, prices track market movements reasonably, liquidations process smoothly. The challenge is that “most of the time” provides false confidence about behavior during the minority of time that actually determines whether a protocol’s risk parameters are adequate.
October 10th demonstrated this gap clearly. The RWG initially struggled with post-event analysis as we lacked infrastructure to conduct price feed performance analysis. Manual review of blockchain data, examining individual transactions, spot-checking specific timestamps; this works for investigating known incidents down to a block number but doesn’t scale when evaluating time-based functions. We needed a methodology that could consume a price feed, liquidity source, time window and produce standardized datasets revealing actual oracle behavior at block-level granularity under real market conditions. The goal was building capability to measure how specific oracles perform under duress, enabling more informed parameterization for collateral markets on FiRM.
The Solution: The Price Feed Analysis Toolkit
Enter our new Price Feed Analysis Toolkit: a modular, general-purpose framework for evaluating how reliably and how quickly oracle systems react during volatile or stressed market conditions. By comparing Curve spot and EMA pool pricing against Chainlink OCR updates at the block level, we can now measure update latency, price dislocation, and the overall effectiveness of oracle configurations during periods of rapid price movement. The system also tracks liquidity-pool pairing depth at each sampled block, giving visibility into how market depth changes during oracle updates. This provides crucial context: deteriorating liquidity or asymmetric pool composition often coincides with delayed or unstable oracle responses, revealing pivotal risk environments that can impair feed reliability.
Each component of the toolkit handles a single responsibility through parameterized inputs, making the entire pipeline adaptable to any asset without code modification.
ChainlinkFeedReporter.pycollects all Chainlink rounds within a date range, walking backwards through round history to gather timestamps and prices for every update.curve_pool_price_reporter_events.pycaptures event-driven pricing from any Curve pool, sampling spot prices, EMA prices, and liquidity depth at every swap event.ChainlinkBlockRecovery_CurveAnchored.pybridges Chainlink’s timestamp-based round data with Ethereum’s block-based architecture by scanning AnswerUpdated logs to recover exact block numbers.FinalCSVExporter.pymerges these datasets into a unified, block-aligned analysis CSV through merge_asof joins, computing critical metrics like chainlink_latency_seconds (time between Curve block timestamp and most recent Chainlink update), price dislocation percentages, and sequential move magnitudes.CHART_Overview.pyprovides interactive baseline visualization that automatically detects crash epicenters and plots oracle behavior relative to market reality, highlighting every sequential price move exceeding specified thresholds.
The entire pipeline requires no code modification between assets; configuration happens through runtime prompts and parameter files. This means anyone on the team can execute the same analysis workflow for a new collateral candidate, a different time window, or a comparative study across multiple feeds, without understanding the implementation details. The framework produces standardized outputs that enable apples-to-apples comparison.
Real-World Example: What the Tool Reveals
The toolkit’s value becomes concrete through actual crash analysis showing how different assets and oracle configurations behave under stress. Below is an example from October 10 2025 that demonstrates the insights this systematic approach reveals.
Example: CRV/crvUSD
This export from
FinalCSVExporter.py sample data is a block-by-block snapshot (25 rows) over a short window on 2025-10-10 21:17:47 → 21:25:47 UTC, comparing Curve pool pricing for CRV/USD (spot + EMA) against the Chainlink CRV/USD feed, and adding derived volatility/dislocation + latency metrics plus pool depth context.
Important headers (what they represent):
-
timestamp_utc— observation time (UTC) for the row. -
block_number— Ethereum block number aligned to the observation. -
curve_spot_price_CRV_in_USD— instantaneous Curve spot price of CRV in USD. -
curve_ema_price_CRV_in_USD— Curve’s EMA oracle price of CRV in USD (smoother / lagging vs spot). -
chainlink_price_CRV_in_USD— Chainlink price of CRV in USD (normalized human-readable price). -
chainlink_round_id— Chainlink OCR round identifier for the reported answer. -
chainlink_updated_at— the timestamp associated with that Chainlink round update. -
chainlink_answer— raw Chainlink answer integer (typically scaled; here it lines up with 1e8, e.g. 51575900 → 0.51575900). -
chainlink_price_change_pct— % change in Chainlink price vs the prior Chainlink update in the dataset. -
ema_price_change_pct— % change in Curve EMA price vs the prior EMA observation in the dataset. -
chainlink_vs_ema_dislocation_pct— % difference between Chainlink price and Curve EMA (directional). -
chainlink_latency_seconds / ema_latency_seconds— timing gap (in seconds) between the row timestamp and the oracle’s update timestamp/observation timestamp (note: if you see negatives, it usually means the subtraction direction is flipped in the calculation). -
pool_balance_crvUSD— pool-side balance (depth proxy) for crvUSD at that block, used to contextualize price reliability/liquidity.
From the
CHART_Overview.py visualization above we note: Chainlink’s CRV price (blue) may have kept reasonable pace with the crash but made significant sequential self-moves of -15.9%, -32.6%, -10.3%, 33.5%, and 17.3%. The Curve spot price (green) shows similar rapid swings but with more volatility (arbs), while the Curve EMA (orange) tracked the general downtrend at the expected rounded pace. In this sample we’ve normalized the Curve pool crvUSD price to USD using the Chainlink crvUSD/USD feed to replicate a FiRM market integration.
Beyond individual asset analysis, this methodology enables direct comparison, revealing patterns that inform strategic decisions. For example, early evidence suggests Curve EMA feeds may be better suited for liquidation cascade situations, where price almost always recovers, while Chainlink oracles are better for situations where collateral actually becomes nearly worthless. Another observation is that feeds experiencing the largest sequential moves don’t necessarily show the longest latencies. Sometimes oracles update reasonably quickly but in very large jumps rather than smaller incremental adjustments, implying different failure modes require different risk considerations. Assets with multi-feed dependencies show compounding reliability challenges versus assets with direct feeds. These insights directly inform parameterization: when we evaluate a potential new market for FiRM, benchmarking of its underlying collateral’s oracle performance during highest historical volatility can inform us on where to set collateral factors.
Lessons and Future Development
The toolkit as currently implemented provides robust crash-episode analysis but several enhancements would expand its utility. Incorporating Redstone, Pyth, Chronicle, and other feeds alongside Chainlink would enable direct comparative analysis of how different oracle architectures respond to identical market conditions. In similar fashion, expanding the Curve-side analysis to incorporate additional DEX architectures e.g. Uniswap v3 concentrated liquidity pools would allow for similar comparisons. This would inform strategic decisions about diversifying oracle dependencies or implementing fallback mechanisms that switch between oracles based on real-time reliability signals. Future iterations may lead us to build out statistical models that predict oracle failure probability based on observable on-chain conditions.
As it stands, the insight made available by the toolkit can be instrumental in developing and evaluating hybrid oracle architectures customized for select collateral assets. Such adaptive systems require deep understanding of each oracle’s specific failure modes and performance characteristics - exactly the knowledge the toolkit generates through systematic stress-test benchmarking. On this note, Chainlink has offered to collaborate with us to build custom, FiRM-specific market specced oracles, similar to their work with Aave.
The broader lesson is that DeFi risk management demands infrastructure that can answer questions about actual system behavior under stress, rather than relying on assumptions about how systems “should” perform. Oracle reliability affects every lending protocol, every liquidation system, every parameter decision that assumes accurate pricing during volatility. The Price Feed Analysis Toolkit provides one systematic framework for transforming oracle evaluation from theoretical review into empirical science, enabling decisions backed by data about what actually happens when markets enter crisis mode. As we continue making use of our existing framework infrastructure and further develop them to better serve other risk dimensions, the philosophy remains consistent: build general-purpose, repeatable frameworks that accumulate institutional knowledge rather than conducting one-off analyses that don’t transfer to the next evaluation. This is how we build risk management practices that become more effective over time rather than repeatedly learning the same lessons through successive failures.
Catch you next time.