Futures Database in Python: Complete Pipeline (Norgate Data)

A complete pipeline for contract specs, individual contracts, and continuous futures in Python

Quantitative futures research depends heavily on how the underlying data is constructed. Small inconsistencies in contract handling, pricing, or roll logic can lead to materially different results.

A usable futures database in Python typically relies on three core datasets: contract specifications, individual contract histories, and a continuous series.

The challenge is not access, but structure. These datasets follow different conventions and are not aligned by default, which makes it easy to combine them incorrectly or use the wrong prices.

The data used here comes from Norgate Data, which provides long historical coverage across futures markets, with some contracts going back to the 1980s.

This article shows how to retrieve and organize them into a single, consistent pipeline.

What You Will Learn

You will be able to retrieve contract specifications, individual contract data, and continuous futures series, and combine them into a set of Parquet files ready for research.

This workflow shows how to structure a futures database in Python so the data can be used directly for research and backtesting.

Futures Database Output Example

The notebook produces four files in the norgate_output/ folder:

File	What’s inside	When to use it
01_contract_specs.parquet	Point value, tick size, margin, currency, exchange, group	Look up contract details for any futures market
02_individual_contracts_ohlcv.parquet	OHLCV for every individual contract (including expired)	Backtesting on actual contract-level data (rolls, expiry, etc.)
03_continuous_adjusted_and_unadjusted.parquet	Back-adjusted OHLCV + unadjusted close & volume in the same row	Standard continuous series for research (adjusted for signals, unadjusted for actual prices)
04_continuous_with_volume_decomposition.parquet	Continuous series enriched with first- and second-to-expire contract volumes	Volume analysis, roll timing, and liquidity research

Output Columns

01_contract_specs.parquet

Header	Explanation
Symbol	Norgate continuous futures symbol (e.g. &ES_CCB for E-mini S&P 500 back-adjusted).
Name	Full security name as provided by Norgate Data.
Exchange	Exchange where the contract trades (e.g. CME, NYMEX, ICE).
Group	Broadest Norgate classification (e.g. Currency, Energy, Grains).
Contract Size	Value of a one-point move in the contract’s currency.
Tick Size	Minimum price increment for the contract.
Tick Value	Value of one tick in the contract’s currency. Calculated as Tick Size × Contract Size.
Point Value	Same as Contract Size.
Currency	Contract denomination (e.g. USD, GBP, EUR).
Margin	Exchange margin requirement in the contract’s currency.

02_individual_contracts_ohlcv.parquet

Header	Explanation
Date	Trading date.
Open, High, Low, Close	Price data for the individual contract.
Volume	Traded volume for this individual contract.
Open Interest	Open interest for this individual contract.
Symbol	Individual contract symbol (e.g. ES-2024Z for Dec 2024 E-mini S&P).

03_continuous_adjusted_and_unadjusted.parquet

Header	Explanation
Date	Trading date.
Open, High, Low, Close	Back-adjusted prices (shifted to eliminate gaps at contract rolls).
Volume	Total volume across all active contracts.
Delivery Month	Delivery month of the current front-month contract (encoded as YYYYMM).
Open Interest	Total open interest across all active contracts for this market.
Symbol	Continuous symbol (e.g. &ES_CCB). The _CCB suffix denotes back-adjusted.
Close Unadjusted	Closing price of the front-month contract

04_continuous_with_volume_decomposition.parquet

Header	Explanation
(all columns from 03)	Same back-adjusted series with unadjusted prices.
FirstVolume	Volume of the first-to-expire (front-month) contract.
FirstContract	Symbol of the first-to-expire contract (e.g. ES-2025M).
SecondVolume	Volume of the second-to-expire contract.
SecondContract	Symbol of the second-to-expire contract (e.g. ES-2025U).

Download & Run the Code

The full pipeline is provided as a single Jupyter notebook.

Quick Setup

1. Start Norgate Data

Open the Norgate Data Updater and keep it running in the background. This application provides the connection required for the Python API, so the code will not run unless it is active.

2. Enable Required Databases

Inside the Norgate Data Updater, navigate to the Database section and ensure that both the Futures and Continuous Futures databases are fully active.

The Futures database contains all individual contracts, including expired ones, while Continuous Futures provides the stitched front-month and back-adjusted series. If either is missing, select it, download it, and wait until it is fully available before proceeding.

3. Run the Notebook

After adjusting the configuration in the first cell (such as date range or output settings), run the entire notebook. In Jupyter, this can be done using the “Run All” option.

The notebook will download the data, process it, and generate the four Parquet files automatically in the output folder.

How the Workflow Operates

1. Contract Specifications

The workflow begins by retrieving contract metadata such as tick size, point value, margin, and currency for each market. This provides the foundation needed for sizing and risk calculations.

2. Individual Contracts

All individual futures contracts are then downloaded. Each contract is queried once to obtain the full OHLCV series with open interest. These are merged into a single dataset.

3. Continuous Futures

Continuous futures series are constructed by combining back-adjusted prices with the corresponding unadjusted front-month data. This produces a dataset where adjusted prices can be used for signal generation, while real prices are preserved for accurate sizing calculations.

The Continuous Futures database contains two types of symbols for each market:

Symbol type	Example	Contains
Without _CCB	&ES	Unadjusted front-month prices – the actual traded price at each point in time.
With _CCB	&ES_CCB	Back-adjusted continuous series – prices shifted to eliminate gaps at contract rolls.

The merge strips the _CCB suffix to create a BaseSymbol, then joins on Date + BaseSymbol so each adjusted row picks up the unadjusted Close and Volume from its non-_CCB counterpart.

4. Volume Decomposition

The final step enriches the continuous series with volume data from individual contracts. For each date, the pipeline identifies the first-to-expire and second-to-expire contracts and records their volumes.

This is done by parsing individual contract symbols (e.g. `ES-2025M`) into base code and expiry date, building a map of all contracts per commodity sorted by expiry, and then matching each date in the continuous series to the two nearest active contracts.

Understanding the Row Counts

Dataset	Symbols	What each symbol is	Typical rows
Individual contracts	~27,000	One specific contract month (e.g. ES-2024Z)	~11 million
Continuous back-adjusted	~112	One market, all rolls stitched (e.g. &ES_CCB)	~880,000

The individual contracts dataset contains roughly 27,000 symbols, each representing a specific contract month such as ES-2024Z, and totals around 11 million rows. The continuous dataset contains roughly 112 symbols, each representing a full market, and totals around 880,000 rows.

This difference reflects the role of continuous series, which compress thousands of individual contracts into a single time series per market.

Conclusion

This workflow produces four datasets for futures research: contract specifications for sizing and margin, individual contract data for contract-level analysis, continuous series that combine adjusted and unadjusted prices, and a volume decomposition layer for roll timing and liquidity analysis.

The first three form the foundation, while the fourth links the continuous series back to its underlying contracts, making it straightforward to move between representations without rebuilding the data.

All outputs are stored as Parquet files for efficient access and can be used directly in a backtesting framework or research pipeline without further preprocessing.