S3 Data Files

Bulk exports of the Probalytics datasets as Parquet files. They’re served from an S3-compatible endpoint, so you can use the AWS CLI, boto3, curl, or any other S3 client. Use them to backfill, for local research, or to load into your own warehouse.

Dataset	Period	One file per
Markets	Monthly	Platform and calendar month of market creation
Fills	Daily and weekly	Platform and UTC day, or Monday–Sunday week
Orderbook snapshots	Daily	Platform and UTC day

Access

S3 Data Files are included in paid Pro, Orderbook, and Custom plans. They aren’t available during a trial. Your Probalytics API key is your S3 credential:

S3 setting	Value
Access key ID	API Key ID (`api_...`)
Secret access key	API Key Secret (`sk_...`)
Endpoint	`https://s3.probalytics.io`
Region	`hel1`
Bucket	`probalytics-data`
Addressing	Path-style

To create a key, see the Quickstart. If an older key is rejected, create a new one. Access is read-only: you can list and download objects.

Browse in the app

Log into app.probalytics.io and open S3 Data Files. You can filter the catalog by platform, type, and period, or search by filename. Each row shows the file’s size and row count. Files are not downloaded through the browser. Click a row’s download button, or select several rows and click Download, to get a ready-to-run cURL, AWS CLI, or Python snippet for those files.

Object layout

datasets/{dataset}/{platform}/{period}/parquet/{date_path}/{filename}

Dataset	Object key example
Fills, daily	`datasets/fills/polymarket/daily/parquet/2026/10/07/fills_polymarket_2026-10-07.parquet`
Fills, weekly	`datasets/fills/kalshi/weekly/parquet/2026/10/fills_kalshi_2026-09-28_to_2026-10-04.parquet`
Markets, monthly	`datasets/markets/polymarket/monthly/parquet/2026/09/markets_polymarket_2026-09-01_to_2026-09-30.parquet`
Orderbook, daily	`datasets/orderbook_snapshots/kalshi/daily/parquet/2026/10/07/orderbook_snapshots_kalshi_2026-10-07.parquet`

{platform} is polymarket or kalshi.
For daily files, {date_path} is YYYY/MM/DD. For weekly and monthly files it’s YYYY/MM, taken from the last day of the period. A week that crosses a month boundary is filed under the month it ends in.
Filenames are deterministic. A daily filename contains its date. Weekly and monthly filenames contain an inclusive start_to_end date range.
Periods with no rows have no file.

Download

The examples set credentials once through the standard AWS environment variables:

export AWS_ACCESS_KEY_ID='api_...'
export AWS_SECRET_ACCESS_KEY='sk_...'

AWS CLI
Python
cURL

# List what's available
aws s3 ls --endpoint-url https://s3.probalytics.io --region hel1 \
  s3://probalytics-data/datasets/fills/polymarket/daily/parquet/2026/10/

# Download one file
aws s3 cp --endpoint-url https://s3.probalytics.io --region hel1 \
  s3://probalytics-data/datasets/fills/polymarket/daily/parquet/2026/10/07/fills_polymarket_2026-10-07.parquet .

# Download a whole month of daily files
aws s3 cp --recursive --endpoint-url https://s3.probalytics.io --region hel1 \
  s3://probalytics-data/datasets/fills/polymarket/daily/parquet/2026/09/ ./fills-2026-09/

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.probalytics.io",
    region_name="hel1",
    # Reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY from the environment.
)

bucket = "probalytics-data"
prefix = "datasets/fills/polymarket/daily/parquet/2026/09/"

for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix):
    for obj in page.get("Contents", []):
        filename = obj["Key"].rsplit("/", 1)[-1]
        s3.download_file(bucket, obj["Key"], filename)
        print(filename, obj["Size"])

Requires curl 7.75 or later for SigV4 signing.

curl --aws-sigv4 "aws:amz:hel1:s3" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  "https://s3.probalytics.io/probalytics-data/datasets/fills/polymarket/daily/parquet/2026/10/07/fills_polymarket_2026-10-07.parquet" \
  -o fills_polymarket_2026-10-07.parquet

File format

Files are standard Parquet with internal ZSTD compression and no outer gzip. Any Parquet reader can open them directly.

import polars as pl

fills = pl.read_parquet("fills_polymarket_2026-10-07.parquet")
print(fills.schema)

The columns match the ClickHouse tables documented in the SQL tables reference and the Datasets pages, with these differences:

UUID columns (id, market_id, and the IDs inside outcome, outcomes, legs, and resolution_outcome_payouts) are written as strings.
outcome and outcomes are Parquet structs with id, platform_id, name, and index fields.
Rows are deduplicated, so each row identity appears once.

Dataset	Rows included	Sort order
Markets	Markets whose `created_at` falls in the month	`created_at`, `id`
Fills	Fills whose `timestamp` falls in the UTC day or week	`market_id`, `timestamp`, `id`
Orderbook snapshots	Snapshots whose `timestamp` falls in the UTC day	`market_id`, `outcome.id`, `timestamp`

Orderbook files contain every snapshot for the platform and day, including INTERMEDIATE rows. See Orderbook snapshots to choose which rows to keep.

Availability and freshness

Files only cover complete periods: a day, week, or month is exported after it has ended. Exports run in the early hours UTC, so files usually appear during the following day:

Dataset	Usually available
Orderbook snapshots, daily	Early hours UTC of the next day
Fills, daily	From about 04:35 UTC of the next day
Fills, weekly	From about 04:45 UTC on the Monday after the week ends
Markets, monthly	From about 04:15 UTC on the first day of the next month

These times are not guaranteed. A large backfill or an operational issue can delay a file.

A file is a snapshot of the data as of when it was generated, and published files are normally not regenerated. Specifically:

A monthly markets file keeps each market’s status and resolution fields as of the export. Markets that resolve later still appear unresolved in older files.
Fills or snapshots that arrive after a period was exported aren’t added to that period’s file.

For current market state or late corrections, query SQL or the REST API.

History

Fills and markets files start at each platform’s earliest data. Orderbook snapshot files start on 2025-11-20 for Polymarket and 2026-05-09 for Kalshi.

Getting Started

Datasets

Access Methods

REST API

Reference

Status

Support

Access

Browse in the app

Object layout

Download

File format

Availability and freshness

History

​Access

​Browse in the app

​Object layout

​Download

​File format

​Availability and freshness

​History

Access

Browse in the app

Object layout

Download

File format

Availability and freshness

History