Home › Key concepts › File formats: CSV, JSON, Avro, Parquet

Key concept · Crawl (beginner) · the same on every platform

File formats: CSV, JSON, Avro, Parquet

How data is laid out in a file decides how fast and cheap it is to read.

What it is

CSV and JSON are text, easy to read by eye, but slow and large. Avro stores rows in binary with a schema, good for streaming messages. Parquet stores data by column and compresses it, so a query that needs three columns reads only those three.

Row-based (CSV, JSON, Avro)datestoreitemamountdatestoreitemamountdatestoreitemamounta whole row is stored togetherColumnar (Parquet)datedatedatestorestorestoreitemitemitemamountamountamount'sales by date' reads only 2 of 4 columns
Analytics reads a few columns across many rows, which is why columnar files are faster and cheaper.

Why it matters

Analytics reads a few columns over many rows, which is exactly what columnar formats are built for. The same data in Parquet is often ten times smaller than CSV.

An example

A 10 GB CSV of sales becomes about 1 GB of Parquet, and 'total sales by month' reads only the date and amount columns.

The common mistake

Keeping CSV as the analytics format because it opened nicely in a spreadsheet once.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Object storagecheap files in the cloud, the floor everything stands onAmazon S3ADLS Gen2 / OneLakeCloud StorageUnity Catalog volumes on cloud storageLocal disk or MinIO

Related ideas

Where it shows up

Part of the Crawl level on these paths: Data Engineering (storing data).