CSV vs Parquet: When Converting Actually Pays Off

Two formats, two design goals

People often treat CSV and Parquet as the old and new version of the same thing. They were built for different jobs, and most bad conversion decisions come from forgetting that.

CSV is plain text with one record per line and values separated by a delimiter. RFC 4180, published in 2005, describes the common conventions, and it is an informational document rather than a strict standard. That looseness is why CSV is everywhere: any program can read and write it, and a person can open it in a text editor. It also causes the problems. There is no schema, no types and no compression, and nothing guarantees the delimiter, the quoting or the encoding.

Apache Parquet is a binary, columnar format that emerged in the early 2010s for analytical workloads. Instead of storing row after row, it stores the values of each column together, grouped into row groups. The file carries its own schema with data types, compresses each column separately, and keeps statistics such as minimum and maximum values in a footer.

Why columns compress and scan better

Values in one column share a type and tend to resemble each other: a country code repeated millions of times, timestamps that increase steadily, a status field with five possible values. Parquet exploits this with encodings such as dictionary encoding and run-length encoding, and then applies a general-purpose codec such as Snappy, Gzip or Zstandard on top.

The same layout helps reading. An engine that needs two columns out of forty can read only those two, and it can skip whole row groups whose minimum and maximum values rule out a match. A CSV reader has no such option: to find the value of one column it must scan and parse every line.

What you gain by converting

• Smaller files when the data is repetitive. Categorical columns, dates and low-cardinality identifiers shrink a lot. Columns of random identifiers or high-precision floating-point numbers shrink much less.
• Faster analytical queries, because engines read only the columns and row groups they need.
• Preserved types. Integers, decimals and timestamps stay what they are, so a zip code written as 00123 is no longer at the mercy of a spreadsheet that turns it into 123.
• A schema that travels with the data, so the next person does not have to guess what each column means.

What you lose

• Direct readability. You cannot open a Parquet file in a text editor or double-click it into a spreadsheet.
• Cheap appends. Parquet files are written in row groups and are not meant to be edited in place, so adding a row means writing a new file.
• Simplicity for small files. The footer and metadata have a fixed cost, which can make a tiny Parquet file larger than the CSV it came from.
• Compatibility. Some business tools and older systems accept only CSV.

A decision guide

Convert to Parquet when the file is large, when you read it repeatedly and usually need only some of the columns, when several pipelines share it and need consistent types, or when it will sit in object storage for a long time. Keep CSV when the file is small, when people edit it by hand, when you send it to someone who just needs to open it, when the destination system accepts only CSV, or when the job is a one-off.

How to measure it on your own data

Compression results depend entirely on the data, so any universal ratio you read online should be treated with suspicion. A reliable answer takes a few minutes with your own file:

1. Profile the CSV first. A data profiler shows column types, null counts and the number of distinct values, which tells you which columns will compress well.
2. Convert the file to Parquet with a CSV to Parquet converter.
3. Compare the two file sizes, and note which columns you expected to shrink.
4. Read the schema of the result. Type inference can guess wrong, and identifiers with leading zeros, long numeric codes and mixed-format dates are the usual casualties. Fix the source or the conversion settings if a column came out with the wrong type.
5. Inspect the metadata: number of row groups, compression codec and per-column statistics. This explains the size you got.
6. Convert the Parquet file back to CSV and compare it with the original. Keep the original until the round trip matches.

Every step above can be done in the browser without uploading the file, which matters if the data is confidential. The linked tools below cover each one.

Common mistakes when converting

• Trusting type inference blindly. A column that looks numeric in the first thousand rows can contain text further down, and the converter may either fail or fall back to strings.
• Converting many tiny files one by one. Parquet shines on larger files, so merge small CSV exports first and write fewer, bigger files.
• Throwing away the source. Treat the CSV as the archive until you have verified the Parquet file, and keep a note of the conversion settings you used.
• Ignoring time zones. A timestamp written without an offset in CSV is ambiguous, and converting it forces a decision. Make that decision explicitly rather than letting a default make it for you.

Why there is no headline ratio

I have not quoted a figure such as "Parquet is ten times smaller". For some datasets it holds, for others the gain is modest, and for tiny files the result can be worse. If a format is going to sit at the centre of your pipeline, spend ten minutes testing it on a representative sample before you commit.