Pandas turned 16 years old in 2024 and it still ships with every data science bootcamp curriculum in India. Then Polars arrived, written in Rust, and the benchmark numbers raised eyebrows — 10x faster groupby, 8x faster joins, memory usage cut by 60%. A reasonable person looks at those figures and wonders why anyone is still using pandas.
The honest answer is that raw speed is only one variable. This post runs both libraries against a 10-million-row order dataset, shows real timing numbers, and explains the parts of a data science workflow where switching creates new problems rather than solving them.
Why Polars Was Built — pandas' Structural Limits
Pandas was designed in 2008, when a "large" dataset meant a few hundred thousand rows and single-core CPUs were the norm. Three architectural decisions from that era cause pain at scale today.
The GIL problem. Python's Global Interpreter Lock prevents true thread-level parallelism. Pandas is bound by it — even operations that look parallel (like groupby) run on a single CPU core. On a 16-core AWS EC2 instance, pandas uses roughly 1/16th of available compute for most aggregation operations.
Memory overhead. A DataFrame in pandas typically consumes 5 to 8 times the raw size of the data it holds. A CSV containing 1 million order rows at roughly 80 bytes per row (about 80 MB on disk) will occupy 400–640 MB once loaded into memory. That's because pandas stores data in NumPy arrays with Python object overhead, reference counting metadata, and per-column dtype wrappers. On a 100M-row dataset, you can exhaust 32 GB of RAM doing operations that touch only two or three columns.
No native lazy evaluation. When you write df.groupby('state').agg({'amount': 'sum'}), Python evaluates each step immediately. In a five-step pipeline, pandas materialises the full intermediate result at every step. Polars supports a lazy execution plan that analyses the entire query, pushes filters down to read only needed rows, and eliminates redundant computation before any data moves.
Polars was built to remove all three constraints. Written in Rust (no GIL), using Apache Arrow columnar memory format (far more compact than NumPy arrays for typed data), and implementing a query optimiser in the style of a SQL engine, it approaches the same analytical problems from a completely different foundation.
Benchmark Setup and Results on 10M Rows
The test dataset is a synthetic order table with 10 million rows and the following schema:
order_id— UUID stringcustomer_id— integer (1 to 500,000)product_id— integer (1 to 2,000)amount— float (10.0 to 50,000.0)state— one of 28 Indian statesorder_date— date between 2022-01-01 and 2025-12-31
The CSV file is 1.4 GB on disk. Tests ran on an M2 MacBook Pro (10-core CPU, 16 GB RAM) with Python 3.12, pandas 2.2.2, and Polars 1.6.0. Each operation was timed with timeit over three runs and the median taken.
| Operation | Pandas | Polars (eager) | Polars (lazy) | Speedup |
|---|---|---|---|---|
| Read CSV (1.4 GB) | 41.2 s | 8.4 s | — | 4.9x |
| Groupby + agg (sum amount per state) | 3.8 s | 0.41 s | 0.38 s | 9.3x |
| Join orders to products (inner, on product_id) | 5.1 s | 0.62 s | 0.54 s | 8.2x |
| Filter (amount > 5000) + sort by amount desc | 2.2 s | 0.29 s | 0.21 s | 7.6x |
| Window function (running total per customer) | 14.7 s | 1.3 s | 1.1 s | 11.3x |
Peak RAM during the groupby: pandas held 9.8 GB versus Polars' 2.1 GB. On a standard 8 GB laptop — what most Indian data analysts at mid-market companies use — the pandas version crashes with a memory error. Polars completes it without issue.
The lazy mode numbers are worth noting. Calling pl.scan_csv() switches to lazy evaluation. For the filter+sort test, the optimiser recognised that only rows where amount > 5000 needed sorting and skipped processing the other 8.6 million rows. That is not a trick — it is what query planners in databases have done for decades, now available in a Python library.
API Differences That Trip Up Pandas Developers
The syntax shift is real. Understanding it upfront prevents frustration during migration.
Lazy vs eager evaluation. Polars has two modes. pl.read_csv() is eager — loads everything immediately, like pandas. pl.scan_csv() is lazy — returns a LazyFrame and nothing executes until you call .collect(). Use lazy for pipelines chaining multiple operations; use eager for interactive exploration.
# Polars lazy — nothing executes until .collect()
result = (
pl.scan_csv("orders.csv")
.filter(pl.col("state") == "Kerala")
.group_by("product_id")
.agg(pl.col("amount").sum().alias("total_amount"))
.sort("total_amount", descending=True)
.collect()
)No Index concept. Pandas' Index is core to how the library works — it tracks row identity through merges, resamples, and pivots. Polars has no Index. Every operation produces a new DataFrame with row positions starting from 0. Code that relies on .loc[] with named indices, .set_index(), or reset_index() needs rethinking.
Expressions syntax vs method chaining. In pandas you write df['amount'].mean(). In Polars the equivalent inside a .select() or .agg() is pl.col('amount').mean(). The pl.col() expression is composable — you can pass multiple expressions in a single .select() call and Polars evaluates them in parallel.
# Pandas
df.groupby('state')['amount'].agg(['mean', 'sum', 'count'])
# Polars — all three run in parallel internally
df.group_by('state').agg([
pl.col('amount').mean().alias('avg_amount'),
pl.col('amount').sum().alias('total_amount'),
pl.col('amount').count().alias('order_count'),
])Null handling differences. Pandas uses NaN (a float sentinel) for missing numeric values and None for objects, causing well-known type promotion issues. Polars uses a proper null bitmask — all types support null natively. pl.col('amount').is_null() works identically for integers, floats, and strings.
When Pandas Still Wins
Speed is not the only consideration. There are situations where staying on pandas is the more defensible choice.
Scikit-learn and seaborn integration. Every sklearn estimator's fit() expects a pandas DataFrame or a NumPy array. Seaborn plots and most AutoML libraries (FLAML, AutoGluon) are built around pandas. You can convert with df.to_pandas(), but if you're converting back and forth frequently the overhead erases gains on smaller datasets.
Datasets under 2 GB RAM. Below roughly 2 million rows, pandas' tooling advantages outweigh Polars' speed. The absolute time difference — say, 0.4 seconds versus 0.05 seconds for a groupby on 500K rows — is not a bottleneck in an interactive notebook session.
Existing Jupyter notebook workflows. If a team has 200 notebooks using pandas idioms, migrating and re-validating them is real work. For greenfield pipelines, Polars is the better default. For maintenance of existing analytical code, don't fix what isn't broken.
Legacy code using pandas-specific features. pd.MultiIndex, pd.Categorical, timezone-aware datetime operations, and pd.to_datetime() with format inference all have Polars equivalents but with different behaviour. Code that relies heavily on these features requires careful testing, not just a find-and-replace migration.
10 Common Operations Side-by-Side
The ten operations a migrating team will encounter most often, shown in both syntaxes.
1. Groupby aggregation
# Pandas
df.groupby('state')['amount'].sum().reset_index()
# Polars
df.group_by('state').agg(pl.col('amount').sum())2. Merge / join
# Pandas
pd.merge(orders, products, on='product_id', how='inner')
# Polars
orders.join(products, on='product_id', how='inner')3. Filter rows
# Pandas
df[df['amount'] > 5000]
# Polars
df.filter(pl.col('amount') > 5000)4. Apply equivalent (element-wise function)
# Pandas — slow for large DataFrames
df['gst'] = df['amount'].apply(lambda x: x * 0.18)
# Polars — vectorised, no Python loop overhead
df = df.with_columns((pl.col('amount') * 0.18).alias('gst'))5. Pivot table
# Pandas
df.pivot_table(values='amount', index='state',
columns='product_id', aggfunc='sum')
# Polars
df.pivot(values='amount', index='state',
on='product_id', aggregate_function='sum')6. Rename columns
# Pandas
df.rename(columns={'amount': 'order_value', 'state': 'region'}, inplace=True)
# Polars
df = df.rename({'amount': 'order_value', 'state': 'region'})7. Fill null values
# Pandas
df['amount'] = df['amount'].fillna(0)
# Polars
df = df.with_columns(pl.col('amount').fill_null(0))8. Date parsing
# Pandas
df['order_date'] = pd.to_datetime(df['order_date'], format='%Y-%m-%d')
# Polars — explicit format for performance
df = df.with_columns(pl.col('order_date').str.to_date('%Y-%m-%d'))9. Read Parquet (column pruning)
# Pandas — reads all columns
df = pd.read_parquet('orders.parquet')
# Polars lazy — reads only columns you select from disk
df = (pl.scan_parquet('orders.parquet')
.select(['order_id', 'amount', 'state'])
.collect())10. Write to CSV
# Pandas
df.to_csv('output.csv', index=False)
# Polars
df.write_csv('output.csv')Polars in Production
Moving from a Jupyter notebook to a production data pipeline changes what matters. Here is how Polars fits into common production patterns.
Out-of-core processing with streaming mode. Polars' streaming mode lets you process a file larger than available RAM by processing it in chunks internally. Enable it by appending .collect(streaming=True) to any lazy query. An Indian logistics company processing 500 million shipment records per month can run aggregations on a 16 GB RAM server without standing up a Spark cluster.
# Process a 50 GB CSV on a 16 GB RAM machine
result = (
pl.scan_csv("/data/shipments_2025.csv")
.filter(pl.col("carrier") == "Delhivery")
.group_by("destination_state")
.agg(pl.col("delivery_days").mean())
.collect(streaming=True)
)Reading Parquet from S3. Polars integrates with fsspec and s3fs for reading directly from S3 buckets. Combined with predicate pushdown, it reads only the row groups matching your filter — meaning a query touching 10% of a dataset reads roughly 10% of the bytes from S3, directly reducing transfer costs on AWS Mumbai region.
df = pl.scan_parquet(
"s3://my-bucket/orders/year=2025/**/*.parquet",
storage_options={"region": "ap-south-1"},
).filter(pl.col("state") == "Tamil Nadu").collect()FastAPI data APIs. A common pattern for Indian SaaS companies is exposing analytics through a FastAPI endpoint. Polars' thread safety and its ability to process queries without holding the GIL make it a natural fit. A typical endpoint reads a Parquet file from S3, runs a groupby, and returns JSON — completing in under 200 ms on 50 million rows versus 2+ seconds with pandas, which matters when customers are waiting on a dashboard to load.
Frequently Asked Questions
Can Polars replace pandas completely in a data science workflow?
Not yet for most teams. The blocker is ecosystem depth. Scikit-learn, seaborn, and most AutoML libraries expect a pandas DataFrame. Polars can convert with .to_pandas(), but that conversion costs time and negates some gains on smaller datasets. For pure data wrangling and ETL pipelines where you control the full stack, Polars is a genuine replacement. For exploratory analysis in Jupyter paired with sklearn models, pandas is still the pragmatic choice in 2026.
Is Polars stable enough for production in 2026?
Yes. Polars crossed the 1.0 milestone in mid-2024 and committed to API stability from that point. The 1.x series has held that promise — breaking changes require a major version bump. Indian companies in fintech and quick-commerce have been running Polars in production data pipelines since late 2024. The main production risk is not stability but hiring: pandas knowledge is far more common among Indian data engineers, so factor in onboarding time when evaluating the switch.
How do I migrate a pandas script to Polars without rewriting everything?
Start with a bridge pattern rather than a full rewrite. Identify your three or four most expensive pandas operations using %timeit in Jupyter, then replace just those with Polars, converting back to pandas with .to_pandas() at the boundary. This hybrid approach typically captures 70–80% of the performance gains with about 20% of the rewrite effort. The trickiest parts to migrate are custom .apply() functions (Polars uses .map_elements() instead) and any code that depends on the pandas Index object, which Polars does not have.