A Coding Guide to Scaling Advanced Pandas Workflows with Modin

In this tutorial, we delve into Modin, a powerful drop-in replacement for Pandas that leverages parallel computing to speed up data workflows significantly. By importing modin.pandas as pd, we transform our pandas code into a distributed computation powerhouse. Our goal here is to understand how Modin performs across real-world data operations, such as groupby, joins, cleaning, and time series analysis, all while running on Google Colab. We benchmark each task against the standard Pandas library to see how much faster and more memory-efficient Modin can be.

!pip install "modin[ray]" -q
import warnings
warnings.filterwarnings('ignore')


import numpy as np
import pandas as pd
import time
import os
from typing import Dict, Any


import modin.pandas as mpd
import ray


ray.init(ignore_reinit_error=True, num_cpus=2)  
print(f"Ray initialized with {ray.cluster_resources()}")

We begin by installing Modin with the Ray backend, which enables parallelized pandas operations seamlessly in Google Colab. We suppress unnecessary warnings to keep the output clean and clear. Then, we import all necessary libraries and initialize Ray with 2 CPUs, preparing our environment for distributed DataFrame processing.

def benchmark_operation(pandas_func, modin_func, data, operation_name: str) -> Dict[str, Any]:
    """Compare pandas vs modin performance"""
   
    start_time = time.time()
    pandas_result = pandas_func(data['pandas'])
    pandas_time = time.time() - start_time
   
    start_time = time.time()
    modin_result = modin_func(data['modin'])
    modin_time = time.time() - start_time
   
    speedup = pandas_time / modin_time if modin_time > 0 else float('inf')
   
    print(f"\n{operation_name}:")
    print(f"  Pandas: {pandas_time:.3f}s")
    print(f"  Modin:  {modin_time:.3f}s")
    print(f"  Speedup: {speedup:.2f}x")
   
    return {
        'operation': operation_name,
        'pandas_time': pandas_time,
        'modin_time': modin_time,
        'speedup': speedup
    }

We define a benchmark_operation function to compare the execution time of a specific task using both pandas and Modin. By running each operation and recording its duration, we calculate the speedup Modin offers. This provides us with a clear and measurable way to evaluate performance gains for each operation we test.

def create_large_dataset(rows: int = 1_000_000):
    """Generate synthetic dataset for testing"""
    np.random.seed(42)
   
    data = {
        'customer_id': np.random.randint(1, 50000, rows),
        'transaction_amount': np.random.exponential(50, rows),
        'category': np.random.choice(['Electronics', 'Clothing', 'Food', 'Books', 'Sports'], rows),
        'region': np.random.choice(['North', 'South', 'East', 'West'], rows),
        'date': pd.date_range('2020-01-01', periods=rows, freq='H'),
        'is_weekend': np.random.choice([True, False], rows, p=[0.3, 0.7]),
        'rating': np.random.uniform(1, 5, rows),
        'quantity': np.random.poisson(3, rows) + 1,
        'discount_rate': np.random.beta(2, 5, rows),
        'age_group': np.random.choice(['18-25', '26-35', '36-45', '46-55', '55+'], rows)
    }
   
    pandas_df = pd.DataFrame(data)
    modin_df = mpd.DataFrame(data)
   
    print(f"Dataset created: {rows:,} rows × {len(data)} columns")
    print(f"Memory usage: {pandas_df.memory_usage(deep=True).sum() / 1024**2:.1f} MB")
   
    return {'pandas': pandas_df, 'modin': modin_df}


dataset = create_large_dataset(500_000)  


print("\n" + "="*60)
print("ADVANCED MODIN OPERATIONS BENCHMARK")
print("="*60)

We define a create_large_dataset function to generate a rich synthetic dataset with 500,000 rows that mimics real-world transactional data, including customer info, purchase patterns, and timestamps. We create both pandas and Modin versions of this dataset so we can benchmark them side by side. After generating the data, we display its dimensions and memory footprint, setting the stage for advanced Modin operations.

def complex_groupby(df):
    return df.groupby(['category', 'region']).agg({
        'transaction_amount': ['sum', 'mean', 'std', 'count'],
        'rating': ['mean', 'min', 'max'],
        'quantity': 'sum'
    }).round(2)


groupby_results = benchmark_operation(
    complex_groupby, complex_groupby, dataset, "Complex GroupBy Aggregation"
)

We define a complex_groupby function to perform multi-level groupby operations on the dataset by grouping it by category and region. We then aggregate multiple columns using functions like sum, mean, standard deviation, and count. Finally, we benchmark this operation on both pandas and Modin to measure how much faster Modin executes such heavy groupby aggregations.

def advanced_cleaning(df):
    df_clean = df.copy()
   
    Q1 = df_clean['transaction_amount'].quantile(0.25)
    Q3 = df_clean['transaction_amount'].quantile(0.75)
    IQR = Q3 - Q1
    df_clean = df_clean[
        (df_clean['transaction_amount'] >= Q1 - 1.5 * IQR) &
        (df_clean['transaction_amount'] <= Q3 + 1.5 * IQR)
    ]
   
    df_clean['transaction_score'] = (
        df_clean['transaction_amount'] * df_clean['rating'] * df_clean['quantity']
    )
    df_clean['is_high_value'] = df_clean['transaction_amount'] > df_clean['transaction_amount'].median()
   
    return df_clean


cleaning_results = benchmark_operation(
    advanced_cleaning, advanced_cleaning, dataset, "Advanced Data Cleaning"
)

We define the advanced_cleaning function to simulate a real-world data preprocessing pipeline. First, we remove outliers using the IQR method to ensure cleaner insights. Then, we perform feature engineering by creating a new metric called transaction_score and labeling high-value transactions. Finally, we benchmark this cleaning logic using both pandas and Modin to see how they handle complex transformations on large datasets.

def time_series_analysis(df):
    df_ts = df.copy()
    df_ts = df_ts.set_index('date')
   
    daily_sum = df_ts.groupby(df_ts.index.date)['transaction_amount'].sum()
    daily_mean = df_ts.groupby(df_ts.index.date)['transaction_amount'].mean()
    daily_count = df_ts.groupby(df_ts.index.date)['transaction_amount'].count()
    daily_rating = df_ts.groupby(df_ts.index.date)['rating'].mean()
   
    daily_stats = type(df)({  
        'transaction_sum': daily_sum,
        'transaction_mean': daily_mean,
        'transaction_count': daily_count,
        'rating_mean': daily_rating
    })
   
    daily_stats['rolling_mean_7d'] = daily_stats['transaction_sum'].rolling(window=7).mean()
   
    return daily_stats


ts_results = benchmark_operation(
    time_series_analysis, time_series_analysis, dataset, "Time Series Analysis"
)

We define the time_series_analysis function to explore daily trends by resampling transaction data over time. We set the date column as the index, compute daily aggregations like sum, mean, count, and average rating, and compile them into a new DataFrame. To capture longer-term patterns, we also add a 7-day rolling average. Finally, we benchmark this time series pipeline with both pandas and Modin to compare their efficiency on temporal data.

def create_lookup_data():
    """Create lookup tables for joins"""
    categories_data = {
        'category': ['Electronics', 'Clothing', 'Food', 'Books', 'Sports'],
        'commission_rate': [0.15, 0.20, 0.10, 0.12, 0.18],
        'target_audience': ['Tech Enthusiasts', 'Fashion Forward', 'Food Lovers', 'Readers', 'Athletes']
    }
   
    regions_data = {
        'region': ['North', 'South', 'East', 'West'],
        'tax_rate': [0.08, 0.06, 0.09, 0.07],
        'shipping_cost': [5.99, 4.99, 6.99, 5.49]
    }
   
    return {
        'pandas': {
            'categories': pd.DataFrame(categories_data),
            'regions': pd.DataFrame(regions_data)
        },
        'modin': {
            'categories': mpd.DataFrame(categories_data),
            'regions': mpd.DataFrame(regions_data)
        }
    }


lookup_data = create_lookup_data()

We define the create_lookup_data function to generate two reference tables: one for product categories and another for regions, each containing relevant metadata such as commission rates, tax rates, and shipping costs. We prepare these lookup tables in both pandas and Modin formats so we can later use them in join operations and benchmark their performance across both libraries.

def advanced_joins(df, lookup):
    result = df.merge(lookup['categories'], on='category', how='left')
    result = result.merge(lookup['regions'], on='region', how='left')
   
    result['commission_amount'] = result['transaction_amount'] * result['commission_rate']
    result['tax_amount'] = result['transaction_amount'] * result['tax_rate']
    result['total_cost'] = result['transaction_amount'] + result['tax_amount'] + result['shipping_cost']
   
    return result


join_results = benchmark_operation(
    lambda df: advanced_joins(df, lookup_data['pandas']),
    lambda df: advanced_joins(df, lookup_data['modin']),
    dataset,
    "Advanced Joins & Calculations"
)

We define the advanced_joins function to enrich our main dataset by merging it with category and region lookup tables. After performing the joins, we calculate additional fields, such as commission_amount, tax_amount, and total_cost, to simulate real-world financial calculations. Finally, we benchmark this entire join and computation pipeline using both pandas and Modin to evaluate how well Modin handles complex multi-step operations.

print("\n" + "="*60)
print("MEMORY EFFICIENCY COMPARISON")
print("="*60)


def get_memory_usage(df, name):
    """Get memory usage of dataframe"""
    if hasattr(df, '_to_pandas'):
        memory_mb = df.memory_usage(deep=True).sum() / 1024**2
    else:
        memory_mb = df.memory_usage(deep=True).sum() / 1024**2
   
    print(f"{name} memory usage: {memory_mb:.1f} MB")
    return memory_mb


pandas_memory = get_memory_usage(dataset['pandas'], "Pandas")
modin_memory = get_memory_usage(dataset['modin'], "Modin")

We now shift focus to memory usage and print a section header to highlight this comparison. In the get_memory_usage function, we calculate the memory footprint of both Pandas and Modin DataFrames using their internal memory_usage methods. We ensure compatibility with Modin by checking for the _to_pandas attribute. This helps us assess how efficiently Modin handles memory compared to pandas, especially with large datasets.

print("\n" + "="*60)
print("PERFORMANCE SUMMARY")
print("="*60)


results = [groupby_results, cleaning_results, ts_results, join_results]
avg_speedup = sum(r['speedup'] for r in results) / len(results)


print(f"\nAverage Speedup: {avg_speedup:.2f}x")
print(f"Best Operation: {max(results, key=lambda x: x['speedup'])['operation']} "
      f"({max(results, key=lambda x: x['speedup'])['speedup']:.2f}x)")


print("\nDetailed Results:")
for result in results:
    print(f"  {result['operation']}: {result['speedup']:.2f}x speedup")


print("\n" + "="*60)
print("MODIN BEST PRACTICES")
print("="*60)


best_practices = [
    "1. Use 'import modin.pandas as pd' to replace pandas completely",
    "2. Modin works best with operations on large datasets (>100MB)",
    "3. Ray backend is most stable; Dask for distributed clusters",
    "4. Some pandas functions may fall back to pandas automatically",
    "5. Use .to_pandas() to convert Modin DataFrame to pandas when needed",
    "6. Profile your specific workload - speedup varies by operation type",
    "7. Modin excels at: groupby, join, apply, and large data I/O operations"
]


for tip in best_practices:
    print(tip)


ray.shutdown()
print("\n✅ Tutorial completed successfully!")
print("🚀 Modin is now ready to scale your pandas workflows!")

We conclude our tutorial by summarizing the performance benchmarks across all tested operations, calculating the average speedup that Modin achieved over pandas. We also highlight the best-performing operation, providing a clear view of where Modin excels most. Then, we share a set of best practices for using Modin effectively, including tips on compatibility, performance profiling, and conversion between pandas and Modin. Finally, we shut down Ray.

In conclusion, we’ve seen firsthand how Modin can supercharge our pandas workflows with minimal changes to our code. Whether it’s complex aggregations, time series analysis, or memory-intensive joins, Modin delivers scalable performance for everyday tasks, particularly on platforms like Google Colab. With the power of Ray under the hood and near-complete pandas API compatibility, Modin makes it effortless to work with larger datasets.

Check out the Codes. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter, and Youtube and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter.

Nikhil is an intern consultant at Marktechpost. He is pursuing an integrated dual degree in Materials at the Indian Institute of Technology, Kharagpur. Nikhil is an AI/ML enthusiast who is always researching applications in fields like biomaterials and biomedical science. With a strong background in Material Science, he is exploring new advancements and creating opportunities to contribute.

Source link