Effective Pandas I Matt Harrison I PyData Salt Lake City Meetup
Why Stack Overflow recommends it
Stack Overflow users link it when answering questions about pandas method chaining, especially how to capture or reference intermediate DataFrames mid-chain with pipe.
Recommended by 3 people across 3 different months, 2021 to 2024.
What it is
A 90-minute PyData Salt Lake City meetup talk in which Matt Harrison, author of several pandas books, shows how to write cleaner and more efficient pandas code. It covers dtypes and memory usage, method chaining, pipe, avoiding mutation and apply, and aggregation.
What people said
Recommends haneulkim · 2021-09 · Stack Overflow · on “How to optimize data preprocessing using pandas assign method”
Using Pandas 1.1.4 From amazing talk by Matt harrison [this video] I've learned to preprocess dataframe by chaining methods which seems very easy to read for me. While trying to use chaining I've came across some lines that seems redundant. Here is MRE: df = pd.DataFrame({"st_date":[20210915, 20210914, 20211014, 20211010, 20211014], "language":["python", "python", "python", "java", "java"], "income":[10000, 12000, 11000, 9000, 8000], "next_raise":[2000,4000,3000,2000,1000], "A":[1,2,3,4,5], "B":[3,4,6,7,1]}) def is_weekday(x): """1 = weekday""" if x <= 4: return 1 else: return 0 Usually I would preprocess by following: df["st_date"] = pd.to_datetime(df["st_date"], format='%Y%m%d') df["dayofweek"]
Recommends markur · 2022-09 · Stack Overflow · on “Reference the Current State of a DataFrame during Method Chaining”
This video ( effective pandas [this video] ) changed my world of pandas. Your solution is at minute 49:10. You can capture a dataframe's state mid-chain by piping in a helper function that defines a new variable name into the global namespace: import pandas as pd import seaborn as sns def catchstate(df, var_name: str) -> 'pd.DataFrame': """ Helper function that captures intermediate Dataframes mid-chain. In the global namespace, make a new variable called var_name and set it to dataframe """ globals()[var_name] = df return df # LOAD TESTDATA df = sns.load_dataset("fmri") df = (df .rename(columns={"timepoint": "TIMEPOINT"}) .pipe(catchstate, var_name="DF2") .where(df["timepoint"] > 15, "foo") .ass
Recommends mouwsy · 2024-09 · Stack Overflow · on “copy a dataframe to new variable with method chaining”
Actually, this video [this video] and (and also that video [LINK] ; same approach for polars) describes what I was looking for. You can check the links, the idea is from Matt Harrison (who wrote multiple books about pandas) for debugging of method chains. This approach is also recommended in this great article 4 Pandas Anti-Patterns to Avoid and How to Fix Them [LINK] by Aidan Cooper. import pandas as pd def to_df(df, name): globals()[name] = df.copy() return df df = (pd.DataFrame([[1, 2, 3], [10, 10, 10], ], columns=["A", "B", "C"] ) .set_index("C") .pipe(to_df, "df_imag") .sum() ) df_imag is then the intermediate dataframe as described in the question. Another approach, which however only works