Applying Custom Functions to Groupby Objects in Pandas

With the end of week three, I am already a quarter of the way through the General Assembly Data Science Immersive bootcamp. This week, the cohort again covered a combination of statistics (t-tests, chi-squared tests of independence, Cohen’s d, and more), as well as more pandas and SQL.

Over this past week, I encountered a tricky problem. A dataframe had a column named order_id, which contained repeated values (see left). The objective was to create a sub_id column, which indexed the line(s) within each order_id. I solved this problem by looping if-else statements, which checked that the order_id from the previous row matched the current row order_id. This worked, but was time-consuming to write.

Alternatively, the problem set solution involved a custom function, and was beautiful. The custom function is applied to a dataframe grouped by order_id. The function splits the grouped dataframe up by order_id. Working order_id group at a time, the function creates an array of sequential whole numbers from zero to the number of rows in each order_id, adds one to each element in the array, and finally fills the sub_id column with the array. This process is repeated…