3.2.7. Common tasks


Yes, this page is kind of long. But that’s because it has a lot of useful info!

Use the page’s table of contents to the right to jump to what you’re looking for. Reshaping data

In the shape of data page, I explained the concept of wide vs. tall data with this example:

import pandas as pd

df = (pd.Series({   ('Ford',2000):10,
Firm Year Sales
0 Ford 2000 10
1 Ford 2001 12
2 Ford 2002 14
3 Ford 2003 16
4 GM 2000 11
5 GM 2001 13
6 GM 2002 13
7 GM 2003 15


To reshape dataframes, you have to work with index and column names.

So before we use stack and unstack here, put the firm and year into the index.

tall = df.set_index(['Firm','Year']) To convert a tall dataframe to wide: df.unstack().

If your index has multiple levels, the level parameter is used to pick which to unstack. “0” is the innermost level of the index.

print("\n\nUnstack (make it shorter+wider) on level 0/Firm:\n") 
print("\n\nUnstack (make it shorter+wider) on level 1/Year:\n") 
Unstack (make it shorter+wider) on level 0/Firm:
Firm Ford GM
2000 10 11
2001 12 13
2002 14 13
2003 16 15
Unstack (make it shorter+wider) on level 1/Year:
Year 2000 2001 2002 2003
Ford 10 12 14 16
GM 11 13 13 15 To convert a wide dataframe to tall/long: df.stack().


Pay attention after reshaping to the order of your index variables and how they are sorted.

# save the wide df above to this name for subseq examples
wide_year = tall.unstack(level=0) 

print("\n\nStack it back (make it tall): wide_year.stack()\n") 
print("\n\nYear-then-firm doesn't make much sense.\nReorder to firm-year: wide_year.stack().swaplevel()") 
print("\n\nYear-then-firm sorting make much sense.\nSort to firm-year: wide_year.stack().swaplevel().sort_index()") 
Stack it back (make it tall): wide_year.stack()
Year Firm
2000 Ford 10
GM 11
2001 Ford 12
GM 13
2002 Ford 14
GM 13
2003 Ford 16
GM 15
Year-then-firm doesn't make much sense.
Reorder to firm-year: wide_year.stack().swaplevel()
Firm Year
Ford 2000 10
GM 2000 11
Ford 2001 12
GM 2001 13
Ford 2002 14
GM 2002 13
Ford 2003 16
GM 2003 15
Year-then-firm sorting make much sense.
Sort to firm-year: wide_year.stack().swaplevel().sort_index()
Firm Year
Ford 2000 10
2001 12
2002 14
2003 16
GM 2000 11
2001 13
2002 13
2003 15

Beautiful! Lambda (in assign or after groupby)

You will see this inside pandas chains a lot: lambda x: someFunc(x), e.g.:

  • .assign(lev = lambda x: (x['dltt']+x['dlc'])/x['at']  )

  • .groupby('industry').assign(avglev = lambda x: x['lev'].mean()  )

Q1: What is that “lambda”?

A1: A lambda function is an anonymous function that is usually one line and usually defined without a name. You write it like this:

lambda <list the input(s)>: <expression that uses the inputs to make an output>

Here, you can see how the lambda function takes inputs and creates output the same way a function does:

dumb_prog = lambda a: a + 10     # I added "dumb_prog =" to name the lambda function and use it
# we could define a fnc to do the exact same thing
def dumb_prog(a):
    return a + 10

Q2: Why is that lambda there?

A2: We use lambdas when we need a function for a short period of time and when the name of the function doesn’t matter.

In the example above, [df].groupby('industry').assign(avglev = lambda x: x['lev'].mean()  ),

  1. groupby splits the dataframe into groups,

  2. then, within each group, it applies a function (here: the mean),

  3. and then returns a new dataframe with one observation for each group (the average leverage for the industry). Visually, this split-apply-combine1 process looks like this:

But notice! The .assign() portion is working on these tiny split up pieces of the dataframe created by df.groupby('industry'). Those pieces are dataframe objects that don’t have names!

So lambda functions let us refer to an unnamed dataframe object! When you type <some df object, like df.groupby()>.assign(newVar = lambda x: someFunc(x)), x is the object (“some df object”) that assign is working on. Ta da!

# common syntax within pandas
.assign(<newvarname> = lambda <tempnameforpriorobj>:  <do stuff to tempnameforpriorobj>   )       

# often, tempname is just "x" for short
.assign(<newvarname> = lambda x: <someFunc(x)> )      

# example:
.assign(lev = lambda x: (x['dltt']+x['dlc'])/x['at'] )


It turns out that lambda functions are very useful in python programming, and not just within pandas. For example, some functions take functions as inputs, like csnap(), map(), and filter(), and lambda functions let us give them custom functions quickly.

But pandas is where we will use lambda functions most in this class. .transform() after groupby

Sometimes you get a statistic for a group, but you want that statistic in every single row of your original dataset.

But groupby creates a new dataframe that is smaller, with only one row per row.

import pandas as pd 
import numpy as np
df = pd.DataFrame({'key':["A",'B','C',"A",'B','C'],

display(df) # the input
A 1
A 4
B 2
B 5
C 3
C 6
# groupby().sum() shrinks the dataset
       .to_frame() ) # just added this line bc df prints prettier than series
A 5
B 7
C 9
# groupby().transform(sum) does NOT shrink the dataset

A 5
A 5
B 7
B 7
C 9
C 9

One last trick: Let’s add that new variable to the original dataset!

# option 1: create the var
df['groupsum'] = df.groupby(level='key').transform(sum)

# option 2: create the var with assign (can be used inside chains)
df = df.assign(groupsum = df.groupby(level='key')['data'].transform(sum))

data groupsum
A 1 5
A 4 5
B 2 7
B 5 7
C 3 9
C 6 9 Using non-pandas functions inside chains

One problem with writing chains on dataframes is that you can only use methods that work on the object (a dataframe) that is getting chained.

So for example, you’ve formatted dataframe to plot. You can’t directly add a seaborn function to the chain: Seaborn functions are methods of the package seaborn, not the dataframe. (It’s sns.lmplot, not df.lmplot.)

.pipe() allows you to hand a dataframe to functions that don’t work directly on dataframes.

The syntax of .pipe()

df.pipe(<'outside function'>, 
        <'if the first parameter of the outside function isnt the df, '
         'the name of the parameter that is expecting the dataframe'>,
        <'any other parameters youd give the outside function'>

Note that the object after the pipe command is run might not be a dataframe anymore! It’s whatever object the piped function produces! Example 1

From one of the pandas devs:

jack_jill = pd.DataFrame()
(jack_jill.pipe(went_up, 'hill')
    .pipe(fetch, 'water')
    .pipe(fell_down, 'jack')
    .pipe(broke, 'crown')
    .pipe(tumble_after, 'jill')

This really is just right-to-left function execution. The first argument to pipe, a callable, is called with the DataFrame on the left as its first argument, and any additional arguments you specify.

I hope the analogy to data analysis code is clear. Code is read more often than it is written. When you or your coworkers or research partners have to go back in two months to update your script, having the story of raw data to results be told as clearly as possible will save you time. Example 2

From Steven Morse:

 .query('cut in ["Ideal", "Good"] & \
         clarity in ["IF", "SI2"] & \
         carat < 3')
 .pipe((sns.FacetGrid, 'data'),
        row='cut', col='clarity', hue='color',
 .map(sns.scatterplot, 'carat', 'price', alpha=0.8)
 .add_legend()) Printing inside of chains


One thing about chains, is that sometimes it’s hard to know what’s going on within them without just commenting out all the code and running it bit-by-bit.

This function, csnap (meaning “C”hain “SNAP”shot) will let you print messages from inside the chain, by exploiting the .pipe() function we just covered!

Copy this into your code:

def csnap(df, fn=lambda x: x.shape, msg=None):
    """ Custom Help function to print things in method chaining.    
        Will also print a message, which helps if you're printing a bunch of these, so that you know which csnap print happens at which point.
        Returns back the df to further use in chaining.
        Usage examples - within a chain of methods:
            df.pipe(csnap, lambda x: <do stuff>)
            df.pipe(csnap, msg="Shape here")
            df.pipe(csnap, lambda x: x.sample(10), msg="10 random obs")
    if msg:
    return df

An example of this in use:

 .pipe(csnap, msg="Shape before describe")
 .describe()['data']  # get the distribution stats of a variable (I'm just doing something to show csnap off)
 .pipe(csnap, msg="Shape after describe and pick one var") # see, it prints a message from within the chain!
 .assign(ones = 1)
 .pipe(csnap, lambda x: x.sample(2), msg="Random sample of df at point #3") # see, it prints a message from within the chain! 
Shape before describe
(6, 2)
Shape after describe and pick one var
Random sample of df at point #3
data ones
max 6.0 1
min 1.0 1
data ones twos threes
count 6.000000 1 2 3
mean 3.500000 1 2 3
std 1.870829 1 2 3
min 1.000000 1 2 3
25% 2.250000 1 2 3
50% 3.500000 1 2 3
75% 4.750000 1 2 3
max 6.000000 1 2 3 Prettier pandas output

A few random things:

  • Want to change the order of rows in an output table? .reindex()

  • Want to format the numbers shown by pandas?

    1. Permanent: Add this line of code to the top of your file: pd.set_option('display.float', '{:.2f}'.format)

    2. Temp: Add style.format to the end of your table command. E.g.: df.describe().style.format("{:.2f}")

  • Want to control the number of columns / rows pandas shows?

    1. pd.set_option('display.max_columns', 50)

    2. pd.set_option('display.max_rows', 50)

  • More formatting controls: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.set_option.html


(This figure is yet another resource I’m borrowing from the awesome PythonDataScienceHandbook.