MetaSnake

Your Corporate Training Shouldn't Be a Game of Telephone

2026-11-07T00:00:00-07:00

There are training companies that offer every course under the sun. What their clients might not realize is that those companies sometimes reach out to me to deliver the training.

You might end up with me as your instructor either way. But how we get there makes a difference.

I've taught through other providers, and it can be a decent experience. The frustrating part is getting an accurate picture of what the client needs before I walk into the classroom.

You probably know the game of telephone. Someone whispers a message to the next person in line. Each person passes it along. By the time the last person says it out loud, everyone is laughing at how much it has changed.

That's less funny when you've paid for training and taken your team away from their work.

Here's how it happens. A manager or learning director decides the team needs training. They discuss the requirements internally and send them to their training provider. The provider checks its catalog. If nothing fits, someone drafts an outline for a course that appears to cover the request.

Then they look for someone to teach it. Sometimes they go through another provider, who repeats the process before finding an instructor. By the time the request reaches me, I might have a course title, an outline, and a vague description of the audience. The people who could explain what the team actually needs are several conversations away.

A great instructor can deliver a great course from that outline. It still might be the wrong course for the team.

When you work directly with me, I can talk to your technical people. I can ask what they're doing, where they're struggling, and what they already know. We can work out what the training should cover together. If something is unclear, I can ask the person who knows the answer.

That conversation is part of preparing the course. It helps me choose the content, examples, and depth that will be useful to your team.

The providers sometimes tell me they're doing the hard work for me by managing the client relationship. I understand the appeal, but talking to the client is work I want to do. I care about delivering a great course, and those conversations give me a better chance of preparing one.

It's also much less frustrating than standing in front of a class and seeing confused faces because a requirement got mistranslated along the way. Finding that out in the classroom wastes everyone's time.

I understand why a company wants one provider for all its training. There's a catalog. There are boxes to check. Someone can say, "Yes, we offer that."

But who are those boxes serving? Does the arrangement help your colleagues get the training they need, or does it just make the purchasing process look easier?

If you want training in Python, data science, or AI, consider going directly to someone who teaches it. In my experience, you can often get a course that fits your team better and costs less. You may even get the same instructor the larger provider would have hired, with a chance to talk to them before the requirements have passed through five other people.

Before you book your next course, take five minutes to look for the people who actually deliver the training. Ask whether you can talk to the instructor about your team's needs.

If you'd like to have that conversation with me, get in touch.

Why MetaSnake Training

2026-08-24T00:00:00-06:00

There are many ways for a technical team to learn.

You can leave training to individual team members. You can purchase access to an on-demand platform. You can bring in an instructor for a workshop or customized course. Each option solves a different problem.

Self-study works well when one person needs to learn a narrow skill. On-demand courses provide flexibility and sacrifice depth and customization for some breadth. Live training becomes especially valuable when the problem is not just individual knowledge, but how the team works together.

Put ten Python developers on a team, and you may get ten different approaches to environments, dependency management, project structure, testing, code review, and AI-assisted development. Each approach may appear reasonable in isolation. Together, they create friction.

The right training should do more than teach another library or demonstrate a collection of features. It should help the team establish a shared vocabulary, a shared set of practices, and a more effective way to make technical decisions.

That is the problem MetaSnake training is designed to solve.

Training creates a neutral environment where the most senior and the newest team members can come together and learn to speak a common language. It is a rising tide that lifts all boats. Involving everyone in the training process creates a sense of unity and shared responsibility. Senior developers have the opportunity to compare assumptions and share implicit knowledge. This alignment is crucial for effective collaboration and maintaining code quality.

I (Matth Harrison) worked as a backend developer and data scientist and have seen the good, the bad, and the ugly of software development.

Because MetaSnake has experience across a wide range of industries and applications, we can cater our training to your specific needs. Rather than using canned examples, we can use your code and your problems to teach the principles and practices that will help your team be more effective.

Finally, perhaps the elephant in the room is AI. I've used AI for development since copying and pasting out of ChatGPT. I've trained teams that use different agents. I've used AI to write more code in the last year than I have in the last decade. The key is to know where to insert the human in the loop. I have a framework for doing that that I've taught to teams building the next generation of computers to some of the largest companies in the world.

Training should not leave participants with a folder full of slides and a few new commands. It should leave the team with stronger judgment, more consistent practices, and a workflow they can continue using after the course ends.

If your team is struggling with inconsistent Python practices, fragile data workflows, ineffective testing, or uncontrolled AI adoption, reach out. We can discuss where the friction is occurring and whether a talk, workshop, or customized course is the right way to address it.

Exaggerate to Calibrate

2026-08-13T00:00:00-06:00

I've long felt that part of being a well-rounded individual is balance. For me that often means moving my body. My work tends to involve a lot of sitting (or standing) in one place. To help with my "cabin fever," I like to get outside. Often that involves some sort of "mountain activity".

During winter months, skiing is my preferred activity. In the summer, I like to bike.

As an educator, I'm constantly thinking of ways to improve my teaching methods and engage my students. At some point, when you participate in an activity a lot, you notice things. Where am I weak? Where can I improve? How did they do that? Why can't I do that?

I've also read self-help books for improving athletic performance. Body mechanics (and awareness of said mechanics) is one of the most important aspects of improving performance. One should be aware of how their body moves. Generally, this is very hard unless you have a mirror, a video of yourself performing an action, or a coach to provide feedback. We tend to overestimate what our body is doing. Truthful awareness of real body mechanics can unlock a new skill.

I'll give an example in mountain biking. There is the notion of "pumping" the bike. When I started mountain biking in the late eighties, this skill (that existed in BMX) was not discussed. Now, we have "pump tracks", a small looped track with rollers and banked turns. Ideally, you should be able to ride around a pump track without pedaling - only pumping the bike.

The first time our high school mountain biking team went to a pump track, I thought "Cool, this will be easy." I proceeded to pedal (an illegal move on a pump track) and push and pull on the bike handlebars. It wasn't working. The unlock for me was pump with my legs. My attempts to pump with my arms (and pedalling) left most of my body static. I wasn't using my largest muscles to manipulate my body position and thus transfer energy to the bike. Once I realized that I needed to pump with my legs, I was able to ride the pump track without pedaling. It was a revelation.

Pump with your legs—or stand there looking deeply concerned.

That's not the whole story, though. Initially, I was barely pumping my legs. I was told to exaggerate the motion. Pump all the way down as far as you can, and then all the way up. Go past where it is comfortable. Exaggerate! By exaggerating the motion, I could physically "see" that my initial leg pump was too small. Only by exaggerating the motion could I see the full range of motion and understand how to use my legs effectively.

This same exaggeration technique can certainly be applied to other sports. I've used it in skiing.

I'm currently teaching my daughter how to throw a forehand and backhand in (ultimate) frisbee. She tends to throw with only the forearm and wrist. By having her exaggerate the motion of her entire arm and body, she can see how much more power and control she can generate. Then she can recalibrate her throw to use the full range of motion without overdoing it.

What does any of this have to do with technology? We are in a space where AI can now write most code for us. I've found that allows me to force the AI to apply all of the software engineering best-practices that I want. The geek in me rejoices! In the past, many of those best practices were often ignored or sacrificed for expediency.

Don't just let AI pull your body down the hill. Exaggerate the best practices and techniques that you want to see in your application. Then you can come back and recalibrate to determine what is actually needed.

If your team isn't aware of best practices, reach out. I think I can help.

Python for Finance: Pandas Resample, Groupby, and Rolling

2023-05-31T00:00:00-06:00

Republication note: I originally wrote this article for Ponder, where it was published on May 31, 2023. Ponder later removed the post following its acquisition, so I have preserved it here. The Ponder-specific code reflects the environment available when the article was written; the pandas concepts remain broadly applicable. View the archived original.

Exploring FDIC bank failures with pandas time-series tools.

Welcome back! Today we will explore how you can leverage pandas to understand trends in time-series data quickly, with a particular focus on time-series methods like .resample, grouping by dates, and performing .rolling operations to smooth out the results. Along the way, I might grumble a bit (I’d like to think that the grey strands in my beard are from wisdom and not fighting with a computer), but I intend to give you some superpowers.

If you work with time series data, you will want to understand .resample, grouping, and .rolling, as they will make your life much easier.

The code below preserves that original environment and demonstrates the pandas concepts as they appeared in the original post.

Let’s start by picking a dataset that might be relevant considering recent news… bank failures.

Data Overview: FDIC BankFind Suite

In recent years, the banking sector in the United States has witnessed many financial institution failures, leading to substantial economic losses and systemic risks. The Federal Deposit Insurance Corporation (FDIC) bank failure data provides valuable insights into the underlying causes of these failures and helps develop effective risk management and regulatory compliance strategies. By exploring this data with the aid of the popular data analysis library, pandas, one can identify patterns and trends, prepare data for modeling, and visualize findings effectively.

BankFind Suite is a comprehensive database provided by the FDIC, which allows users to search for current and former FDIC-insured banking institutions based on name, FDIC certificate number, website, or location. This tool offers detailed information on an institution’s financial history and trends, enabling users to track the performance of individual institutions, groups of institutions, or the industry as a whole.

Let’s use the Bank Failure data to explore. I will download the data from 1934 to 2023 containing over 4000 entries.

import json
import pandas as pd

>>> df = pd.read_csv('bankfail.csv')
>>> df
    CERT    CHCLASS1    CITYST          COST    FAILDATE    FIN ID  NAME                    QBFASSET    QBFDEP      RESTYPE RESTYPE1    SAVR
0   15426.0 NM      ALMENA, KS      16806.0 10/23/2020  10538   4104    ALMENA STATE BANK           65733.0     64941.0     FAILURE PA      DIF
1   16748.0 NM      FORT WALTON BEACH, FL   7247.0  10/16/2020  10537   4103    FIRST CITY BANK OF FLORIDA      136566.0    133936.0    FAILURE PA      DIF
2   14361.0 NM      BARBOURSVILLE, WV   45913.0 4/3/2020    10536   4102    THE FIRST STATE BANK            151808.0    143102.0    FAILURE PA      DIF
3   18265.0 NM      ERICSON, NE     25293.0 2/14/2020   10535   4101    ERICSON STATE BANK          100879.0    95159.0     FAILURE PA      DIF
4   21111.0 N       NEWARK, NJ      1946.0  11/1/2019   10534   4100    CITY NATIONAL BANK OF NEW JERSEY    120574.0    111234.0    FAILURE PA      DIF
... ... ...     ...         ... ...     ... ... ...                 ...     ...     ... ...     ...
4099    NaN NM      LEWISPORT, KY       NaN 8/6/1934    0   137 BANK OF LEWISPORT           81.0        68.0        FAILURE PO      BIF
4100    NaN N       LIMA, MT        NaN 7/18/1934   0   135 FIRST NATIONAL BANK OF LIMA     91.0        42.0        FAILURE PO      BIF
4101    NaN NM      FLORENCE, IN        NaN 7/18/1934   0   136 FLORENCE DEPOSIT BANK           105.0       69.0        FAILURE PO      BIF
4102    NaN NM      EAST PEORIA, IL     NaN 5/28/1934   0   133 FON DU LAC STATE BANK           374.0       238.0       FAILURE PO      BIF
4103    NaN NM      PITTSBURGH, PA      NaN 4/19/1934   0   134 BANK OF AMERICA TRUST CO.       1435.0      1064.0      FAILURE PO      BIF
4104 rows × 13 columns

It looks like my CSV export loaded ok. However, the existing column names are a little annoying. I’m going to clean them up.

Here’s some code to rename the columns, create a city and state column, and convert the date column to a datetime type.

>>> colnames = {'CERT': 'cert',
...   'CHCLASS1': 'charter_class',
...    'CITYST': 'location',
...    'COST': 'estimated_loss',
...    'FAILDATE': 'effective_date',
...    'FIN': 'fin',
...    'ID': 'id',
...    'NAME': 'institution',
...    'QBFASSET': 'total_assets',
...    'QBFDEP': 'total_deposits',
...    'RESTYPE': 'resolution',
...    'RESTYPE1': 'transaction_type',
...    'SAVR': 'insurance_fund'}

>>> def tweak_bank(df):
...   return (df
...           .assign(FAILDATE=pd.to_datetime(df.FAILDATE, format='%m/%d/%Y'),
...                   city=df.CITYST.str.split(',').str[0],
...                   state=df.CITYST.str.split(',').str[1],
...                  )
...           .drop(columns=['CITYST'])
...           .rename(columns=colnames)
...          )

>>> bank = tweak_bank(df)
>>> bank
    cert    charter_class   estimated_loss  effective_date  fin id  institution             total_assets    total_deposits  resolution  transaction_type    insurance_fund  city            state
0   15426.0 NM      16806.0     2020-10-23  10538   4104    ALMENA STATE BANK           65733.0     64941.0     FAILURE     PA          DIF     ALMENA          KS
1   16748.0 NM      7247.0      2020-10-16  10537   4103    FIRST CITY BANK OF FLORIDA      136566.0    133936.0    FAILURE     PA          DIF     FORT WALTON BEACH   FL
2   14361.0 NM      45913.0     2020-04-03  10536   4102    THE FIRST STATE BANK            151808.0    143102.0    FAILURE     PA          DIF     BARBOURSVILLE       WV
3   18265.0 NM      25293.0     2020-02-14  10535   4101    ERICSON STATE BANK          100879.0    95159.0     FAILURE     PA          DIF     ERICSON         NE
4   21111.0 N       1946.0      2019-11-01  10534   4100    CITY NATIONAL BANK OF NEW JERSEY    120574.0    111234.0    FAILURE     PA          DIF     NEWARK          NJ
... ... ...     ...     ...     ... ... ...                 ...     ...     ...     ...         ...     ...         ...
4099    NaN NM      NaN     1934-08-06  0   137 BANK OF LEWISPORT           81.0        68.0        FAILURE     PO          BIF     LEWISPORT       KY
4100    NaN N       NaN     1934-07-18  0   135 FIRST NATIONAL BANK OF LIMA     91.0        42.0        FAILURE     PO          BIF     LIMA            MT
4101    NaN NM      NaN     1934-07-18  0   136 FLORENCE DEPOSIT BANK           105.0       69.0        FAILURE     PO          BIF     FLORENCE        IN
4102    NaN NM      NaN     1934-05-28  0   133 FON DU LAC STATE BANK           374.0       238.0       FAILURE     PO          BIF     EAST PEORIA     IL
4103    NaN NM      NaN     1934-04-19  0   134 BANK OF AMERICA TRUST CO.       1435.0      1064.0      FAILURE     PO          BIF     PITTSBURGH      PA
4104 rows × 14 columns

Not bad. This is going to be great data for learning .resample, .groupby, and .rolling.

Data Dictionary

Here is the description of the columns for those (like me) who aren’t subject matter experts. (All descriptions taken from here.)

cert – Certificate number assigned by FDIC to identify the institution
charter_class – The FDIC assigns classification codes indicating an institution’s charter type (commercial bank, savings bank, or savings association), its chartering agent (state or federal government), its Federal Reserve membership status (member or nonmember), and its primary federal regulator (state-chartered institutions are subject to both federal and state supervision). These codes are:
N National chartered commercial bank supervised by the Office of the Comptroller of the
SM State charter Fed member commercial bank supervised by the Federal Reserve
NM State charter Fed nonmember commercial bank supervised by the FDIC
SA State or federal charter savings association supervised by the Office of Thrift Supervision or Office of the Comptroller of the Currency
SB State charter savings bank supervised by the FDIC
estimated_loss – the difference between the amount disbursed from the Deposit Insurance Fund (DIF) to cover obligations to insured depositors and the amount estimated to be ultimately recovered from the liquidation of the receivership estate. Estimated losses reflect unpaid principal amounts deemed unrecoverable and do not reflect interest that may be due on the DIF’s administrative or subrogated claims should its principal be repaid in full.
effective_date – date of failure.
fin – Financial institute number. Another unique identifier.
id – Institution directory number
name – Legal name of the institution.
total_assets – The Total assets owned by the institution, including cash, loans, securities, bank premises, and other assets as of the last Call Report or Thrift Financial Report.
total_deposits – Total including demand deposits, money market deposits, other savings deposits, time deposits and deposits in foreign offices as of the last Call Report.
resolution – Failure stature. Failure or assistance by merging with other institution.
transaction_type – Resolution type. Cat 1 – Institution’s charter survives. Cat 2 – Charter terminated. Cat 3 – Payout:
A/A – Cat1. (Assistance Transactions). Assistance was provided to the acquirer, who purchased the entire institution.
REP – Cat1. Re-privatization, management takeover with or without assistance at takeover, followed by a sale with or without additional assistance.
P&A – Cat2. Purchase and Assumption, where some or all of the deposits, certain other liabilities, and a portion of the assets (sometimes all of the assets) were sold to an acquirer. It was not determined if all the deposits (PA) or only the insured deposits (PI) were assumed.
PA – Cat2. Purchase and Assumption, where the insured and uninsured deposits, certain other liabilities, and a portion of the assets were sold to an acquirer.
PI – Cat2. Purchase and Assumption of the insured deposits only, where the traditional P&A was modified so that the acquiring institution assumed only the insured deposits.
IDT – Cat2. Insured deposit Transfer, where the acquiring institution served as a paying agent for the insurer, established accounts on their books for depositors and often acquired some assets as well.
ABT – Cat2. Asset-backed transfer, an FSLIC transaction that is very similar to an IDT.
MGR – Cat2.An institution where FSLIC took over management and generally provided financial assistance. FSLIC closed down before the institution was sold. transactions
PO – Cat3 – Payout, where the insurer paid the depositors directly and placed the assets in a liquidating receivership.
insurance_fund – Insurance fund.

Counts of failures by year

Let’s use pandas to find the count of failures by year.

When I hear the words “by year,” I immediately think we need to group this data by year. This is like one of those word math problems that everyone hated in junior high until they understood how to convert the words to math. The same thing happens here. When your boss tells you they want “failures by year,” you should think, “I’m going to use the year column in the pandas .groupby method.”

In this case, we don't have a year column. But due to the magic of datetime columns, we can access the year through the .dt accessor.

>>> bank.effective_date
0      2020-10-23
1      2020-10-16
2      2020-04-03
3      2020-02-14
4      2019-11-01
          ...    
4099   1934-08-06
4100   1934-07-18
4101   1934-07-18
4102   1934-05-28
4103   1934-04-19
Name: effective_date, Length: 4104, dtype: datetime64[ns]

>>> (bank
...  .groupby(bank.effective_date.dt.year)
...  .size()
... )
effective_date
1934     9
1935    25
1936    69
1937    75
1938    74
        ..
2015     8
2016     5
2017     8
2019     4
2020     4
Length: 84, dtype: int64

This is ok. I would prefer a visualization to this table of data. Let’s try a bar chart.

(bank
 .groupby(bank.effective_date.dt.year)
 .size()
 .plot.bar()
)

This is a start, but the x-axis is not happy. Just be aware that if you intend on making bar plots with pandas, it converts the index to categories. In this case, the dates are converted to strings. This makes me sad, but not every library is perfect.

https://github.com/pandas-dev/pandas/issues/17001

Let’s try and make a line plot instead.

(bank
 .groupby(bank.effective_date.dt.year)
 .size()
 .plot()
)

This looks good. Line plots in pandas (unlike bar plots) respect dates in the index.

Now let’s jump into grouping functionality that is time-series specific.

Using Offset Aliases with Pandas Resample

Hidden away in the pandas documentation is a mention of offset aliases. (Sadly, these are not accessible from the docstrings inside of Jupyter, which is one lament I have with pandas otherwise excellent documentation).

You can find them here, or documented in my book, Effective Pandas.

In short, Y, Q, M, W, and D mean year, quarter, month, week, and day respectively. If you memorize these single-letter shortcuts, the .resample method, or the pd.Grouper class, you can quickly aggregate date information by different time intervals.

Let's try and view failures by month using the .resample method. Pandas resampling is powerful because it lets you convert time-series data with one time interval into time-series data with different time intervals. You can upsample so you have more rows over shorter intervals, or downsample so you have fewer rows over longer intervals. The key to using this method is sticking a date column into the index and then calling .resample instead of .groupby:

>>> (bank
...  .set_index('effective_date')
...  .resample('M')
...  .size()
... )
effective_date
1934-04-30    1
1934-05-31    1
1934-07-31    2
1934-08-31    1
1934-09-30    1
             ..
2019-10-31    2
2019-11-30    1
2020-02-29    1
2020-04-30    1
2020-10-31    2
Length: 540, dtype: int64

Notice the index in the resulting Series. Each entry ends on the last day of the month. This is because we passed the M (month) offset alias into the .resample method.

I generally plot this by chaining on a call to .plot. This will plot the date on the x-axis and draw the values on the y-axis.

(bank
 .set_index('effective_date')
 .resample('M')
 .size()
 .plot()
)

Smoothing By Rougher Aggregations Using Pandas Resample

Let’s try something a little fancier. We will make a function that plots the monthly failures (which we already saw was bumpy) and then plot the yearly aggregation on top.

I'll use the .pipe method to refactor the plotting logic into a single function.

>>> import matplotlib.pyplot as plt
>>> fig, ax = plt.subplots(figsize=(10,4))
>>> def plot_monthly_and_yearly(df, ax):
...     (df
...       .resample('M')
...       .size()
...       .plot(ax=ax, label='Monthly')
...     )
...     (df
...       .resample('Y')
...       .size()
...       .plot(ax=ax, label='Annual')
...     )
...     ax.legend()
...     return df

>>> (bank
...  .set_index('effective_date')
...  .pipe(plot_monthly_and_yearly, ax=ax)
... )
        cert    charter_class   estimated_loss  fin id  institution             total_assets    total_deposits  resolution  transaction_type    insurance_fund  city            state
effective_date                                                  
2020-10-23  15426.0 NM      16806.0     10538   4104    ALMENA STATE BANK           65733.0     64941       FAILURE     PA          DIF     ALMENA          KS
2020-10-16  16748.0 NM      7247.0      10537   4103    FIRST CITY BANK OF FLORIDA      136566.0    133936      FAILURE     PA          DIF     FORT WALTON BEACH   FL
2020-04-03  14361.0 NM      45913.0     10536   4102    THE FIRST STATE BANK            151808.0    143102      FAILURE     PA          DIF     BARBOURSVILLE       WV
2020-02-14  18265.0 NM      25293.0     10535   4101    ERICSON STATE BANK          100879.0    95159       FAILURE     PA          DIF     ERICSON         NE
2019-11-01  21111.0 N       1946.0      10534   4100    CITY NATIONAL BANK OF NEW JERSEY    120574.0    111234      FAILURE     PA          DIF     NEWARK          NJ
...     ... ...     ...     ... ... ...                 ...     ...     ...     ...         ...     ...         ...
1934-08-06  NaN NM      NaN     0   137 BANK OF LEWISPORT           81.0        68      FAILURE     PO          BIF     LEWISPORT       KY
1934-07-18  NaN N       NaN     0   135 FIRST NATIONAL BANK OF LIMA     91.0        42      FAILURE     PO          BIF     LIMA            MT
1934-07-18  NaN NM      NaN     0   136 FLORENCE DEPOSIT BANK           105.0       69      FAILURE     PO          BIF     FLORENCE        IN
1934-05-28  NaN NM      NaN     0   133 FON DU LAC STATE BANK           374.0       238     FAILURE     PO          BIF     EAST PEORIA     IL
1934-04-19  NaN NM      NaN     0   134 BANK OF AMERICA TRUST CO.       1435.0      1064        FAILURE     PO          BIF     PITTSBURGH      PA
4104 rows x 13 columns

Resolution Type Over Time

I want to talk about one more feature of pandas: the pd.Grouper class. I use this class to group by different frequencies of a date column without having to stick the date in the index and call .resample. You might ask, why?

The .resample method only allows us to have a single grouping. If you have played around with .groupby, then you know that you can provide multiple columns or series to .groupby. In this example, let's assume this word problem:

What are the counts of the different resolution types by month?

This is a little tricky. We have “by month,” so we want to group by month, but we also really have “size by resolution type by month.”

A first stab might look like this:

>>> (bank
...  .groupby([bank.effective_date.dt.month, 'resolution'])
...  .size()
... )
effective_date  resolution
1               ASSISTANCE     22
                FAILURE       274
2               ASSISTANCE     26
                FAILURE       302
3               ASSISTANCE     32
                FAILURE       443
4               ASSISTANCE     87
                FAILURE       328
5               ASSISTANCE     30
                FAILURE       283
6               ASSISTANCE     28
                FAILURE       298
7               ASSISTANCE     48
                FAILURE       359
8               ASSISTANCE     62
                FAILURE       280
9               ASSISTANCE     57
                FAILURE       219
10              ASSISTANCE     47
                FAILURE       284
11              ASSISTANCE     30
                FAILURE       212
12              ASSISTANCE    110
                FAILURE       243
dtype: int64

This sort of works if we want to know how many banks failed in December. But we want monthly data over time. Because we also want to group by resolution, we can't use the .resample method. This is where pd.Grouper comes in.

We provide the name of a datetime column and a frequency (this is that offset alias that we talked about above). The code looks like this:

>>> (bank
...  .groupby([pd.Grouper(key='effective_date', freq='M'),
...            'resolution'])
...  .size()
... )
effective_date  resolution
1934-04-30      FAILURE       1
1934-05-31      FAILURE       1
1934-07-31      FAILURE       2
1934-08-31      FAILURE       1
1934-09-30      FAILURE       1
                             ..
2019-10-31      FAILURE       2
2019-11-30      FAILURE       1
2020-02-29      FAILURE       1
2020-04-30      FAILURE       1
2020-10-31      FAILURE       2
Length: 635, dtype: int64

By now, you know that I want to plot this. However, tacking on .plot is unsatisfying as we have a hierarchical index and this sticks both date and resolution as a tuple on the x-axis:

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'resolution'])
 .size()
 .plot()
)

What I want to do instead is unstack the innermost index, the resolution index. The .unstack method will pull out the inner index by default and stick it up into the columns. Now we have monthly dates in the index and a column for each resolution type.

>>> (bank
...  .groupby([pd.Grouper(key='effective_date', freq='M'),
...            'resolution'])
...  .size()
...  .unstack(level=1)
... )
        __reduced___ASSISTANCE  __reduced___FAILURE
effective_date      
1934-04-30  NaN         1
1934-05-31  NaN         1
1934-07-31  NaN         2
1934-08-31  NaN         1
1934-09-30  NaN         1
...     ...         ...
2019-10-31  NaN         2
2019-11-30  NaN         1
2020-02-29  NaN         1
2020-04-30  NaN         1
2020-10-31  NaN         2
540 rows x 2 columns

If we plot this, we will get a line for each resolution type over time. It looks like the FAILURE resolution is much more common since 1989.

def fix_cols(name):
    return name.replace('__reduced___', '')

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'resolution'])
 .size()
 .unstack(level=1)
 .fillna(0)
 .rename(columns=fix_cols)
 .plot()
)

Let’s do a 12-month rolling average to smooth this out a bit.

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'resolution'])
 .size()
 .unstack(level=1)
 .rename(columns=fix_cols)
 .fillna(0)
 .rolling(12)
 .mean()
 .plot()
)

Below I plot the yearly resolution size. Again, you can see that the basic shape is the same, but the yearly rolling average tells a better story.

(bank
 .groupby([pd.Grouper(key='effective_date', freq='Y'),
           'resolution'])
 .size()
 .unstack(level=1)
 .rename(columns=fix_cols)
 .fillna(0)
 .plot()
)

And finally, let’s look at the 12-month rolling average by resolution type.

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'transaction_type'])
 .size()
 .unstack(level=1)
 .rename(columns=fix_cols)
 .fillna(0)
 .rolling(12)
 .mean()
 .plot()
)

Now You Have Pandas Resample, Groupby, and Rolling Superpowers

I hope that this blog post exposed you to the powerful time-series functionality in pandas, particularly .resample, .groupby, and .rolling.

Folks often ask me why I use pandas to manipulate my data. Could I do this in SQL? Yes… probably. But I’ve written enough SQL in my life to know that it wouldn’t be fun for me.

I Wrote A Book

2023-03-27T00:00:00-06:00

TLDR - I made over $25,000 in the two-day launch of my new self-published book, Effective XGBoost. I discuss how that happened and if writing a book is worth the time and effort.

My new Effective XGBoost book walks through end-to-end classification from model creation to tuning to interpretation, to deployment.

Why Write a Book?

Every potential author has to answer this for themself. I have written and worked as a technical editor for many books (Illustrated Guide to Python 3, Intermediate Python, Effective Pandas, Machine Learning Pocket Reference, Learning the Pandas Library, Pandas 1.x Cookbook, Tiny Python Notebook (3.5-3.10), Effective Pycharm, Hands-on Machine Learning with Scikit-Learn, Keras & Tensorflow, Hands-on Unsupervised Learning Using Python, and maybe a few others), so I'm aware of what is involved in bringing a book from an idea to something you can hold in your hand. I also know and have interviewed over a dozen technical authors and know many other Python and data science authors. The general advice that you would make more per hour consulting is true.

Some of the books I have written.

Many folks write because they have the bug. Others write for credibility. Others want a book with an animal on the cover. Since transitioning from development into mostly corporate training and consulting, authoring has been great marketing for me.

With self-publishing being relatively easy these days, the sole gatekeeper to writing a book is yourself. Be aware, book creation is a marathon. Don't jump in thinking it is a sprint.

You Made More in Two Days Then I Make in...

Effective XGBoost has been my best launch to date. After two days I had over $25,000 in sales. Perhaps this feels like bragging or an overnight success. But I've been writing books since 2011 and finally feel like I'm starting to crack the nut (or at least listen to common advice like setting up a mailing list).

At this point, I have a few thousand on my mailing list, 140k followers on Twitter, and around 20k connections on LinkedIn. I've fully succumbed to the reality that I must do the marketing myself, and I've worked pretty hard (particularly on the Twitter side) to grow my audience.

Also, note that I've been working on this book since the summer of 2022. Cranking out a quality book is hard work and takes a lot of time. Creating video courses is much easier and less time-consuming for me. Due to the nature of my work, I was often able to spend entire days working on the book. That was not the case when I had fulltime employment and wrote my first books. Many days I woke up early to crank out content. My back-of-the-napkin calculation is that I spent 10 hours a week over 25 weeks writing this book. That comes out to about $100/hour.

If you are now itching to write a book, note that my consulting fees are slightly higher than $100 per hour. Plus, I've authored multiple books, and this book performed much better than previous launches. For comparison, Effective Pandas, only did around $8,000 on its release.

Audience

Since the dawn of self-publishing, authors have touted the importance of a mailing list. I didn't have an email list when I created my first books. Bad choice, Matt!

I used the Gumroad platform initially, and while the features were great for sales, that was about the end of the features. Eventually, I moved on to Podia, which includes mailing list functionality. Honestly, my mailing list-fu is lacking, but I'm continually looking to improve. I might create a Substack, but as a one-man shop, I like to keep things simple and limit platforms. The draw of Podia is that it is all in one.

Launch sales.

I have a few thousand subscribers on my mailing list right now, and they accounted for 35% of my sales.

My next largest source of sales was from Twitter. (If you are wondering how I tracked this, I used Podia coupons with platform-specific codes.) I doubled down on Twitter during Covid and now have 140K followers. They accounted for around 32% of my sales. With all of the hijinks Elon has been pulling recently, having all my "eggs" in the Twitter basket is scary.

LinkedIn came in around 11%, but I only have 20K followers there. I'm actively working on growing that audience. Just another thing to put in the to-do list...

Around 15% of my sales aren't attributable or are from folks who didn't use a coupon. Many of my followers pay full price to support creators like me. I'm grateful to those "super fans". Some of these folks might not have figured out how to enter the coupon code at checkout. Podia can be tricky like that sometimes.

Around 5% of the sales during this time were upsells. So these weren't sales of Effective XGBoost but in-cart discounts to my Effective Pandas material. Podia can be tricky like that sometimes.

You'll note that I haven't mentioned the elephant in the book sales business ... Amazon.

Physical Books

My initial release was only for a digital book with an option for a course and consulting. I highly recommend creating an accompanying course for your book. Around 70% of my sales included the course. Here's the math. 100 books at $50 = $5,000. Contrast that with having a course, offering 70 courses at $100 and 30 books = $7,000 + $1,500 = $8,500.

Oh yeah, Amazon. Amazon accounts for a large number of sales. However, they also take a large cut. And they don't give me access to my readers.

The compromise I have settled on is letting Amazon (and other long-tail providers) handle the physical book. Many prefer to read from a physical book, and Amazon does a great job pushing your book for you, especially if you can get that fly-wheel moving. However, I feed my family with my book sales and would prefer not to have a middleman for my audience's digital sales.

I created both a paperback and hardback book (Amazon now offers these) and announced to my mailing list that these books were available without disclosing them to the rest of the world. I also deeply discounted it (to the point where I'm making $1/sale) as a bonus for my mailing list and a way to say "thanks" to my audience.

Amazon ranking after launch to mailing list.

This (hopefully) kicks in the Amazon machine, as my mailing list pushed the book to the top of the lists and a rank of around 3,000 overall on Amazon.

As of this post, I've bumped the price back up to normal. Amazon can take the cut of sales they bring in, but I will push my digital version and let readers know about the Amazon option if they prefer physical books.

Based on the performance of Effective Pandas (which has done well on Amazon), the Amazon machine can get close to Podia in terms of ongoing revenue.

Note that I'm doing a full-color book on Amazon. These are expensive to print. I could probably double my Amazon income if I keep the same price but sold only a grayscale interior. However, my book has dozens of charts and accompanying code, and the color print on this batch of Effective XGBoost has been excellent. (I'm thinking that Amazon upgraded the POD printers, or I've been really lucky.)

Traditional Publishing?

What about traditional publishing? Are my readers getting an inferior product? My content is not prefect. I make tpyos, like the rest of the world. But editors are also not perfect and also miss typos as well. 😉

I feel like my process is decent, and having been through the process of two other publishers, I feel confident about the final outcome. I can "execute" my books via doctests or notebooks, so I know the code works. Grammarly helps with most typos these days.

I recruit dozens of volunteers from my mailing list review the drafts of the book. This turns out a quality product based on feedback and experience.

What about the other stuff? Layout, cover design, marketing? Yep, that is work. Consider using a publisher if you can't figure out how to do it professionally.

What about distribution? A publisher does provide more distribution than just Amazon. However, Amazon is the elephant here. And I've found that after any initial promotion, the onus is solely on me to promote my book. I use Ingram Spark to reach the long tail, but the squeeze is a lot of effort for the juice when I could be doing other things.

AI and the Death of Books

When I was about 90% done with the book, ChatGPT was released. As a technologist, I jumped on this boat and was really happy using it to augment where possible. As a one-man shop, Chat (as I like to call it) is a game-changer.

I won't go deep into how I used Chat here (perhaps in a different post). I will say, Chat's hallucinations are something to be aware of. Did I use Chat content in the book? Yes, however, I edited everything that Chat created.

Will I use Chat in future content creation? Yes.

Do I think Chat means the end of books? No. I don't feel like Chat can completely automate book creation now. I also think there is some creative direction that Chat can't create on its own.

Does that mean that books won't be written by Chat? No. They already are. This isn't particularly disturbing to me because I've seen folks creating garbage books since Amazon opened the floodgates of self-publishing. You can even pick out these scam books on the charts. Shady folks sell shady stuff to make a buck and then use bots to help game the system. Alas, this is the world we live in. I feel like my audience trusts me to provide a quality product, and the reviews confirm that.

Will Writing A Book Help You?

Thanks for getting this far.

Should you write a book? I get this question a lot.

As a corporate trainer, getting a foot in the door is much easier if you have "written the book on the subject". I know a lot of trainers, and many of them have written books... I wonder why that is? 😉

For more software or data-type folks, should you write a book? Again, if you want to be able to show expertise, this is one (albeit a long one) road to accomplish that task. Having a portfolio project and being able to discuss it in detail would probably accomplish the same for most folks but take much less time.

If you have a large platform or means to promote the book, it can be profitable. Note that this is a long tail game, and the news that you read about is typically the winners with outlier results.

Many just have a "book inside of them" that needs to get out. Just realize that getting it out is a marathon and like most things tech, will probably take two to three times as long as you think it will.

Let me know if you have any questions. I'm at @__mharrison__.

(Chat did not write this post.)

Professional Pandas: Indexing with Pandas Loc

2023-01-13T00:00:00-07:00

Republication note: I originally wrote this article for Ponder, where it was published on January 13, 2023. Ponder later removed the post following its acquisition, so I have preserved it here. The Ponder- and Modin-specific code reflects the environment available when the article was written; the pandas concepts remain broadly applicable. View the archived original.

Selecting dataframe rows and columns precisely with .loc.

Introduction to Pandas Loc

In this post, I describe how Pandas loc (.loc) can be used for advanced indexing. We start by reviewing basic indexing and slicing in Pandas. We then dive into .loc, and explore how this indexing attribute can be used to select and filter data in your DataFrames. Along the way, I provide simple and complex examples to help you understand how Pandas loc works, and discuss common gotchas to watch out for. By the end, you'll have a solid understanding of how to use Pandas loc to access data. I will cover Pandas iloc (.iloc) in a future article.

This is the third piece in our Professional Pandas series on teaching best practices about writing professional-grade Pandas code. If you have questions or topics that you would like to dive into, please reach out on Twitter to @ponderdata or @__mharrison__, and if you’d like to see the notebook version of this blogpost, you can find it on Github here.

The Data for Our Examples

For our examples, we will use Los Angeles crime data.

See below for descriptions of the relevant columns, taken directly from the data producer’s data dictionary:

DR_NO (Plain Text) Division of Records Number: Official file number made up of a 2 digit year, area ID, and 5 digits
Date Rptd (Date & Time) - MM/DD/YYYY
DATE OCC (Date & Time) - MM/DD/YYYY
TIME OCC (Plain Text) - In 24 hour military time.
AREA NAME (Plain Text) - The 21 Geographic Areas or Patrol Divisions are also given a name designation that references a landmark or the surrounding community that it is responsible for. For example 77th Street Division is located at the intersection of South Broadway and 77th Street, serving neighborhoods in South Los Angeles.
Crm Cd (Plain Text) - Indicates the crime committed. (Same as Crime Code 1)
Crm Cd Desc (Plain Text) - Defines the Crime Code provided.
Mocodes (Plain Text) - Modus Operandi: Activities associated with the suspect in commission of the crime. See attached PDF for the list of MO Codes in numerical order.
Vict Age (Plain Text) - Two character numeric
Vict Sex (Plain Text) - F - Female M - Male X - Unknown
Vict Descent (Plain Text) - Descent Code: A - Other Asian B - Black C - Chinese D - Cambodian F - Filipino G - Guamanian H - Hispanic/Latin/Mexican I - American Indian/Alaskan Native J - Japanese K - Korean L - Laotian O - Other P - Pacific Islander S - Samoan U - Hawaiian V - Vietnamese W - White X - Unknown Z - Asian Indian
Premis Cd (Number) - The type of structure, vehicle, or location where the crime took place.
Premis Desc (Plain Text) - Defines the Premise Code provided.
Weapon Used Cd (Plain Text) - The type of weapon used in the crime.
Weapon Desc (Plain Text) - Defines the Weapon Used Code provided.
Status (Plain Text) - Status of the case. (IC is the default)
Status Desc (Plain Text) - Defines the Status Code provided.
Crm Cd 1 (Plain Text) - Indicates the crime committed. Crime Code 1 is the primary and most serious one. Crime Code 2, 3, and 4 are respectively less serious offenses. Lower crime class numbers are more serious.
LOCATION (Plain Text) - Street address of crime incident rounded to the nearest hundred block to maintain anonymity.
Cross Street (Plain Text) - Cross Street of rounded Address
LAT (Number) - Latitude
LON (Number) - Longtitude

Let’s load the data.

import pandas as pd
url = "https://data.lacity.org/api/views/2nrs-mtv8/rows.csv?accessType=DOWNLOAD"
df = pd.read_csv(url)

The Basics on Indexing and Slicing with Pandas

Before we dive into .loc, it's important to review some basic indexing and slicing in Pandas. In Pandas, a DataFrame is a two-dimensional size-mutable tabular data structure with rows and columns. You can think of it like a spreadsheet or an Excel sheet.

To access a specific column in a DataFrame, you can use square brackets (e.g. df[col]). This is called index access and will return a series in Pandas. Let's get the Crime Description column.

>>> df['Crm Cd Desc']
0   BATTERY - SIMPLE ASSAULT
1   BATTERY - SIMPLE ASSAULT
2   SEX OFFENDER REGISTRANT OUT OF COMPLIANCE
3   VANDALISM - MISDEAMEANOR ($399 OR UNDER)
4   VANDALISM - FELONY ($400 & OVER, ALL CHURCH VA...
    ...
632385  THEFT OF IDENTITY
632386  THEFT FROM MOTOR VEHICLE - GRAND ($950.01 AND ...
632387  VEHICLE - STOLEN
632388  VEHICLE - STOLEN
632389  THEFT FROM MOTOR VEHICLE - GRAND ($950.01 AND ...
Name: Crm Cd Desc, Length: 632390, dtype: object

Note that we can also access a single column by attribute access IF the column has a valid Python attribute name. Meaning that the column must start with an alphabetic character or underscore and the rest of the name must be alphanumerics or underscores. Because Crm Cd Desc has spaces, we will get an error when we try to access it:

>>> df.Crm Cd Desc
  File "", line 1
    df.Crm Cd Desc
           ^
SyntaxError: invalid syntax

However, we can access the AREA column this way:

>>> df.AREA
0          3
1          1
2          1
3         15
4         19
          ..
632385     3
632386    10
632387    10
632388    11
632389    16
Name: AREA, Length: 632390, dtype: int64

Note that I generally try to have attribute-friendly column names or update them if they are not. Most code editing environments for Python have better completion on attribute access, and for that reason I tend to use it.

We can also pull out multiple columns if we pass in a list of column names instead of a scalar. This returns a dataframe.

>>> df[['DATE OCC', 'AREA', 'Vict Age']]
        DATE OCC                  AREA    Vict Age
0       01/08/2020 12:00:00 AM    3       36
1       01/01/2020 12:00:00 AM    1       25
2       02/13/2020 12:00:00 AM    1       0
3       01/01/2020 12:00:00 AM    15      76
4       01/01/2020 12:00:00 AM    19      31
...     ...                       ...     ...
632385  08/02/2022 12:00:00 AM    3       34
632386  12/01/2022 12:00:00 AM    10      68
632387  02/09/2022 12:00:00 AM    10      0
632388  02/08/2022 12:00:00 AM    11      0
632389  02/14/2022 12:00:00 AM    16      61
632390 rows × 3 columns

What Is Pandas Loc, and What Inputs Does it Accept?

Pulling out a subset of columns is fine, but you might also want a subset of rows. Neither index access nor attribute access will give you that. However, the focus of this article, Pandas loc (and .iloc in future articles), allows you to pull out columns and rows.

Here is the key thing to remember about Pandas loc, and if you remember anything from this article, remember this: .loc allows you to pull out rows and columns by label name.

What even is .loc? It is a property. A property could be a post of its own, but in short, it allows you to call a method without parameters (or parentheses). When you say df.loc, pandas is actually running the .loc method and returning the result of it (which is a _LocIndexer).

How do I know all of this? Well, I looked at the source code for Pandas. (This is really easy in Jupyter, just type df.loc?? and run the code. Jupyter will pull up the source code when you put ?? after your code.)

It being a property is not particularly interesting. Perhaps the more interesting part is the docstring which describes how you can index off of the property.

What the Loc Documentation Says About Allowed Inputs

Allowed inputs are:

A single label, e.g. 5 or 'a', (note that 5 is interpreted as a label of the index, and never as an integer position along the index).

A list or array of labels, e.g. ['a', 'b', 'c'].

A slice object with labels, e.g. 'a':'f'.

warning:: Note that contrary to usual python slices, both the start and the stop are included

A boolean array of the same length as the axis being sliced, e.g. [True, False, True].

An alignable boolean Series. The index of the key will be aligned before masking.

An alignable Index. The Index of the returned selection will be the input.

A callable function with one argument (the calling Series or DataFrame) and that returns valid output for indexing (one of the above)

One more thing to be aware of that the docs don’t make clear. (This is also confusing because this is completely different from how indexing normally occurs in Python.) The inputs to the index include both a row indexer and an optional column indexer. You separate these by a comma. Keep that in mind through the following examples.

How to Use Pandas Loc with Each Input Type

Now I will walk you through how to use Pandas loc with each of the above input types, and will turn to the crime data as real-world example data. Note that this data has the default index, the RangeIndex, which is a monotonic increasing integer value.

Part 1: Using Pandas Loc to Index with Scalars

In this case, we will select the row with label 5. (Note that I highlighted label, because remember Pandas loc indexes by label and not by index! Even though in this example the index labels also happen to be the same as the index position.) The integer 5 inside the square brackets is used to select the row with label 5 in the DataFrame.

The syntax for indexing with .loc is df.loc[row_indexer, column_indexer]. In this case, row_indexer is the label 5, and column_indexer is not specified, so all columns are selected.

Here is the code:

>>> df.loc[5]
DR_NO       200100501
Date Rptd   01/02/2020 12:00:00 AM
DATE OCC    01/01/2020 12:00:00 AM
TIME OCC    30
AREA        1
AREA NAME   Central
Rpt Dist No 163
Part 1-2    1
Crm Cd      121
Crm Cd Desc RAPE, FORCIBLE
Mocodes     0413 1822 1262 1415
Vict Age    25
Vict Sex    F
Vict Descent    H
Premis Cd   735.0
Premis Desc NIGHT CLUB (OPEN EVENINGS ONLY)
Weapon Used Cd  500.0
Weapon Desc UNKNOWN WEAPON/OTHER WEAPON
Status      IC
Status Desc Invest Cont
Crm Cd 1    121.0
Crm Cd 2    998.0
Crm Cd 3    NaN
Crm Cd 4    NaN
LOCATION    700 S  BROADWAY
Cross Street    NaN
LAT     34.0452
LON     -118.2534
Name: 5, dtype: object

This might not look like a row. However, because we asked for a single index value, it gave us a row and used a Series to represent it.

One thing to be aware of is that if the row with label 5 does not exist in the DataFrame, this code will raise a KeyError. In this example, we try to pull out a row with the non-existent label of Crm Cd 1.

>>> df.loc['Crm Cd 1']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
 in
----> 1 df.loc['Crm Cd 1']
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in __getitem__(self, key)
    929
    930             maybe_callable = com.apply_if_callable(key, self.obj)
--> 931             return self._getitem_axis(maybe_callable, axis=axis)
    932
    933     def _is_scalar_access(self, key: tuple):
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1162         # fall thru to straight lookup
   1163         self._validate_key(key, axis)
-> 1164         return self._get_label(key, axis=axis)
   1165
   1166     def _get_slice_axis(self, slice_obj: slice, axis: int):
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in _get_label(self, label, axis)
   1111     def _get_label(self, label, axis: int):
   1112         # GH#5667 this will fail if the label is not present in the axis.
-> 1113         return self.obj.xs(label, axis=axis)
   1114
   1115     def _handle_lowerdim_multi_index_axis0(self, tup: tuple):
/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py in xs(self, key, axis, level, drop_level)
   3774                 raise TypeError(f"Expected label or tuple of labels, got {key}") from e
   3775         else:
-> 3776             loc = index.get_loc(key)
   3777
   3778             if isinstance(loc, np.ndarray):
/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/range.py in get_loc(self, key, method, tolerance)
    386                 except ValueError as err:
    387                     raise KeyError(key) from err
--> 388             raise KeyError(key)
    389         return super().get_loc(key, method=method, tolerance=tolerance)
    390
KeyError: 'Crm Cd 1'

In the next example we will select a single column. In this case, row_indexer is a slice with a start and end index of :, which selects all rows, and column_indexer is the label ‘Crm Cd 1’, which selects the column with that label.

Here is the full code:

>>> df.loc[:, 'Crm Cd 1']
0         624.0
1         624.0
2         845.0
3         745.0
4         740.0
          ...
632385    354.0
632386    331.0
632387    510.0
632388    510.0
632389    331.0
Name: Crm Cd 1, Length: 632390, dtype: float64

Notice that the row selector for the example was a single colon, :. This syntax might seem weird, but it is actually a slice. You could rewrite this slice as 0:len(df), indicating to take all of rows.

Part 2: Using Pandas Loc to Index with Lists

In our next example, the row_indexer is a list of labels [0,1,2], which selects the rows with those labels, and column_indexer is not specified, so all columns are selected.

Here is the full code:

>>> df.loc[[0,1,2]]
    DR_NO       Date Rptd       DATE OCC        TIME OCC    AREA    AREA NAME   Rpt Dist No Part 1-2    Crm Cd  Crm Cd Desc                 ... Status  Status Desc Crm Cd 1    Crm Cd 2    Crm Cd 3    Crm Cd 4    LOCATION    Cross Street    LAT     LON
0   10304468    01/08/2020 12:00:00 AM  01/08/2020 12:00:00 AM  2230        3   Southwest   377     2       624 BATTERY - SIMPLE ASSAULT            ... AO  Adult Other 624.0       NaN     NaN     NaN     1100 W 39TH PL  NaN     34.0141     -118.2978
1   190101086   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  330     1   Central     163     2       624 BATTERY - SIMPLE ASSAULT            ... IC  Invest Cont 624.0       NaN     NaN     NaN     700 S HILL ST   NaN     34.0459     -118.2545
2   200110444   04/14/2020 12:00:00 AM  02/13/2020 12:00:00 AM  1200        1   Central     155     2       845 SEX OFFENDER REGISTRANT OUT OF COMPLIANCE   ... AA  Adult Arrest    845.0       NaN     NaN     NaN     200 E 6TH ST    NaN     34.0448     -118.2474
3 rows × 28 columns

In this next example, we will return all of the rows and two columns, AREA and TIME OCC.

>>> df.loc[:, ['AREA', 'TIME OCC']]
    AREA    TIME OCC
0   3   2230
1   1   330
2   1   1200
3   15  1730
4   19  415
... ... ...
632385  3   10
632386  10  1800
632387  10  1530
632388  11  2000
632389  16  1800
632390 rows × 2 columns

Now let’s combine using a list of row labels and column labels:

>>> df.loc[[0,1,2], ['AREA', 'TIME OCC']]
        AREA    TIME OCC
0   3   2230
1   1   330
2   1   1200

Part 3: Using Pandas Loc to Index with Slices

Now we will dive more into passing slices into row and column selectors. Again, this example is passing in two slices separated by the comma.

In this case, the syntax df.loc[0:2, 'TIME OCC':'Part 1-2'] is used to select the rows with labels 0, 1, and 2, and the columns with labels from 'TIME OCC' to 'Part 1-2', inclusive, in the DataFrame.

>>> df.loc[0:2, 'TIME OCC':'Part 1-2']
    TIME OCC    AREA    AREA NAME   Rpt Dist No Part 1-2
0   2230        3   Southwest   377     2
1   330     1   Central         163     2
2   1200        1   Central         155     2

One gotcha to be aware of is that slicing with Pandas loc is inclusive. It includes the stop value. Normal Python doesn't behave this way. If I slice a string with 0:2, it uses the half-open interval, returning values from index 0 up to but not including 2.

>>> 'hello'[0:2]
'he'

You can also pass in Python lists with boolean values for the index and column selectors. These lists need to be the same size as the index and columns respectively. In practice, I don’t find that I use this option too often, but for completeness I will show it here.

One place where this is useful is if you have a bunch of columns that have the same prefix/suffix/substring. For example, this dataset has a bunch of columns with Crm in them. This example pulls out every third row and selects only the columns that have a 'Crm' in the them.

>>> df.loc[[True if i%3 == 2 else False for i in range(len(df))],
...        [True if 'Crm' in col else False for col in df.columns]]
    Crm Cd  Crm Cd Desc                     Crm Cd 1    Crm Cd 2    Crm Cd 3    Crm Cd 4
2   845 SEX OFFENDER REGISTRANT OUT OF COMPLIANCE       845.0       NaN     NaN     NaN
5   121 RAPE, FORCIBLE                      121.0       998.0       NaN     NaN
8   341 THEFT-GRAND ($950.01 & OVER)EXCPT,GUNS,FOWL,LI...   341.0       998.0       NaN     NaN
11  930 CRIMINAL THREATS - NO WEAPON DISPLAYED          930.0       NaN     NaN     NaN
14  442 SHOPLIFTING - PETTY THEFT ($950 & UNDER)        442.0       NaN     NaN     NaN
... ... ...                         ...     ...     ...     ...
632375  510 VEHICLE - STOLEN                    510.0       NaN     NaN     NaN
632378  341 THEFT-GRAND ($950.01 & OVER)EXCPT,GUNS,FOWL,LI...   341.0       NaN     NaN     NaN
632381  310 BURGLARY                        310.0       998.0       NaN     NaN
632384  626 INTIMATE PARTNER - SIMPLE ASSAULT           626.0       NaN     NaN     NaN
632387  510 VEHICLE - STOLEN                    510.0       NaN     NaN     NaN
210796 rows × 6 columns

I would probably just write the list comprehension for the column selector in the previous example as:

[col for col in df.columns if 'Crm' in col]

Part 4: Using Pandas Loc to Index with Boolean Arrays

In this next example we pass in what the docs call an alignable boolean Series. Often this is called a boolean array in the real world (which comes from NumPy). In the Pandas context, a boolean array is a series with true/false values that has the same index as another dataframe or series.

In this example, row_indexer is a boolean array created by comparing the values in the AREA NAME column to the string 'Central' using the == operator. Only the rows where the value is True are selected. The column_indexer is not specified, so all columns are selected.

>>> df['AREA NAME'] == 'Central' # boolean array
0         False
1          True
2          True
3         False
4         False
          ...
636127    False
636128    False
636129    False
636130    False
636131    False
Name: AREA NAME, Length: 636132, dtype: bool

You can use a boolean array directly in .loc. This should return only the rows from the Central area.

>>> df.loc[df['AREA NAME'] == 'Central']
    DR_NO       Date Rptd       DATE OCC        TIME OCC    AREA    AREA NAME   Rpt Dist No Part 1-2    Crm Cd  Crm Cd Desc                 ... Status  Status Desc Crm Cd 1    Crm Cd 2    Crm Cd 3    Crm Cd 4    LOCATION        Cross Street    LAT LON
1   190101086   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  330     1   Central     163     2       624 BATTERY - SIMPLE ASSAULT            ... IC  Invest Cont 624.0       NaN     NaN     NaN     700 S HILL ST       NaN 34.0459 -118.2545
2   200110444   04/14/2020 12:00:00 AM  02/13/2020 12:00:00 AM  1200        1   Central     155     2       845 SEX OFFENDER REGISTRANT OUT OF COMPLIANCE   ... AA  Adult Arrest    845.0       NaN     NaN     NaN     200 E 6TH ST        NaN     34.0448 -118.2474
5   200100501   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  30      1   Central     163     1       121 RAPE, FORCIBLE                  ... IC  Invest Cont 121.0       998.0       NaN     NaN     700 S BROADWAY      NaN     34.0452 -118.2534
6   200100502   01/02/2020 12:00:00 AM  01/02/2020 12:00:00 AM  1315        1   Central     161     1       442 SHOPLIFTING - PETTY THEFT ($950 & UNDER)    ... IC  Invest Cont 442.0       998.0       NaN     NaN     700 S FIGUEROA ST   NaN     34.0483 -118.2631
7   200100504   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  40      1   Central     155     2       946 OTHER MISCELLANEOUS CRIME           ... IC  Invest Cont 946.0       998.0       NaN     NaN     200 E 6TH ST        NaN     34.0448 -118.2474
... ...     ...         ...         ...     ... ...     ...     ...     ... ...                     ... ... ...     ...     ...     ...     ...     ...         ...     ... ...
632334  220123144   10/24/2022 12:00:00 AM  10/20/2022 12:00:00 AM  1930        1   Central     159     1       330 BURGLARY FROM VEHICLE               ... IC  Invest Cont 330.0       NaN     NaN     NaN     1500 JESSE ST       NaN     34.0362 -118.2325
632336  220112110   05/10/2022 12:00:00 AM  01/10/2022 12:00:00 AM  1150        1   Central     195     2       354 THEFT OF IDENTITY               ... IC  Invest Cont 354.0       NaN     NaN     NaN     100 W 17TH ST       NaN     34.0329 -118.2631
632340  220107479   03/01/2022 12:00:00 AM  02/28/2022 12:00:00 AM  1400        1   Central     129     1       330 BURGLARY FROM VEHICLE               ... IC  Invest Cont 330.0       NaN     NaN     NaN     400 N VIGNES ST     NaN     34.0511 -118.2334
632360  220106541   02/13/2022 12:00:00 AM  02/13/2022 12:00:00 AM  1745        1   Central     145     2       624 BATTERY - SIMPLE ASSAULT            ... AO  Adult Other 624.0       NaN     NaN     NaN     100 E 5TH ST        NaN     34.0471 -118.2474
632361  220118304   08/15/2022 12:00:00 AM  08/14/2022 12:00:00 AM  420     1   Central     195     1       310 BURGLARY                    ... IC  Invest Cont 310.0       NaN     NaN     NaN     1400 S MAIN ST      NaN     34.0351 -118.2614
41748 rows × 28 columns

You can see that rows with index label 3 and 4 are missing because those rows didn’t have crimes in the Central area.

>>> df.loc[[3,4], ['AREA NAME']]
        AREA NAME
3   N Hollywood
4   Mission

We can also use a boolean array (or alignable boolean Series as the docs call it) for the column selector. It is normal to think of a boolean array for the index selector, a True or False value for each row with the same index. What is a boolean array for the column selector?

To use a boolean array for a column selector you would have a Series that had the column names in the index and True or False values.

In this example, we will select all the columns that have more than the median number of unique column entries.

First, let’s calculate the number of unique entries for each column.

>>> df.nunique()
DR_NO       632390
Date Rptd   1091
DATE OCC    1091
TIME OCC    1439
AREA        21
AREA NAME   21
Rpt Dist No 1191
Part 1-2    2
Crm Cd      137
Crm Cd Desc 137
Mocodes     219372
Vict Age    101
Vict Sex    4
Vict Descen 19
Premis Cd   309
Premis Desc 306
Weapon Used Cd  79
Weapon Desc 79
Status      6
Status Desc 6
Crm Cd 1    139
Crm Cd 2    120
Crm Cd 3    35
Crm Cd 4    6
LOCATION    60076
Cross Street    8884
LAT     5363
LON     4953
dtype: int64

Then we will calculate the median of these values.

>>> df.nunique().median()
137.0

The boolean array will be the columns where the number of unique values is greater than 137.

>>> df.nunique() > 137
DR_NO       True
Date Rptd   True
DATE OCC    True
TIME OCC    True
AREA        False
AREA NAME   False
Rpt Dist No True
Part 1-2    False
Crm Cd      False
Crm Cd Desc False
Mocodes     True
Vict Age    False
Vict Sex    False
Vict Descent    False
Premis Cd   True
Premis Desc True
Weapon Used Cd  False
Weapon Desc False
Status      False
Status Desc False
Crm Cd 1    True
Crm Cd 2    False
Crm Cd 3    False
Crm Cd 4    False
LOCATION    True
Cross Street    True
LAT     True
LON     True
dtype: bool

Let's use this boolean array as the column selector. We will use the slice, :, as the row selector to return all of the rows.

>>> df.loc[:, df.nunique() > 137]
    DR_NO       Date Rptd       DATE OCC        TIME OCC    Rpt Dist No Mocodes     Premis Cd   Premis Desc                 Crm Cd 1    LOCATION        Cross Street    LAT LON
0   10304468    01/08/2020 12:00:00 AM  01/08/2020 12:00:00 AM  2230        377     0444 0913   501.0       SINGLE FAMILY DWELLING              624.0       1100 W 39TH PL      NaN     34.0141 -118.2978
1   190101086   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  330     163     0416 1822 1414  102.0       SIDEWALK                    624.0       700 S HILL ST       NaN     34.0459 -118.2545
2   200110444   04/14/2020 12:00:00 AM  02/13/2020 12:00:00 AM  1200        155     1501        726.0       POLICE FACILITY                 845.0       200 E 6TH ST        NaN     34.0448 -118.2474
3   191501505   01/01/2020 12:00:00 AM  01/01/2020 12:00:00 AM  1730        1543        0329 1402   502.0       MULTI-UNIT DWELLING (APARTMENT, DUPLEX, ETC)    745.0       5400 CORTEEN PL     NaN     34.1685 -118.4019
4   191921269   01/01/2020 12:00:00 AM  01/01/2020 12:00:00 AM  415     1998        0329        409.0       BEAUTY SUPPLY STORE             740.0       14400 TITUS ST      NaN     34.2198 -118.4468
... ...     ...         ...         ...     ...     ...     ...     ...                     ...     ...         ...     ... ...
632385  220314584   08/02/2022 12:00:00 AM  08/02/2022 12:00:00 AM  10      334     0100 1822 0922  501.0       SINGLE FAMILY DWELLING              354.0       4300 W 28TH ST      NaN     34.0311 -118.3335
632386  221017286   12/05/2022 12:00:00 AM  12/01/2022 12:00:00 AM  1800        1047        1822 0385   101.0       STREET                      331.0       17200 BURBANK BL    NaN     34.1722 -118.5077
632387  221005507   02/10/2022 12:00:00 AM  02/09/2022 12:00:00 AM  1530        1024        NaN     108.0       PARKING LOT                 510.0       18800 SHERMAN WY    NaN     34.2011 -118.5426
632388  221105477   02/10/2022 12:00:00 AM  02/08/2022 12:00:00 AM  2000        1171        NaN     101.0       STREET                      510.0       4000 FOUNTAIN AV    NaN     34.0958 -118.2787
632389  221605448   02/15/2022 12:00:00 AM  02/14/2022 12:00:00 AM  1800        1613        0385 1300   101.0       STREET                      331.0       12700 VAN NUYS BL   NaN     34.2755 -118.4092
632390 rows × 13 columns

Part 5: Using Pandas Loc to Index with Indexes

Let’s look at using an alignable index for selection. This consists of passing in an Index that has entries that correspond to entries in the dataframe.

>>> df.loc[pd.Index([0,5,100]),
...        pd.Index(['AREA', 'TIME OCC'])]
        AREA    TIME OCC
0   3   2230
5   1   30
100 1   2210

Why would you use this? In the example above it is just more typing that passing in a list to the row and column indexers.

I grepped through a bunch of my Pandas code to look for examples of this. Let me show two.

This first example comes from creating a dataset for machine learning. It creates the X dataset by dropping rows with missing values. To find the corresponding y labels we use the index from X. Note the line .loc[X.index].

url = 'https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/DAAG/possum.csv'
possum = pd.read_csv(url, index_col=[0])
X = (possum
     .dropna()
     .assign(ones=1)
     .loc[:,['ones',
             'hdlngth', 'skullw', 'totlngth', 'taill',
             'footlgth', 'earconch', 'eye', 'chest', 'belly']]
    )
y = (possum
     .loc[X.index]
     .age
    )

Here's one more example that combines the .intersection method of an Index to gather financial data where the we had the smallest openings and the largest gains.

>>> import pandas as pd
>>> url = 'https://query1.finance.yahoo.com/v7/finance/download/TSLA?period1=1669852800&period2=1672444800&interval=1d&events=history&includeAdjustedClose=true'
>>> tsla = pd.read_csv(url)
>>> def get_smallest_open_largest_gain(df_):
...     smallest = df_.nsmallest(10, 'Open')
...     largest = df_.nlargest(10, 'Gain')
...     return df_.loc[smallest.index.intersection(largest.index)]

>>> (tsla
...  .astype({"Date": 'datetime64[ns]',
...           "Open":'float',
...           "High":'float',
...           "Low":'float',
...           "Close":'float',
...           "Adj Close":'float',
...           "Volume":'float'})
...  .assign(Gain=lambda df_:df_.Close - df_.Open)
...  .pipe(get_smallest_open_largest_gain)
... )
    Date        Open        High        Low     Close       Adj Close   Volume      Gain
18  2022-12-28  110.349998  116.269997  108.239998  112.709999  112.709999  221070500.0 2.360001
20  2022-12-30  119.949997  124.480003  119.750000  123.180000  123.180000  157304500.0 3.230003
19  2022-12-29  120.389999  123.570000  117.500000  121.820000  121.820000  221923300.0 1.430001
14  2022-12-21  139.339996  141.259995  135.889999  137.570007  137.570007  145417400.0 -1.769989
10  2022-12-15  153.440002  160.929993  153.279999  157.669998  157.669998  122334500.0 4.229996

Part 6: Using Pandas Loc with Functions as Indexers

Now comes the fun part and the brain-bending part: Passing in a function for the selectors. These come in very hand when you are chaining operations and the index has different values (very common with grouping operations).

Here is an example showing passing in a function that just returns a slice to get the labels from 0 through 10. Again, I don’t need to use a function here, but I’m showing it to introduce the concept before completely melting brains.

>>> def first_ten(df):
...   return slice(0,10)

>>> df.loc[first_ten]  # same as df.loc[0:10]
    DR_NO       Date Rptd       DATE OCC        TIME OCC    AREA    AREA NAME   Rpt Dist No Part 1-2    Crm Cd  Crm Cd Desc                     ... Status  Status Desc Crm Cd 1    Crm Cd 2    Crm Cd 3    Crm Cd 4    LOCATION        Cross Street    LAT LON
0   10304468    01/08/2020 12:00:00 AM  01/08/2020 12:00:00 AM  2230        3   Southwest   377     2       624 BATTERY - SIMPLE ASSAULT                ... AO  Adult Other 624.0       NaN     NaN     NaN     1100 W 39TH PL      NaN     34.0141 -118.2978
1   190101086   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  330     1   Central     163     2       624 BATTERY - SIMPLE ASSAULT                ... IC  Invest Cont 624.0       NaN     NaN     NaN     700 S HILL ST       NaN     34.0459 -118.2545
2   200110444   04/14/2020 12:00:00 AM  02/13/2020 12:00:00 AM  1200        1   Central     155     2       845 SEX OFFENDER REGISTRANT OUT OF COMPLIANCE       ... AA  Adult Arrest    845.0       NaN     NaN     NaN     200 E 6TH ST        NaN     34.0448 -118.2474
3   191501505   01/01/2020 12:00:00 AM  01/01/2020 12:00:00 AM  1730        15  N Hollywood 1543        2       745 VANDALISM - MISDEAMEANOR ($399 OR UNDER)        ... IC  Invest Cont 745.0       998.0       NaN     NaN     5400 CORTEEN PL     NaN     34.1685 -118.4019
4   191921269   01/01/2020 12:00:00 AM  01/01/2020 12:00:00 AM  415     19  Mission     1998        2       740 VANDALISM - FELONY ($400 & OVER, ALL CHURCH VA...   ... IC  Invest Cont 740.0       NaN     NaN     NaN     14400 TITUS ST      NaN     34.2198 -118.4468
5   200100501   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  30      1   Central     163     1       121 RAPE, FORCIBLE                      ... IC  Invest Cont 121.0       998.0       NaN     NaN     700 S BROADWAY      NaN     34.0452 -118.2534
6   200100502   01/02/2020 12:00:00 AM  01/02/2020 12:00:00 AM  1315        1   Central     161     1       442 SHOPLIFTING - PETTY THEFT ($950 & UNDER)        ... IC  Invest Cont 442.0       998.0       NaN     NaN     700 S FIGUEROA ST   NaN     34.0483 -118.2631
7   200100504   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  40      1   Central     155     2       946 OTHER MISCELLANEOUS CRIME               ... IC  Invest Cont 946.0       998.0       NaN     NaN     200 E 6TH ST        NaN     34.0448 -118.2474
8   200100507   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  200     1   Central     101     1       341 THEFT-GRAND ($950.01 & OVER)EXCPT,GUNS,FOWL,LI...   ... IC  Invest Cont 341.0       998.0       NaN     NaN     700 BERNARD ST      NaN     34.0677 -118.2398
9   201710201   06/19/2020 12:00:00 AM  05/26/2020 12:00:00 AM  1925        17  Devonshire  1708        1       341 THEFT-GRAND ($950.01 & OVER)EXCPT,GUNS,FOWL,LI...   ... AO  Adult Other 341.0       NaN     NaN     NaN     11900 BALBOA BL     NaN     34.2864 -118.5021
10  200100509   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  2200        1   Central     192     1       330 BURGLARY FROM VEHICLE                   ... IC  Invest Cont 330.0       NaN     NaN     NaN     15TH            OLIVE       34.0359 -118.2648
11 rows × 28 columns

Unless I'm reusing the function in multiple places in my code, I generally will inline it with an anonymous function, a lambda.

This is the same code as above but it uses a lambda.

>>> df.loc[lambda ignore_df: slice(0,10)]
    DR_NO       Date Rptd       DATE OCC        TIME OCC    AREA    AREA NAME   Rpt Dist No Part 1-2    Crm Cd  Crm Cd Desc                     ... Status  Status Desc Crm Cd 1    Crm Cd 2    Crm Cd 3    Crm Cd 4    LOCATION        Cross Street    LAT LON
0   10304468    01/08/2020 12:00:00 AM  01/08/2020 12:00:00 AM  2230        3   Southwest   377     2       624 BATTERY - SIMPLE ASSAULT                ... AO  Adult Other 624.0       NaN     NaN     NaN     1100 W 39TH PL      NaN     34.0141 -118.2978
1   190101086   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  330     1   Central     163     2       624 BATTERY - SIMPLE ASSAULT                ... IC  Invest Cont 624.0       NaN     NaN     NaN     700 S HILL ST       NaN     34.0459 -118.2545
2   200110444   04/14/2020 12:00:00 AM  02/13/2020 12:00:00 AM  1200        1   Central     155     2       845 SEX OFFENDER REGISTRANT OUT OF COMPLIANCE       ... AA  Adult Arrest    845.0       NaN     NaN     NaN     200 E 6TH ST        NaN     34.0448 -118.2474
3   191501505   01/01/2020 12:00:00 AM  01/01/2020 12:00:00 AM  1730        15  N Hollywood 1543        2       745 VANDALISM - MISDEAMEANOR ($399 OR UNDER)        ... IC  Invest Cont 745.0       998.0       NaN     NaN     5400 CORTEEN PL     NaN     34.1685 -118.4019
4   191921269   01/01/2020 12:00:00 AM  01/01/2020 12:00:00 AM  415     19  Mission     1998        2       740 VANDALISM - FELONY ($400 & OVER, ALL CHURCH VA...   ... IC  Invest Cont 740.0       NaN     NaN     NaN     14400 TITUS ST      NaN     34.2198 -118.4468
5   200100501   01/02/2020 12:00:00 AM  01/01/2020 12:00:00 AM  30      1   Central     163     1       121 RAPE, FORCIBLE                      ... IC  Invest Cont 121.0       998.0       NaN     NaN     700 S BROADWAY      NaN     34.0452 -118.2534
6   200100502   01/02/2020 12:00:00 AM  01/02/2020 12:00:00 AM  1315        1   Central     161     1       442 SHOPLIFTING - PETTY THEFT ($950 & UNDER)        ... IC  Invest Cont 442.0       998.0       NaN     NaN     700 S FIGUEROA ST   NaN     34.0483 -118.2631
7   200100504   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  40      1   Central     155     2       946 OTHER MISCELLANEOUS CRIME               ... IC  Invest Cont 946.0       998.0       NaN     NaN     200 E 6TH ST        NaN     34.0448 -118.2474
8   200100507   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  200     1   Central     101     1       341 THEFT-GRAND ($950.01 & OVER)EXCPT,GUNS,FOWL,LI...   ... IC  Invest Cont 341.0       998.0       NaN     NaN     700 BERNARD ST      NaN     34.0677 -118.2398
9   201710201   06/19/2020 12:00:00 AM  05/26/2020 12:00:00 AM  1925        17  Devonshire  1708        1       341 THEFT-GRAND ($950.01 & OVER)EXCPT,GUNS,FOWL,LI...   ... AO  Adult Other 341.0       NaN     NaN     NaN     11900 BALBOA BL     NaN     34.2864 -118.5021
10  200100509   01/04/2020 12:00:00 AM  01/04/2020 12:00:00 AM  2200        1   Central     192     1       330 BURGLARY FROM VEHICLE                   ... IC  Invest Cont 330.0       NaN     NaN     NaN     15TH            OLIVE       34.0359 -118.2648
11 rows × 28 columns

Note that we can also pass in a function to the column selector, in this case we get an error because the columns are strings and the function trys to slice with numbers.

>>> df.loc[:, first_ten]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-34-484bd8722f00> in <module>
----> 1 df.loc[:, first_ten]
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in __getitem__(self, key)
    923                 with suppress(KeyError, IndexError):
    924                     return self.obj._get_value(*key, takeable=self._takeable)
--> 925             return self._getitem_tuple(key)
    926         else:
    927             # we by definition only have the 0th axis
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in _getitem_tuple(self, tup)
   1107             return self._multi_take(tup)
   1108
-> 1109         return self._getitem_tuple_same_dim(tup)
   1110
   1111     def _get_label(self, label, axis: int):
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in _getitem_tuple_same_dim(self, tup)
    804                 continue
    805
--> 806             retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
    807             # We should never have retval.ndim < self.ndim, as that should
    808             #  be handled by the _getitem_lowerdim call above.
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1140         if isinstance(key, slice):
   1141             self._validate_key(key, axis)
-> 1142             return self._get_slice_axis(key, axis=axis)
   1143         elif com.is_bool_indexer(key):
   1144             return self._getbool_axis(key, axis=axis)
/usr/local/lib/python3.8/dist-packages/pandas/core/indexing.py in _get_slice_axis(self, slice_obj, axis)
   1174
   1175         labels = obj._get_axis(axis)
-> 1176         indexer = labels.slice_indexer(slice_obj.start, slice_obj.stop, slice_obj.step)
   1177
   1178         if isinstance(indexer, slice):
/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py in slice_indexer(self, start, end, step, kind)
   5683         slice(1, 3, None)
   5684         """
-> 5685         start_slice, end_slice = self.slice_locs(start, end, step=step)
   5686
   5687         # return a slice
/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py in slice_locs(self, start, end, step, kind)
   5885         start_slice = None
   5886         if start is not None:
-> 5887             start_slice = self.get_slice_bound(start, "left")
   5888         if start_slice is None:
   5889             start_slice = 0
/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py in get_slice_bound(self, label, side, kind)
   5795         # For datetime indices label may be a string that has to be converted
   5796         # to datetime boundary according to its resolution.
-> 5797         label = self._maybe_cast_slice_bound(label, side)
   5798
   5799         # we need to look up the label
/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py in _maybe_cast_slice_bound(self, label, side, kind)
   5747         # reject them, if index does not contain label
   5748         if (is_float(label) or is_integer(label)) and label not in self._values:
-> 5749             raise self._invalid_indexer("slice", label)
   5750
   5751         return label
TypeError: cannot do slice indexing on Index with these indexers [0] of type int

Here is the obligatory example with a function in the column selector.

>>> df.loc[:, lambda adf: slice('TIME OCC','Part 1-2')]
    TIME OCC    AREA    AREA NAME   Rpt Dist No Part 1-2
0   2230        3   Southwest   377     2
1   330     1   Central     163     2
2   1200        1   Central     155     2
3   1730        15  N Hollywood 1543        2
4   415     19  Mission     1998        2
... ...     ... ...     ...     ...
632385  10      3   Southwest   334     2
632386  1800        10  West Valley 1047        1
632387  1530        10  West Valley 1024        1
632388  2000        11  Northeast   1171        1
632389  1800        16  Foothill    1613        1
632390 rows × 5 columns

Here is a more realistic example showing using a function in .loc in a chain of operations. Assume we want to understand how vehicle crimes occur over monthly periods. We could use some code like this.

This code uses several functions and methods from the Pandas library to clean, organize, and summarize data in a DataFrame. First, the .astype method is used to convert the data types of the 'Date Rptd' and 'DATE OCC' columns to 'Datetime64'. Then, the .groupby method is used to group the data by the 'DATE OCC' column and the 'AREA NAME' and 'Crm Cd Desc' columns. The .size method is then used to count the number of rows in each group, and the .unstack method is used to pivot the data so that the 'Crm Cd Desc' values become column labels.

Finally, the .loc indexer is used to select only the columns where the label contains the string 'VEHICLE'. We need to use a function to get the column labels because these columns come from the value of the Crm Cd Desc column. They were not columns in the original data frame.

The a_df parameter for the lambda function is the intermediate state of the data in the chain. It is the result of the .unstack method on the line above it.

I use functions all the time in .loc because I'm a big proponent of chaining your Pandas code to make it easier to understand. (See our article on chaining for more details on chaining.)

>>> (df
...  .astype({'Date Rptd': 'Datetime64',
...           'DATE OCC': 'Datetime64',
...           })
...  .groupby([pd.Grouper(key='DATE OCC', freq='M'), 'AREA NAME', 'Crm Cd Desc'])
...  .size()
...  .unstack()
...  .loc[:,
...       lambda a_df: [col for col in a_df.columns if 'VEHICLE' in col]]
...  )
        Crm Cd Desc BURGLARY FROM VEHICLE   BURGLARY FROM VEHICLE, ATTEMPTED    SHOTS FIRED AT MOVING VEHICLE, TRAIN OR AIRCRAFT    THEFT FROM MOTOR VEHICLE - ATTEMPT  THEFT FROM MOTOR VEHICLE - GRAND ($950.01 AND OVER) THEFT FROM MOTOR VEHICLE - PETTY ($950 & UNDER) THROWING OBJECT AT MOVING VEHICLE   VEHICLE - ATTEMPT STOLEN    VEHICLE - MOTORIZED SCOOTERS, BICYCLES, AND WHEELCHAIRS VEHICLE - STOLEN
DATE OCC    AREA NAME
2020-01-31  77th Street 75.0            NaN                 9.0                         3.0                 17.0                            49.0                        2.0                 4.0             NaN                         123.0
        Central     144.0           NaN                 NaN                         1.0                 16.0                            31.0                        1.0                 NaN             1.0                         45.0
        Devonshire  67.0            NaN                 NaN                         NaN                 20.0                            49.0                        1.0                 3.0             NaN                         40.0
        Foothill    37.0            NaN                 NaN                         NaN                 13.0                            40.0                        NaN                 2.0             NaN                         69.0
        Harbor      29.0            NaN                 NaN                         NaN                 19.0                            40.0                        1.0                 1.0             NaN                         77.0
...     ...     ...         ...                 ...                         ...                 ...                         ...                     ...                 ...             ...                         ...
2020-06-30  Topanga     55.0            2.0                 NaN                         1.0                 25.0                            39.0                        2.0                 NaN             NaN                         74.0
        Van Nuys    81.0            2.0                 NaN                         NaN                 24.0                            45.0                        1.0                 3.0             NaN                         97.0
        West LA     80.0            1.0                 NaN                         NaN                 32.0                            32.0                        NaN                 1.0             2.0                         62.0
        West Valley 51.0            NaN                 NaN                         2.0                 23.0                            37.0                        NaN                 3.0             NaN                         79.0
        Wilshire    50.0            1.0                 NaN                         1.0                 16.0                            37.0                        1.0                 1.0             NaN                         60.0
126 rows × 10 columns

A Practical Example of Pandas Loc

We will wrap up with a visualization of the previous analysis. I will dive into the Burglary from Vehicle column and plot that over time for the different Area Name values.

In this example, in the latter .loc I use a lambda function to move the Hollywood column to the end so that it appears on top of the other columns.

def set_colors(df):
  global colors
  colors = ['#991111' if col == 'Hollywood' else '#999999' for col in df.columns]
  return df

ax = (df
 .astype({'Date Rptd': 'Datetime64',
          'DATE OCC': 'Datetime64',
          })
 .groupby([pd.Grouper(key='DATE OCC', freq='M'), 'AREA NAME', 'Crm Cd Desc'])
 .size()
 .unstack()
 .loc[:, 'BURGLARY FROM VEHICLE']
 .unstack()
 .loc[:, lambda adf: sorted(adf.columns, key=lambda col:1 if col == 'Hollywood' else -1)]
 .pipe(set_colors)
 .plot(color=colors, title='BURGLARIES FROM VEHICLE in LA')
 )
ax.legend(bbox_to_anchor=(1,1), ncol=2)

Using Loc with Modin

In Modin's pandas API, .loc works identically to the original Pandas. As you can imagine, this is tricky in a distributed dataframe system. There were a lot of new problems to solve:

What machine is a given row label on?
How do we know if a row is not on any machine and throw an error for the user?
How do we make this fast?

We solved all three of these by separating the dataframe's data from the Index and storing the Index close to the user for fast lookups with .loc. We maintain an internal mapping from the fast Index to the machine that the row is on. This way we can make the row lookups fast!

Though .loc is fast in Modin, it is not as fast as Pandas because of the extra layer between the data and the Index. Still, the lookup should be within a few milliseconds of Pandas — and still well within the window of fast interactive performance.

Conclusion

Thanks for reading part one of our ultimate guide to indexing in Pandas (and the third article in our Professional Pandas series). In this article we introduced using Pandas loc to select rows and columns. You should have a good understanding of the different inputs that the .loc index accepts.

To practice and further improve your skills, try using .loc on your own datasets and see how you can effectively index and slice your data. Don't forget to also review the common gotchas discussed in this tutorial to avoid any pitfalls when using these indexing attributes.

For more in this series, read the Pandas assign method and chaining and the Pandas merge method.

Professional Pandas: The Pandas Merge Method

2022-11-03T00:00:00-06:00

Republication note: I originally wrote this article for Ponder, where it was published on November 3, 2022. Ponder later removed the post following its acquisition, so I have preserved it here. The Ponder- and Modin-specific code reflects the environment available when the article was written; the pandas concepts remain broadly applicable. View the archived original.

Combining related datasets safely with the pandas merge method.

Intro

In this post, I discuss the relationship between the Pandas merge function (and Pandas merge method), and the Pandas join method, and then show examples of how to merge Pandas dataframes with the merge method. Along the way, I describe two useful merge method parameters: the validate parameter, and the indicator parameter.

This article is the second in our Professional Pandas blog post series. The goal of this series is to teach best practices about writing professional-grade Pandas code. If you have questions or topics that you would like to dive into, please reach out on Twitter to @ponderdata or @__mharrison__, and if you’d like to see the notebook version of this blogpost, you can find it on Github here.

(Note: This content is modified from the author’s book, Effective Pandas.)

Joining Data

Terminology Refresher: Method vs. Function

Pandas often has one, two, three, or even more ways to do something. This paradox of choice can be overwhelming. (Indeed software engineering purists would say “There should be one– and preferably only one –obvious way to do it.”) However, here we are with Pandas joining.

In short, there is :

The .join method found on a DataFrame
The .merge method found on a DataFrame
The pd.merge function found in the Pandas namespace (pd)
The pd.concat function to horizontally or vertically stack data
Various methods that align on the index and work similar to some joins (like using df.assign(**other_df))

A Quick Aside on the Python Language

A Python class defines methods. Methods are called on an instance of a class. In the Pandas world, we often have a DataFrame (df) or Series (s) instance. We could call the merge method by saying:

df.merge(another_df)

However, there is also a function, merge. Generally, when folks use Pandas they import the library like this:

import pandas as pd

Since pd is the module where the Pandas merge function is available, you can call the merge function like this:

pd.merge(df, another_df)

Some think that in this last example that merge is a method on pd. That is not correct. If we change the import to load the function, we could run the code like this:

from pandas import merge

merge(df, another_df)

If you aren’t familiar with the difference between methods and functions and would like to explore more, check out the Python tutorial.

End of the Aside

From this author's point of view, forget that the Pandas merge function and the join method exist. You can get along just fine only using the merge method (which I will often refer to in this article as .merge with a period in front as a reminder that it needs to be called on the instance).

In fact, if you dig into the implementation of .join, you will see that it calls the concat or merge functions. Don't waste your brain power trying to keep them straight. Just ignore the merge function and the join method.

One final note, I wish that .merge was named .join because everyone coming from a database world knows what a join is, but .merge offers a little more functionality, hence my recommendation to favor it.

A Short Overview of Joining

Feel free to skip this if you are familiar with database joins. It shows the basic joins using some sales data. Our task might be to find products with no sales, sales that don’t have products, or only sales for products that we have information for.

The supported join types for .merge (specified with the how parameter) are:

'left' – Keep all values from the merging column DataFrame the method was called on.
'right' – Keep all values from the merging column of the right DataFrame the method was called on.
'inner' – Keep the intersection of all values from the merging columns.
'outer' – Keep the union of all values from the merging columns.
'cross' – For every row in the instance dataframe, combine with every row from the right DataFrame.

import pandas as pd
products = pd.DataFrame({'id': [1, 3, 10], 'product':
                         ['Cat food', 'Sparkling Water', 'Apple Fritter']})
sales = pd.DataFrame({'product_id': [1, 3, 1, 1, 15],
                      'cost': [4.99, .99, 4.99, 4.99*.8, 7.45]})

Left Join

>>> # keep all *id* columns
>>> products.merge(sales, left_on='id', right_on='product_id', how='left')
    id  product     product_id  cost
0   1   Cat food    1.0     4.990
1   1   Cat food    1.0     4.990
2   1   Cat food    1.0     3.992
3   3   Sparkling Water 3.0     0.990
4   10  Apple Fritter   NaN     NaN

Right Join

>>> # keep all *product_id* columns
>>> products.merge(sales, left_on='id', right_on='product_id', how='right')
    id  product     product_id  cost
0   1.0 Cat food    1       4.990
1   3.0 Sparkling Water 3       0.990
2   1.0 Cat food    1       4.990
3   1.0 Cat food    1       3.992
4   NaN NaN     15      7.450

Inner Join

>>> products.merge(sales, left_on='id', right_on='product_id', how='inner')
    id  product     product_id  cost
0   1   Cat food    1       4.990
1   1   Cat food    1       4.990
2   1   Cat food    1       3.992
3   3   Sparkling Water 3       0.990

Outer Join

>>> products.merge(sales, left_on='id', right_on='product_id', how='outer')
    id  product     product_id  cost
0   1.0 Cat food    1.0     4.990
1   1.0 Cat food    1.0     4.990
2   1.0 Cat food    1.0     3.992
3   3.0 Sparkling Water 3.0     0.990
4   10.0    Apple Fritter   NaN     NaN
5   NaN NaN     15.0        7.450

Cross Join

A merge you might not be aware of is the cross join. This repeats every row of the dataframe with every row that it is merged with. (So this gets big real fast).

>>> # cross join
>>> products.merge(sales, how='cross')
    id  product     product_id  cost
0   1   Cat food    1       4.990
1   1   Cat food    3       0.990
2   1   Cat food    1       4.990
3   1   Cat food    1       3.992
4   1   Cat food    15      7.450
5   3   Sparkling Water 1       4.990
6   3   Sparkling Water 3       0.990
7   3   Sparkling Water 1       4.990
8   3   Sparkling Water 1       3.992
9   3   Sparkling Water 15      7.450
10  10  Apple Fritter   1       4.990
11  10  Apple Fritter   3       0.990
12  10  Apple Fritter   1       4.990
13  10  Apple Fritter   1       3.992
14  10  Apple Fritter   15      7.450

A cross join might come in hand if you need to create all the permutations of some values. In this example, all ways to select three different values from four or ten.

>>> nums = pd.DataFrame({'values':[4, 10]})
>>> (nums
...  .merge(nums, how='cross')
...  .merge(nums, how='cross')
... )
    values_x    values_y    values
0   4       4       4
1   4       4       10
2   4       10      4
3   4       10      10
4   10      4       4
5   10      4       10
6   10      10      4
7   10      10      10

Validation Using the Pandas Merge Method

I will sometimes use a .pipe method to track the size of my data before and after the merge. If you are doing a lot of merges, this will be a snippet of code you will want to keep around.

(Check out the article Professional Pandas: The Pandas Assign Method for more examples of the .pipe method.)

>>> def debug_shape(a_df, comment=''):
...     print(f'{a_df.shape} {comment}')
...     return a_df

>>> (products
...  .pipe(debug_shape, 'before merge')
...  .merge(sales, left_on='id', right_on='product_id', how='outer')
...  .pipe(debug_shape, 'after merge')
... )
(3, 2) before merge
(6, 4) after merge
    id  product     product_id  cost
0   1.0 Cat food    1.0     4.990
1   1.0 Cat food    1.0     4.990
2   1.0 Cat food    1.0     3.992
3   3.0 Sparkling Water 3.0     0.990
4   10.0    Apple Fritter   NaN     NaN
5   NaN NaN     15.0        7.450

A very nice feature of the Pandas merge method is the validate parameter. It allows us to specify if our merge should be one to one ('1:1'), one to many ('1:m'), many to one ('m:1'), or many to many ('m:m').

If your data grows much faster after a merge that might be because you are merging on duplicate values. In our example, the product_id repeats in the products DataFrame. If we were expecting only a single row for the Cat food product and one row from sales for each row from product, we could pass in validate='1:1' parameter.

In this case, Pandas complains that the entries from sales are not unique. This is because there were multiple sales for the Cat food product:

>>> (products
...  .merge(sales, left_on='id', right_on='product_id', how='outer',
...         validate='1:1')
... )
---------------------------------------------------------------------------
MergeError                                Traceback (most recent call last)
 in
      1 (products
      2  .merge(sales, left_on='id', right_on='product_id', how='outer',
----> 3         validate='1:1')
      4 )

3 frames
/usr/local/lib/python3.7/dist-packages/pandas/core/reshape/merge.py in _validate(self, validate)
   1428             elif not right_unique:
   1429                 raise MergeError(
-> 1430                     "Merge keys are not unique in right dataset; not a one-to-one merge"
   1431                 )
   1432

MergeError: Merge keys are not unique in right dataset; not a one-to-one merge

We would need to use validate='1:m' on this example:

>>> (products
...  .merge(sales, left_on='id', right_on='product_id', how='outer',
...         validate='1:m')
... )
    id  product     product_id  cost
0   1.0 Cat food    1.0     4.990
1   1.0 Cat food    1.0     4.990
2   1.0 Cat food    1.0     3.992
3   3.0 Sparkling Water 3.0     0.990
4   10.0    Apple Fritter   NaN     NaN
5   NaN NaN     15.0        7.450

(Note that specifying validate='m:m' can't ever fail, but it could be a nice indicator to whoever reads the code next.)

One more parameter to be aware of is the indicator parameter. This allows us to create a new column indicating which DataFrame the resulting row came from. In this example we specify that the new column should be named side:

>>> (products
...  .merge(sales, left_on='id', right_on='product_id', how='outer',
...         indicator='side')
... )
    id  product     product_id  cost    side
0   1.0 Cat food    1.0     4.990   both
1   1.0 Cat food    1.0     4.990   both
2   1.0 Cat food    1.0     3.992   both
3   3.0 Sparkling Water 3.0     0.990   both
4   10.0    Apple Fritter   NaN     NaN left_only
5   NaN NaN     15.0        7.450   right_only

Example: Joining Dirty Devil Flow and Weather Data Using the Pandas Merge Method

Now let’s look at a real-world example.

We are going to explore data from the Dirty Devil river. This is a river in Utah that the author had some interest in floating down. However, it is only floatable during a small period of time each year. This analysis was part of his attempt to determine when would be a good time to float down this river.

I have some historic data about the flow of the Dirty Devil river on my Github. Let’s load the flow (cubic feet per second, or cfs) and gage height data. In this case we will leave the datetime column as a column and not use it for the index:

>>> import pandas as pd
>>> url = 'https://github.com/mattharrison/datasets/raw/master'\
...     '/data/dirtydevil.txt'
>>> df = pd.read_csv(url, skiprows=lambda num: num <34 or num == 35,
...                  sep='\t')
>>> def to_denver_time(df_, time_col, tz_col):
...     return (df_
...      .assign(**{tz_col: df_[tz_col].replace('MDT', 'MST7MDT')})
...      .groupby(tz_col)
...      [time_col]
...      .transform(lambda s: pd.to_datetime(s)
...          .dt.tz_localize(s.name, ambiguous=True)
...          .dt.tz_convert('America/Denver'))
...     )

>>> def tweak_river(df_):
...     return (df_
...      .assign(datetime=to_denver_time(df_, 'datetime', 'tz_cd'))
...      .rename(columns={'144166_00060': 'cfs',
...                       '144167_00065': 'gage_height'})
...     )

>>> dd = tweak_river(df)
>>> dd
/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py:3326: DtypeWarning: Columns (7) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code_obj, self.user_global_ns, self.user_ns)
    agency_cd   site_no     datetime            tz_cd   cfs 144166_00060_cd gage_height 144167_00065_cd
0   USGS        9333500     2001-05-07 01:00:00-06:00   MDT 71.00   A:[91]      NaN     NaN
1   USGS        9333500     2001-05-07 01:15:00-06:00   MDT 71.00   A:[91]      NaN     NaN
2   USGS        9333500     2001-05-07 01:30:00-06:00   MDT 71.00   A:[91]      NaN     NaN
3   USGS        9333500     2001-05-07 01:45:00-06:00   MDT 70.00   A:[91]      NaN     NaN
4   USGS        9333500     2001-05-07 02:00:00-06:00   MDT 70.00   A:[91]      NaN     NaN
... ...     ...     ...             ... ... ...     ...     ...
539300  USGS        9333500     2020-09-28 08:30:00-06:00   MDT 9.53    P       6.16        P
539301  USGS        9333500     2020-09-28 08:45:00-06:00   MDT 9.20    P       6.15        P
539302  USGS        9333500     2020-09-28 09:00:00-06:00   MDT 9.20    P       6.15        P
539303  USGS        9333500     2020-09-28 09:15:00-06:00   MDT 9.20    P       6.15        P
539304  USGS        9333500     2020-09-28 09:30:00-06:00   MDT 9.20    P       6.15        P
539305 rows × 8 columns

I’m also going to load some meteorological data from Hanksville, Utah, a city nearby the river. We will then use the Pandas merge method to join both datasets together so we have flow data as well as temperature and precipitation information.

Some of the columns that are interesting are:

DATE – Date
PRCP – Precipitation in inches
TMIN – Minimum temperature (F) for day
TMAX – Maximum temperature (F) for day
TOBS – Observed temperature (F) when measurement made

>>> url = 'https://github.com/mattharrison/datasets/raw/master/data/'\
...       'hanksville.csv'
>>> temp_df = pd.read_csv(url)
>>> def tweak_temp(df_):
...     return (df_
...            .assign(DATE=pd.to_datetime(df_.DATE)
...                 .dt.tz_localize('America/Denver', ambiguous=False))
...            .loc[:,['DATE', 'PRCP', 'TMIN', 'TMAX', 'TOBS']]
...     )
>>> temp_df = tweak_temp(temp_df)
>>> temp_df
    DATE                PRCP    TMIN    TMAX    TOBS
0   2000-01-01 00:00:00-07:00   0.02    21.0    43.0    28.0
1   2000-01-02 00:00:00-07:00   0.03    24.0    39.0    24.0
2   2000-01-03 00:00:00-07:00   0.00    7.0 39.0    18.0
3   2000-01-04 00:00:00-07:00   0.00    5.0 39.0    25.0
4   2000-01-05 00:00:00-07:00   0.00    10.0    44.0    22.0
... ...             ... ... ... ...
6843    2020-09-20 00:00:00-06:00   0.00    46.0    92.0    83.0
6844    2020-09-21 00:00:00-06:00   0.00    47.0    92.0    84.0
6845    2020-09-22 00:00:00-06:00   0.00    54.0    84.0    77.0
6846    2020-09-23 00:00:00-06:00   0.00    47.0    91.0    87.0
6847    2020-09-24 00:00:00-06:00   0.00    43.0    94.0    88.0
6848 rows × 5 columns

Joining Data

Let's use the Pandas merge method to merge by date. This method will try to merge by columns that have the same name, but our dataframes don't have overlapping column names. The dd dataframe has a datetime column and temp_df has a DATE column. We can use the left_on and right_on parameters to help it know how to align the data. The merge method tries to do an inner join by default. That means that rows with values that are the same in the merge columns will be joined together:

>>> (dd
... .merge(temp_df, left_on='datetime', right_on='DATE')
... )
    agency_cd   site_no     datetime            tz_cd   cfs 144166_00060_cd gage_height 144167_00065_cd DATE                PRCP    TMIN    TMAX    TOBS
0   USGS        9333500     2001-05-08 00:00:00-06:00   MDT 75.00   A:[91]      NaN     NaN     2001-05-08 00:00:00-06:00   0.0 43.0    85.0    58.0
1   USGS        9333500     2001-05-09 00:00:00-06:00   MDT 64.00   A:[91]      NaN     NaN     2001-05-09 00:00:00-06:00   0.0 36.0    92.0    64.0
2   USGS        9333500     2001-05-10 00:00:00-06:00   MDT 54.00   A:[91]      NaN     NaN     2001-05-10 00:00:00-06:00   0.0 50.0    92.0    67.0
3   USGS        9333500     2001-05-11 00:00:00-06:00   MDT 56.00   A:[91]      NaN     NaN     2001-05-11 00:00:00-06:00   0.0 46.0    87.0    60.0
4   USGS        9333500     2001-05-12 00:00:00-06:00   MDT 49.00   A:[91]      NaN     NaN     2001-05-12 00:00:00-06:00   0.0 45.0    93.0    72.0
... ...     ...     ...             ... ... ...     ...     ...     ...             ... ... ... ...
4968    USGS        9333500     2020-09-20 00:00:00-06:00   MDT 6.04    P       6.04        P       2020-09-20 00:00:00-06:00   0.0 46.0    92.0    83.0
4969    USGS        9333500     2020-09-21 00:00:00-06:00   MDT 6.83    P       6.07        P       2020-09-21 00:00:00-06:00   0.0 47.0    92.0    84.0
4970    USGS        9333500     2020-09-22 00:00:00-06:00   MDT 6.83    P       6.07        P       2020-09-22 00:00:00-06:00   0.0 54.0    84.0    77.0
4971    USGS        9333500     2020-09-23 00:00:00-06:00   MDT 7.68    P       6.10        P       2020-09-23 00:00:00-06:00   0.0 47.0    91.0    87.0
4972    USGS        9333500     2020-09-24 00:00:00-06:00   MDT 9.86    P       6.17        P       2020-09-24 00:00:00-06:00   0.0 43.0    94.0    88.0
4973 rows × 13 columns

This appears to have worked, but is somewhat problematic. Remember that the dd dataset has a 15-minute frequency, but temp_df only has daily data, so we are only using the value from midnight. We should probably use our resampling skills to calculate the median flow value for each date and then merge. In that case we will want to use the index of the grouped data to merge, so we specify left_index=True:

>>> (dd
...  .groupby(pd.Grouper(key='datetime', freq='D'))
...  .median()
...  .merge(temp_df, left_index=True, right_on='DATE')
... )
    site_no     cfs     gage_height DATE                PRCP    TMIN    TMAX    TOBS
492 9333500.0   71.50       NaN     2001-05-07 00:00:00-06:00   0.0 41.0    82.0    55.0
493 9333500.0   69.00       NaN     2001-05-08 00:00:00-06:00   0.0 43.0    85.0    58.0
494 9333500.0   63.50       NaN     2001-05-09 00:00:00-06:00   0.0 36.0    92.0    64.0
495 9333500.0   55.00       NaN     2001-05-10 00:00:00-06:00   0.0 50.0    92.0    67.0
496 9333500.0   55.00       NaN     2001-05-11 00:00:00-06:00   0.0 46.0    87.0    60.0
... ...     ...     ...     ...             ... ... ... ...
6843    9333500.0   6.83        6.07        2020-09-20 00:00:00-06:00   0.0 46.0    92.0    83.0
6844    9333500.0   6.83        6.07        2020-09-21 00:00:00-06:00   0.0 47.0    92.0    84.0
6845    9333500.0   7.39        6.09        2020-09-22 00:00:00-06:00   0.0 54.0    84.0    77.0
6846    9333500.0   7.97        6.11        2020-09-23 00:00:00-06:00   0.0 47.0    91.0    87.0
6847    9333500.0   9.53        6.16        2020-09-24 00:00:00-06:00   0.0 43.0    94.0    88.0
6356 rows × 8 columns

That looks better (and gives us a few more rows of data).

Validating Joined Data

Let's validate that we had a one-to-one join, i.e. each date from the flow data matched up with a single date from the temperature data. We can use the validate parameter to do this:

>>> (dd
...  .groupby(pd.Grouper(key='datetime', freq='D'))
...  .median()
...  .merge(temp_df, left_index=True, right_on='DATE', how='inner',
...         validate='1:1')
... )
    site_no     cfs gage_height DATE                PRCP    TMIN    TMAX    TOBS
492 9333500.0   71.50   NaN     2001-05-07 00:00:00-06:00   0.0 41.0    82.0    55.0
493 9333500.0   69.00   NaN     2001-05-08 00:00:00-06:00   0.0 43.0    85.0    58.0
494 9333500.0   63.50   NaN     2001-05-09 00:00:00-06:00   0.0 36.0    92.0    64.0
495 9333500.0   55.00   NaN     2001-05-10 00:00:00-06:00   0.0 50.0    92.0    67.0
496 9333500.0   55.00   NaN     2001-05-11 00:00:00-06:00   0.0 46.0    87.0    60.0
... ...     ... ...     ...             ... ... ... ...
6843    9333500.0   6.83    6.07        2020-09-20 00:00:00-06:00   0.0 46.0    92.0    83.0
6844    9333500.0   6.83    6.07        2020-09-21 00:00:00-06:00   0.0 47.0    92.0    84.0
6845    9333500.0   7.39    6.09        2020-09-22 00:00:00-06:00   0.0 54.0    84.0    77.0
6846    9333500.0   7.97    6.11        2020-09-23 00:00:00-06:00   0.0 47.0    91.0    87.0
6847    9333500.0   9.53    6.16        2020-09-24 00:00:00-06:00   0.0 43.0    94.0    88.0
6356 rows × 8 columns

Because this did not raise a MergeError, we know that our data had non-repeating date fields.

Visualization of Data Joined with the Pandas Merge Method

I'm a big fan of visualization. Let's visualize the merged time series. We will add on to our merge chain, stick the date in the index (so that we can plot it on the x-axis), pull out the years from 2014 forward, use the cfs (cubic feet per second), gage_height, PRCP, and TOBS columns, interpolate the missing values, do a rolling 15-day average, and plot the result in their own subplot:

import matplotlib.pyplot as plt
fig, ax = plt.subplots(dpi=600, figsize=(10,8))
(dd
 .groupby(pd.Grouper(key='datetime', freq='D'))
 .median()
 .merge(temp_df, left_index=True, right_on='DATE', how='inner',
        validate='1:1')
 .set_index('DATE')
 .loc['2014':,['cfs', 'gage_height', 'PRCP', 'TOBS']]
 .interpolate()
 .rolling(15)
 .mean()
 .plot(subplots=True, ax=ax)
)
fig.suptitle('Dirty Devil Metrics (15-day average)')
fig.tight_layout()

Here’s another plot that is kind of cool once you piece together what is happening. It is a scatterplot of temperature against river flow. I’m coloring this by month of the year. You can see the start of the year at the bottom of the plot. As the year progresses and the temperature warms up, the cfs goes up a little, and then as summer hits, the river almost dries up. During the fall, it starts to cool off and the river starts running higher:

fig, ax = plt.subplots(dpi=600)
dd2 = (dd
 .groupby(pd.Grouper(key='datetime', freq='D'))
 .median()
 .merge(temp_df, left_index=True, right_on='DATE', how='inner',
        validate='1:1')
 .query('cfs < 400')
)
(dd2
 .plot.scatter(x='cfs', y='TOBS', c=dd2.DATE.dt.month,
               ax=ax, cmap='hsv', alpha=.5, s=2)
)
ax.set_title('Observation Temperature (TOBS) '
   'vs River Flow (cubic feet per sec)\nColored by Month')

Summary

Data can often have more utility if we combine it with other data. In the '70s, relational algebra was invented to describe various joins among tabular data. The Pandas merge method of the DataFrame lets us apply these operations to tabular data in the Pandas world. This post explored using the Pandas merge method and the parameters you can use to validate your joining logic.

We would love to hear your feedback. Was this helpful? Did you learn anything new? Tag @ponderdata and @__mharrison__ on Twitter with your insights, and follow both for more content like this. And feel free to suggest new topics you’d like us to dive into for our next Professional Pandas blog post!

Professional Pandas: The Pandas Assign Method and Chaining

2022-08-24T00:00:00-06:00

Republication note: I originally wrote this article for Ponder, where it was published on August 24, 2022. Ponder later removed the post following its acquisition, so I have preserved it here. The Ponder- and Modin-specific code reflects the environment available when the article was written; the pandas concepts remain broadly applicable. View the archived original.

Using .assign to build readable pandas method chains.

Intro

This article on the Pandas assign method is the first in a series of Professional Pandas blog posts that we will put out over the coming months. The goal of this series is to teach best practices about writing professional-grade pandas code. If you have questions or topics that you would like to dive into, please reach out on Twitter to @ponderdata or @__mharrison__, and if you’d like to see the notebook version of this blogpost, you can find it on Github here.

In my opinion, one of the most underutilized methods in Pandas is the .assign method. My take (which may be extreme) is that .assign is the one true way to create a new column or update an existing column.

For those who don't want to read the whole article, here are my reasons for preferring .assign:

You get around the dreaded SettingWithCopyWarning
It enables chaining
It forces you to write readable code that you, your future self, and your colleagues will appreciate
It eases otherwise difficult operations

I get that this "only use .assign" may be extreme, but I encourage you to try it out first before you knock it. If you still hate it after trying it out, please send me your before and after code, because I am curious what happened.

For those still reading, let’s jump in!

Load data

To explore uses of the Pandas assign method, I'm going to use a dataset of meteorological data from a ski resort called Alta.

This data set has some descriptive columns and some daily facts:

PRCP – liquid precipitation (inches)
SNOW – snowfall (inches)
SNWD – snow on the ground (inches)
TMAX – maximum temperature (F)
TMIN – minimum temperature (F)
TOBS – temperature (F) at observation time

Let's load the data and create a tweak_alta function to pull out the columns we want.

>>> import modin.pandas as pd

>>> raw = pd.read_csv('https://github.com/mattharrison/datasets/raw/master'
...                  '/data/alta-noaa-1980-2019.csv', parse_dates=['DATE'])
>>> def tweak_alta(df):
...     return (df
...             .loc[:, ['STATION', 'NAME', 'LATITUDE', 'LONGITUDE', 'ELEVATION', 'DATE',
...                      'PRCP', 'SNOW', 'SNWD', 'TMAX', 'TMIN', 'TOBS']]
...            )
>>> alta = tweak_alta(raw)
>>> alta
STATION NAME    LATITUDE    LONGITUDE   ELEVATION   DATE    PRCP    SNOW    SNWD    TMAX    TMIN    TOBS
0   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-01  0.10    2.0 29.0    38.0    25.0    25.0
1   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-02  0.43    3.0 34.0    27.0    18.0    18.0
2   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-03  0.09    1.0 30.0    27.0    12.0    18.0
3   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-04  0.00    0.0 30.0    31.0    18.0    27.0
4   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-05  0.00    0.0 30.0    34.0    26.0    34.0
... ... ... ... ... ... ... ... ... ... ... ... ...
14155   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-03  0.01    0.0 0.0 74.0    57.0    73.0
14156   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-04  0.00    0.0 0.0 77.0    52.0    74.0
14157   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-05  0.00    0.0 0.0 76.0    54.0    65.0
14158   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-06  0.07    0.0 0.0 66.0    52.0    60.0
14159   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-07  0.00    0.0 0.0 68.0    45.0    64.0
14160 rows x 12 columns

(Notice that we are using the Modin library above, import modin.pandas as pd. However, this same code will run with the Pandas library by only changing that line to import pandas as pd.)

Create a new column

A common operation is adding a column to a dataframe.

Most create a column by using index assignment. This is convenient because:

It looks like a dictionary operation that Python users are used to
It (usually) overwrites the data making it a low overhead operation

There are also some disadvantages:

It might cause a SettingWithCopyWarning
It does not return a dataframe. Hence it cannot be used in a chain of operations
Many users create intermediate variables (and call .copyon the dataframe), thus removing the low overhead benefit

Below I will create a copy of the dataframe, alta_bad, and add a new column, PRCP_CM, that stores daily precipitation measured in centimeters. This is how many folks add columns to dataframes:

alta_bad = alta.copy()
alta_bad['PRCP_CM'] = alta_bad.PRCP * 2.54

Using Pandas assign instead

I prefer using the .assign method to create new columns. It also has some advantages and disadvantages. The advantages are:

It returns a new dataframe so you can chain operations
It sidesteps the SettingWithCopyWarning completely. (As in, you will NEVER see it or deal with it).
It makes your code read like a recipe
It forces you to think about the order of column creation for dependent columns
It allows you to create multiple columns
It lets you merge in a dataframe

The disadvantages of using the .assign method in Pandas:

It copies the dataframe (in the Pandas implementation)
Because it uses keyword parameters, you are limited to column names that are valid Python parameter names (unless you do ** unpacking with a dictionary that has the column stored as a key)
It requires knowledge of lambda functions if you want to work on intermediate dataframes

Here is how I would add the PRCP_CM column using .assign:

>>> (alta
...  .assign(PRCP_CM=alta.PRCP * 2.54)
... )
    STATION NAME    LATITUDE    LONGITUDE   ELEVATION   DATE    PRCP    SNOW    SNWD    TMAX    TMIN    TOBS    PRCP_CM
0   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-01  0.10    2.0 29.0    38.0    25.0    25.0    0.2540
1   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-02  0.43    3.0 34.0    27.0    18.0    18.0    1.0922
2   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-03  0.09    1.0 30.0    27.0    12.0    18.0    0.2286
3   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-04  0.00    0.0 30.0    31.0    18.0    27.0    0.0000
4   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  1980-01-05  0.00    0.0 30.0    34.0    26.0    34.0    0.0000
... ... ... ... ... ... ... ... ... ... ... ... ... ...
14155   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-03  0.01    0.0 0.0 74.0    57.0    73.0    0.0254
14156   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-04  0.00    0.0 0.0 77.0    52.0    74.0    0.0000
14157   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-05  0.00    0.0 0.0 76.0    54.0    65.0    0.0000
14158   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-06  0.07    0.0 0.0 66.0    52.0    60.0    0.1778
14159   USC00420072 ALTA, UT US 40.5905 -111.6369   2660.9  2019-09-07  0.00    0.0 0.0 68.0    45.0    64.0    0.0000
14160 rows x 13 columns

Chaining interlude

Let's take a break from discussing the .assign method for a moment.

This section might be controversial. My advice is to hold back your cringe reflex and hear me out. (Also, I would advise you to try out this style on your own code. My experience is that many who initially are repulsed by this have a wonderful epiphany when they try it, and it completely changes the way they write Pandas code.)

Folks who read my book, Effective Pandas, or follow my Pandas content know that I’m a big proponent of chaining.

What is chaining?

Chaining is writing a series of operations to a dataframe to manipulate it. Each operation works on the result of the previous operation.

One complaint I have against the many “learn to use Pandas” blog posts scattered about the internet is that they teach a method in isolation. In practice, you almost never perform an operation to data in isolation. There are multiple operations that are required to change column types, update values, deal with missing data, aggregate, plot, etc. But generally, you only care about the end result. The intermediate data along the way is not important.

Let’s assume that you wanted to summarize the Alta data by week. You might:

Filter out columns
Group by the date
Aggregate different columns (by different functions)
Create a new column with the NAME of the ski resort
Create a new column with the weekly range

Let’s also assume that the resort was named “Alta Ski Resort” before year 2000 and just “Alta” after.

Most folks would write code like this:

>>> my_cols =  ['STATION', 'NAME', 'LATITUDE', 'LONGITUDE', 'ELEVATION', 'DATE',
...            'PRCP', 'SNOW', 'SNWD', 'TMAX', 'TMIN', 'TOBS']
>>> df1 = alta[my_cols]
>>> grouped_df = df1.groupby(pd.Grouper(key='DATE', freq='W'))
>>> df2 = grouped_df.agg({'PRCP': 'sum', 'TMAX': 'max', 'TMIN': 'min',
...                      'SNOW': 'sum', 'SNWD': 'mean'})
>>> df3 = df2.reset_index()
>>> df3[df3.DATE < '2000']['LOCATION'] = 'Alta Ski Resort'
>>> df3[df3.DATE >= '2000']['LOCATION'] = 'Alta'
>>> df3['T_RANGE'] = df3.TMAX - df3.TMIN
>>> df3
DATE    PRCP    TMAX    TMIN    SNOW    SNWD    T_RANGE
0   1980-01-06  0.68    42.0    12.0    7.0 30.500000   30.0
1   1980-01-13  8.76    33.0    -5.0    63.0    48.142857   38.0
2   1980-01-20  4.68    40.0    15.0    34.0    63.285714   25.0
3   1980-01-27  0.00    43.0    10.0    0.0 57.000000   33.0
4   1980-02-03  2.58    38.0    -6.0    40.0    70.571429   44.0
... ... ... ... ... ... ... ...
2066    2019-08-11  1.09    78.0    42.0    0.0 0.000000    36.0
2067    2019-08-18  0.00    75.0    42.0    0.0 0.000000    33.0
2068    2019-08-25  0.00    76.0    45.0    0.0 0.000000    31.0
2069    2019-09-01  0.02    78.0    41.0    0.0 0.000000    37.0
2070    2019-09-08  0.08    78.0    45.0    0.0 0.000000    33.0
2071 rows × 7 columns

They would find that the lines:

df3[df3.DATE < '2000']['LOCATION'] = 'Alta Ski Resort'
df3[df3.DATE >= '2000']['LOCATION'] = 'Alta'

don’t really work and would probably change them to this (after searching around on the internet):

>>> df3.loc[df3.DATE < '2000','LOCATION'] = 'Alta Ski Resort'
>>> df3.loc[df3.DATE >= '2000','LOCATION'] = 'Alta'
>>> df3
DATE    PRCP    TMAX    TMIN    SNOW    SNWD    LOCATION    T_RANGE SEASON
0   1980-01-06  0.68    42.0    12.0    7.0 30.500000   Alta Ski Resort 30.0    Ski 1979
1   1980-01-13  8.76    33.0    -5.0    63.0    48.142857   Alta Ski Resort 38.0    Ski 1979
2   1980-01-20  4.68    40.0    15.0    34.0    63.285714   Alta Ski Resort 25.0    Ski 1979
3   1980-01-27  0.00    43.0    10.0    0.0 57.000000   Alta Ski Resort 33.0    Ski 1979
4   1980-02-03  2.58    38.0    -6.0    40.0    70.571429   Alta Ski Resort 44.0    Ski 1979
... ... ... ... ... ... ... ... ... ...
2066    2019-08-11  1.09    78.0    42.0    0.0 0.000000    Alta    36.0    Summer 2019
2067    2019-08-18  0.00    75.0    42.0    0.0 0.000000    Alta    33.0    Summer 2019
2068    2019-08-25  0.00    76.0    45.0    0.0 0.000000    Alta    31.0    Summer 2019
2069    2019-09-01  0.02    78.0    41.0    0.0 0.000000    Alta    37.0    Summer 2019
2070    2019-09-08  0.08    78.0    45.0    0.0 0.000000    Alta    33.0    Summer 2019
2071 rows × 9 columns

Now let’s add a ski season to the rows. Generally a ski season starts in November and ends in April.

>>> df3['SEASON'] = 'Summer'
>>> df3.loc[(df3.DATE.dt.month <= 4)
...        |(df3.DATE.dt.month >= 11),  'SEASON'] = 'Ski'
>>> df3
DATE    PRCP    TMAX    TMIN    SNOW    SNWD    LOCATION    T_RANGE SEASON
0   1980-01-06  0.68    42.0    12.0    7.0 30.500000   Alta Ski Resort 30.0    Ski
1   1980-01-13  8.76    33.0    -5.0    63.0    48.142857   Alta Ski Resort 38.0    Ski
2   1980-01-20  4.68    40.0    15.0    34.0    63.285714   Alta Ski Resort 25.0    Ski
3   1980-01-27  0.00    43.0    10.0    0.0 57.000000   Alta Ski Resort 33.0    Ski
4   1980-02-03  2.58    38.0    -6.0    40.0    70.571429   Alta Ski Resort 44.0    Ski
... ... ... ... ... ... ... ... ... ...
2066    2019-08-11  1.09    78.0    42.0    0.0 0.000000    Alta    36.0    Summer
2067    2019-08-18  0.00    75.0    42.0    0.0 0.000000    Alta    33.0    Summer
2068    2019-08-25  0.00    76.0    45.0    0.0 0.000000    Alta    31.0    Summer
2069    2019-09-01  0.02    78.0    41.0    0.0 0.000000    Alta    37.0    Summer
2070    2019-09-08  0.08    78.0    45.0    0.0 0.000000    Alta    33.0    Summer
2071 rows × 9 columns

Finally, let’s add a year to the SEASON:

>>> df3['SEASON'] = df3.SEASON + ' ' \
...                 + df3.DATE.dt.year.astype(str).where(df3.DATE.dt.month > 4,
...                                                     (df3.DATE.dt.year - 1).astype(str))
>>> df3
DATE    PRCP    TMAX    TMIN    SNOW    SNWD    LOCATION    T_RANGE SEASON
0   1980-01-06  0.68    42.0    12.0    7.0 30.500000   Alta Ski Resort 30.0    Ski 1979
1   1980-01-13  8.76    33.0    -5.0    63.0    48.142857   Alta Ski Resort 38.0    Ski 1979
2   1980-01-20  4.68    40.0    15.0    34.0    63.285714   Alta Ski Resort 25.0    Ski 1979
3   1980-01-27  0.00    43.0    10.0    0.0 57.000000   Alta Ski Resort 33.0    Ski 1979
4   1980-02-03  2.58    38.0    -6.0    40.0    70.571429   Alta Ski Resort 44.0    Ski 1979
... ... ... ... ... ... ... ... ... ...
2066    2019-08-11  1.09    78.0    42.0    0.0 0.000000    Alta    36.0    Summer 2019
2067    2019-08-18  0.00    75.0    42.0    0.0 0.000000    Alta    33.0    Summer 2019
2068    2019-08-25  0.00    76.0    45.0    0.0 0.000000    Alta    31.0    Summer 2019
2069    2019-09-01  0.02    78.0    41.0    0.0 0.000000    Alta    37.0    Summer 2019
2070    2019-09-08  0.08    78.0    45.0    0.0 0.000000    Alta    33.0    Summer 2019
2071 rows × 9 columns

This looks ok, and you have probably seen code like this. You’ve probably written code like this if you are reading this post.

It does have some issues. It spans multiple cells, creates unneeded intermediate variables, and might be buggy (note that the LOCATION column initially failed silently).

Let's use the Pandas .assign method and chaining to write it a little differently.

>>> def monthly_summary(df):
...     my_cols =  ['STATION', 'NAME', 'LATITUDE', 'LONGITUDE', 'ELEVATION', 'DATE',
...                'PRCP', 'SNOW', 'SNWD', 'TMAX', 'TMIN', 'TOBS']
...     return (df
...             [my_cols]
...             .groupby(pd.Grouper(key='DATE', freq='W'))
...             .agg({'PRCP': 'sum', 'TMAX': 'max', 'TMIN': 'min', 'SNOW': 'sum', 'SNWD': 'mean'})
...             .reset_index()
...             .assign(LOCATION='Alta',
...                     SEASON='Ski')
...             .assign(LOCATION=lambda df_: df_.LOCATION.where(
...                                                             df_.DATE.dt.year >= 2000
...                                                             ,'Alta Ski Resort'),
...                     T_RANGE=lambda df_: df_.TMAX - df_.TMIN,
...                     SEASON=lambda df_:(df_
...                                        .SEASON
...                                        .where(
...                      (df_.DATE.dt.month <= 4) | (df_.DATE.dt.month >= 11), 'Summer')
...                                        .add(' ')
...                                        .add(df_
...                                             .DATE.dt.year.astype(str)
...                                             .where(df_.DATE.dt.month > 4,
...                                                    (df_.DATE.dt.year - 1).astype(str))))
...                    )
...            )
>>> monthly_summary(alta)
DATE    PRCP    TMAX    TMIN    SNOW    SNWD    LOCATION    SEASON  T_RANGE
0   1980-01-06  0.68    42.0    12.0    7.0 30.500000   Alta Ski Resort Ski 1979    30.0
1   1980-01-13  8.76    33.0    -5.0    63.0    48.142857   Alta Ski Resort Ski 1979    38.0
2   1980-01-20  4.68    40.0    15.0    34.0    63.285714   Alta Ski Resort Ski 1979    25.0
3   1980-01-27  0.00    43.0    10.0    0.0 57.000000   Alta Ski Resort Ski 1979    33.0
4   1980-02-03  2.58    38.0    -6.0    40.0    70.571429   Alta Ski Resort Ski 1979    44.0
... ... ... ... ... ... ... ... ... ...
2066    2019-08-11  1.09    78.0    42.0    0.0 0.000000    Alta    Summer 2019 36.0
2067    2019-08-18  0.00    75.0    42.0    0.0 0.000000    Alta    Summer 2019 33.0
2068    2019-08-25  0.00    76.0    45.0    0.0 0.000000    Alta    Summer 2019 31.0
2069    2019-09-01  0.02    78.0    41.0    0.0 0.000000    Alta    Summer 2019 37.0
2070    2019-09-08  0.08    78.0    45.0    0.0 0.000000    Alta    Summer 2019 33.0
2071 rows × 9 columns

A false positive issue

One common comment that I hear when folks see code that looks like monthly_summary is: "This code must be hard to debug."

I beg to differ. It is written as a chain. But I didn’t write out all the code at once. I wrote one operation at a time. Then I ran the code and validated (debugged) that it was doing what I wanted.

There are a few pleasant aspects of code written this way. First, it is encapsulated in a function. This makes it easy to pull out into a script and use elsewhere. Also, because it is a function, there aren't any intermediate variables hanging around messing up the global namespace. For example, the my_cols variable is garbage collected when the function has finished executing.

Also, this chain reads like a recipe of steps:

Pull out the columns
Groupby by the date with a weekly ('W') frequency
Aggregate the columns as indicated
Push the index (the week date) into a column
Add a temperature range (T_RANGE) column
Add columns for LOCATION and SEASON
Update the LOCATION and SEASON columns with logic

In fact, writing chains forces you to think about each step that you will perform on the data. It is a constraint that I find allows me to write better code. (Again, you might think I’m spewing crazy talk. Try it out and see how it helps with your code.)

This code is also easy to execute. It is all in one place. Another struggle with notebooks is cells strewn with data manipulation logic (that can be executed in arbitrary order). With the chained code, you can come back to this notebook in the future and run this cell with confidence, knowing that you weren’t missing running other random cells from the notebook.

Debugging this code is easy as well. I'm not going to dive into that here as I have devoted a whole chapter to that in my book, Effective Pandas (maybe I will do a future blog post on debugging if there is interest). In short, you can comment out the operations to see previous intermediate results or combine a .pipe call with a breakpoint() to jump into the debugger if you really want to.

Real issues with Pandas assign

Everything is not peaches and cream with the .assign method. There are some real issues with this code. One is that it requires understanding lambda expressions.

However, I think if you want to be a serious user of Pandas, it pays off to take a few minutes to understand lambda functions.

We use lambda expressions to create anonymous functions. These are just functions that we can inline in a call. In fact, we don’t really need to use them. We can define a normal function instead. I could write the line:

            .assign(T_RANGE=lambda df_: df_.TMAX - df_.TMIN,

            .assign(T_RANGE=calc_range,

If I had a calc_range function defined as:

def calc_range(df_):
    return df_.TMAX - df_.TMIN,

(Note I use the variable df_ in my lambda to distinguish it from df that was passed into the monthly_summary function.)

Using a lambda function (or a normal function) with .assign passes in the current state of the dataframe into the function. Because we have aggregated the daily data to a weekly level, we can only calculate the weekly range from the current data and not the original daily data, df.

The lambda accepts the current state of the dataframe and should return a scalar value or a series to use for the new or updated column values.

The lines:

            .assign(...
                    LOCATION='Alta',
                    SEASON='Ski')

Create columns for LOCATION and SEASON with a single value. You can also set them to a series as well. They don't need lambdas because we are not dependent on the current state of the dataframe.

Another "issue" with Pandas assign is that we need a new call to it if we want to update a column again (like we do with LOCATION and SEASON). (Because .assign is leveraging named parameters in Python, we can't have two parameters with the same name).

Complaints that this code is “complicated” are valid, in my opinion. But the complication comes from Pandas’ lack of providing a clean interface for if else logic, not from the chain.

(I personally find the .where method lacking and wish that Pandas had something similar to NumPy's select function for implementing conditional logic.)

Further refactoring

To address the “complicated” nature of creating the LOCATION and SEASON columns, you might consider refactoring the logic to create these columns into a function:

>>> def monthly_summary(df):
...     def get_location(df_):
...         return (df_
...                 .assign(LOCATION='Alta')
...                 .LOCATION
...                 .where(df_.DATE.dt.year >= 2000, 'Alta Ski Resort')
...                )

...     def get_season(df_):
...         return (df_
...                 .assign(SEASON='Ski')
...                 .SEASON
...                 .where((df_.DATE.dt.month <= 4) | (df_.DATE.dt.month >= 11), 'Summer')
...                 .add(' ')
...                 .add(df_
...                    .DATE.dt.year.astype(str)
...                    .where(df_.DATE.dt.month > 4, (df_.DATE.dt.year - 1).astype(str)))
...                )

...     my_cols =  ['STATION', 'NAME', 'LATITUDE', 'LONGITUDE', 'ELEVATION', 'DATE',
...                 'PRCP', 'SNOW', 'SNWD', 'TMAX', 'TMIN', 'TOBS']
...     return (df
...             [my_cols]
...             .groupby(pd.Grouper(key='DATE', freq='W'))
...             .agg({'PRCP': 'sum', 'TMAX': 'max', 'TMIN': 'min', 'SNOW': 'sum', 'SNWD': 'mean'})
...             .reset_index()
...             .assign(T_RANGE=lambda df_: df_.TMAX - df_.TMIN,
...                     LOCATION=get_location,
...                     SEASON=get_season
...                    )
...            )
>>> monthly_summary(alta)
DATE    PRCP    TMAX    TMIN    SNOW    SNWD    T_RANGE LOCATION    SEASON
0   1980-01-06  0.68    42.0    12.0    7.0 30.500000   30.0    Alta Ski Resort Ski 1979
1   1980-01-13  8.76    33.0    -5.0    63.0    48.142857   38.0    Alta Ski Resort Ski 1979
2   1980-01-20  4.68    40.0    15.0    34.0    63.285714   25.0    Alta Ski Resort Ski 1979
3   1980-01-27  0.00    43.0    10.0    0.0 57.000000   33.0    Alta Ski Resort Ski 1979
4   1980-02-03  2.58    38.0    -6.0    40.0    70.571429   44.0    Alta Ski Resort Ski 1979
... ... ... ... ... ... ... ... ... ...
2066    2019-08-11  1.09    78.0    42.0    0.0 0.000000    36.0    Alta    Summer 2019
2067    2019-08-18  0.00    75.0    42.0    0.0 0.000000    33.0    Alta    Summer 2019
2068    2019-08-25  0.00    76.0    45.0    0.0 0.000000    31.0    Alta    Summer 2019
2069    2019-09-01  0.02    78.0    41.0    0.0 0.000000    37.0    Alta    Summer 2019
2070    2019-09-08  0.08    78.0    45.0    0.0 0.000000    33.0    Alta    Summer 2019
2071 rows × 9 columns

Create a plot

Ok, a weekly summary is nice. But the goal of most manipulating data is not the individual step but creating a result that we can use to inspire, inform, or take action. Let’s use Matplotlib to create a plot of snow on the ground during a season.

This is similar to what is found in my book, Pandas Cookbook 1.x, and was inspired by a plot created by Trud Antzee (@Antzee_).

import matplotlib.pyplot as plt

def plot_season(df, year, fig, ax):
    blue = '#99ddee'
    white = '#fff'
    data = (df
     .query(f'SEASON=="Ski {year}"')
    )
    for side in 'top,right,bottom,left'.split(','):
        ax.spines[side].set_visible(False)
    ax.tick_params(axis='both', colors=white)
    ax.tick_params(axis='both', colors=white)
    ax.tick_params(axis='x', which='minor', colors=white)
    ax.fill_between(data.DATE, data.SNWD, color='white')
    ax.set_facecolor(blue)
    ax.set_ylim(0, 150)
    ax.set_title(f'{year} Season Snowfall', fontsize=16, color=white)

def facet_plot(df, years):
    blue = '#99ddee'
    white = '#fff'
    fig, axs = plt.subplots(ncols=2, nrows=(len(years)//2),
                          figsize=(16, 10), facecolor=blue)
    axs = axs.flatten()
    for i, year in enumerate(years):
        plot_season(df, year, fig, axs[i])
    fig.suptitle('Alta Snowfall (@__mharrison__)', fontsize=20, color=white)
    fig.tight_layout()
    return fig

_ = (monthly_summary(alta)
 .pipe(facet_plot, years=['2000', '2005','2010', '2015'])
)

Final code

Thanks for getting this far. I hope you have a better understanding of using the .assign method. Also I hope you appreciate that operations do not occur in isolation.

We showed how to take "traditional" Pandas code, refactor it into chains and functions. I would love to hear your feedback. How are you going to use the Pandas assign method? Please share examples of applying chains like this to your code on Twitter, and tag @ponderdata and @mharrison (and follow us on Twitter for more content like this). What insights did you have as you were doing it?

import matplotlib.pyplot as plt
import modin.pandas as pd

raw = pd.read_csv('https://github.com/mattharrison/datasets/raw/master/data/alta-noaa-1980-2019.csv',
                  parse_dates=['DATE'])

def tweak_alta(df):
    return (df
            .loc[:, ['STATION', 'NAME', 'LATITUDE', 'LONGITUDE', 'ELEVATION', 'DATE',
                     'PRCP', 'SNOW', 'SNWD', 'TMAX', 'TMIN', 'TOBS']]
           )

def monthly_summary(df):
    def get_location(df_):
        return (df_
                .assign(LOCATION='Alta')
                .LOCATION
                .where(df_.DATE.dt.year >= 2000, 'Alta Ski Resort')
               )

    def get_season(df_):
        return (df_
                .assign(SEASON='Ski')
                .SEASON
                .where((df_.DATE.dt.month <= 4) | (df_.DATE.dt.month >= 11), 'Summer')
                .add(' ')
                .add(df_
                   .DATE.dt.year.astype(str)
                   .where(df_.DATE.dt.month > 4, (df_.DATE.dt.year - 1).astype(str)))
               )

    my_cols =  ['STATION', 'NAME', 'LATITUDE', 'LONGITUDE', 'ELEVATION', 'DATE',
                'PRCP', 'SNOW', 'SNWD', 'TMAX', 'TMIN', 'TOBS']
    return (df
            [my_cols]
            .groupby(pd.Grouper(key='DATE', freq='W'))
            .agg({'PRCP': 'sum', 'TMAX': 'max', 'TMIN': 'min', 'SNOW': 'sum', 'SNWD': 'mean'})
            .reset_index()
            .assign(T_RANGE=lambda df_: df_.TMAX - df_.TMIN,
                    LOCATION=get_location,
                    SEASON=get_season
                   )
           )

def plot_season(df, year, fig, ax):
    blue = '#99ddee'
    white = '#fff'
    data = (df
     .query(f'SEASON=="Ski {year}"')
    )
    for side in 'top,right,bottom,left'.split(','):
        ax.spines[side].set_visible(False)
    ax.tick_params(axis='both', colors=white)
    ax.tick_params(axis='both', colors=white)
    ax.tick_params(axis='x', which='minor', colors=white)
    ax.fill_between(data.DATE, data.SNWD, color='white')
    ax.set_facecolor(blue)
    ax.set_ylim(0, 150)
    ax.set_title(f'{year} Season Snowfall', fontsize=16, color=white)

def facet_plot(df, years):
    blue = '#99ddee'
    white = '#fff'
    fig, axs = plt.subplots(ncols=2, nrows=(len(years)//2),
                          figsize=(16, 10), facecolor=blue)
    axs = axs.flatten()
    for i, year in enumerate(years):
        plot_season(df, year, fig, axs[i])
    fig.suptitle('Alta Snowfall (@__mharrison__)', fontsize=20, color=white)
    fig.tight_layout()
    return fig


_ = (raw
     .pipe(tweak_alta)
     .pipe(monthly_summary)
     .pipe(facet_plot, years=['2000', '2005','2010', '2015'])
)

Pandas Series Introduction

2021-06-03T00:00:00-06:00

A Series is used to model one-dimensional data, similar to a list in Python. The Series object also has a few more bits of data, including an index and a name. A common idea through pandas is the notion of an axis. Because a series is one dimensiona-, it has a single axis—the index.

Below is a table of counts of songs artists composed. We will use this to explore how the series type functions:

Artist	Data
0	145
1	142
2	38
3	13

If you wanted to represent this data in pure Python, you could use a data structure similar to the one that follows. It is a dictionary that has a list of the data points stored under the 'data' key. In addition to an entry in the dictionary for the actual data, there is an explicit entry for the corresponding index values for the data (in the 'index' key), as well as an entry for the name of the data (in the 'name' key):

>>> ser = {
...   'index':[0, 1, 2, 3],
...   'data':[145, 142, 38, 13],
...   'name':'songs'
...   }

The get function defined below can pull items out of this data structure based on the index:

>>> def get(ser, idx):
...     value_idx = ser['index'].index(idx)
...     return ser['data'][value_idx]

>>> get(ser, 1)
142

Note

The code samples in this book are shown as if they were typed directly into an interpreter. Lines starting with >>> and ... are interpreter markers for the input prompt and continuation prompt respectively. Lines that are not prefixed by one of those sequences are the output from the interpreter after running the code.

The Python interpreter will print the return value of the last invocation (even if the print statement is missing) automatically. If you desire to use the code samples found in this book, leave the interpreter prompts out.

The index abstraction

This double abstraction of the index seems unnecessary at first glance—a list already has integer indexes. But there is a trick up pandas' sleeves. By allowing non-integer values, the data structure supports other index types such as strings, dates, as well as arbitrarily ordered indices or even duplicate index values.

Below is an example that has string values for the index:

>>> songs = {
...   'index':['Paul', 'John', 'George', 'Ringo'],
...   'data':[145, 142, 38, 13],
...   'name':'counts'
...   }

>>> get(songs, 'John')
142

The index is a core feature of pandas’ data structures given the library’s past in analysis of financial data or time series data. Many of the operations performed on a Series operate directly on the index or by index lookup.

The pandas `Series`

With that background in mind, let’s look at how to create a Series in pandas. It is easy to create a Series object from a list:

>>> import pandas as pd
>>> songs2 = pd.Series([145, 142, 38, 13],
...      name='counts')

>>> songs2
0    145
1    142
2     38
3     13
Name: counts, dtype: int64

When the interpreter prints our series, pandas makes a best effort to format it for the current terminal size. The leftmost column is the index column which contains entries for the index. The generic name for an index is an axis, and the values of the index—0, 1, 2, 3—are called axis labels. The two-dimensional structure in pandas—a DataFrame—has two axes, one for the rows and another for the columns.

The parts of a Series.

The rightmost column in the output contains the values of the series—145, 142, 38, and 13. In this case, they are integers (the console representation says dtype: int64, dtype meaning data type, and int64 meaning 64-bit integer), but in general values of a Series can hold strings, floats, booleans, or arbitrary Python objects. To get the best speed (and to leverage vectorized operations), the values should be of the same type, though this is not required.

It is easy to inspect the index of a series (or data frame), as it is an attribute of the object:

>>> songs2.index
RangeIndex(start=0, stop=4, step=1)

The default values for an index are monotonically increasing integers. songs2 has an integer-based index.

Note

The index can be string based as well, in which case pandas indicates that the datatype for the index is object (not string):

>>> songs3 = pd.Series([145, 142, 38, 13],
...      name='counts',
...      index=['Paul', 'John', 'George', 'Ringo'])

Note that the dtype that we see when we print a Series is the type of the values, not of the index:

>>> songs3
Paul      145
John      142
George     38
Ringo      13
Name: counts, dtype: int64

When we inspect the index attribute, we see that the dtype is object:

>>> songs3.index   # doctest: +NORMALIZE_WHITESPACE
Index(['Paul', 'John', 'George', 'Ringo'],
dtype='object')

The actual data for a series does not have to be numeric or homogeneous. We can insert Python objects into a series:

>>> class Foo:
...     pass

>>> ringo = pd.Series(
...      ['Richard', 'Starkey', 13, Foo()],
...      name='ringo')

>>> ringo  # doctest: +SKIP +NORMALIZE_WHITESPACE
0                             Richard
1                             Starkey
2                                  13
3    <__main__.Foo instance at 0x...>
Name: ringo, dtype: object

In the above case, the dtype-datatype-of the Series is object (meaning a Python object). This can be good or bad.

The object data type is used for strings. But, it is also used for values that have heterogeneous types. If you have just numeric data in a series, you wouldn't want it stored as a Python object, but rather as an int64 or float64, which allow you to do vectorized numeric operations.

If you have time data and it says it has the object type, you probably have strings for the dates. Using strings instead of date types is bad as you don't get the date operations that you would get if the type were datetime64[ns]. Strings, on the other hand, are stored in pandas as object. Don't worry; we will see how to convert types later in the book.

The `NaN` value

A value that may be familiar to NumPy users, but not Python users in general, is NaN. When pandas determines that a series holds numeric values, but it cannot find a number to represent an entry, it will use NaN. This value stands for Not A Number and is usually ignored in arithmetic operations. (Similar to NULL in SQL).

Here is a series that has NaN in it:

>>> import numpy as np
>>> nan_ser = pd.Series([2, np.nan],
...    index=['Ono', 'Clapton'])
>>> nan_ser
Ono        2.0
Clapton    NaN
dtype: float64

Note

One thing to note is that the type of this series is float64, not int64! The type is a float because float64 supports NaN, which int64 does not. When pandas sees numeric data (2) as well as the None, it coerced the 2 to a float value.

Below is an example of how pandas ignores NaN. The .count method, which counts the number of values in a series, disregards NaN. In this case, it indicates that the count of items in the Series is one, one for the value of 2 at index location Ono, ignoring the NaN value at index location Clapton:

>>> nan_ser.count()
1

Note

If you load data from a CSV file, an empty value for an otherwise numeric column will become NaN. Later, methods such as .fillna and .dropna will explain how to deal with NaN.

None, NaN, nan, and null are synonyms in this book when referring to empty or missing data found in a pandas series or data frame.

Optional Integer Support for `NaN`

As of pandas 0.24, there is optional support for the integer type to hold NaN values. The documentation calls this type the nullable integer type. When you create a series, you can pass in dtype='Int64' (note the capitalization):

>>> nan_ser2 = pd.Series([2, None],
...    index=['Ono', 'Clapton'],
...    dtype='Int64')
>>> nan_ser2
Ono           2
Clapton    <NA>
dtype: Int64

Operations on these series still ignore NaN:

>>> nan_ser2.count()
1

Note

You can use the .astype method to convert columns to the nullable integer type. Just use the string 'Int64' as the type:

>>> nan_ser.astype('Int64')
Ono           2
Clapton    <NA>
dtype: Int64

Similar to NumPy

The Series object behaves similarly to a NumPy array. As shown below, both types respond to index operations:

>>> import numpy as np
>>> numpy_ser = np.array([145, 142, 38, 13])
>>> songs3[1]
142
>>> numpy_ser[1]
142

They both have methods in common:

>>> songs3.mean()
84.5
>>> numpy_ser.mean()
84.5

They also both have a notion of a boolean array. A boolean array is a series with the same index as the series you are working with that has boolean values. It can be used as a mask to filter out items. Normal Python lists do not support such fancy index operations:

>>> mask = songs3 > songs3.median()  # boolean array

>>> mask
Paul       True
John       True
George    False
Ringo     False
Name: counts, dtype: bool

Once we have a mask, we can use that as a filter. We just need to pass the mask into an index operation. If the mask has a True value for a given index, the value is kept. Otherwise, the value is dropped. The mask above represents the locations that have a value higher than the median value of the series.

>>> songs3[mask]
Paul    145
John    142
Name: counts, dtype: int64

Filtering a series with a boolean array.

NumPy also has filtering by boolean arrays, but lacks the .median method on an array. Instead, NumPy provides a median function in the NumPy namespace:

>>> numpy_ser[numpy_ser > np.median(numpy_ser)]
array([145, 142])

Note

Both NumPy and pandas have adopted the convention of using import statements in combination with an as statement to rename their imports to two letter acronyms:

>>> import pandas as pd
>>> import numpy as np

Renaming imports provides a slight typing benefit (four fewer characters) while still allowing the user to be explicit with their namespaces.

Be careful, as you may see the following cast about in code samples, blogs, or documentation:

>>> from pandas import *

Though you see star imports frequently used in examples online, I would advise not to use star imports. I never use them in examples or my work. They have the potential to clobber items in your namespace and make tracing the source of a definition more difficult (especially if you have multiple star imports). As the Zen of Python states, “Explicit is better than implicit” [1].

[1]	Type `import this` into an interpreter to see the Zen of Python. Or search for "PEP 20".

Categorical Data

When you load data, you can indicate that the data is categorical. If we know that our data is limited to a few values; we might want to use categorical data. Categorical values have a few benefits:

Use less memory than strings
Can have an ordering
Can perform operations on categories
Enforce membership on values

Categories are not limited to strings; we can also use numbers or datetime values.

To create a category, we pass dtype="category" into the Series constructor. Alternatively, we can call the .as_type("category") method on a series:

>>> s = pd.Series(['m', 'l', 'xs', 's', 'xl'], dtype='category')
>>> s
0     m
1     l
2    xs
3     s
4    xl
dtype: category
Categories (5, object): ['l', 'm', 's', 'xl', 'xs']

If this series represents size, there is a natural ordering as a small is less than a medium. By default, categories don't have an ordering. We can verify this by inspecting the .cat attribute that has various properties:

>>> s.cat.ordered
False

To convert a non-categorical series to an ordered category, we can create a type with the CategoricalDtype constructor and the appropriate parameters. Then we pass this type into the .astype method:

>>> s2 = pd.Series(['m', 'l', 'xs', 's', 'xl'])
>>> size_type = pd.api.types.CategoricalDtype(
...     categories=['s','m','l'], ordered=True)
>>> s3 = s2.astype(size_type)
...
>>> s3
0      m
1      l
2    NaN
3      s
4    NaN
dtype: category
Categories (3, object): ['s' < 'm' < 'l']

In this case, we limited the categories to just 's', 'm', and 'l', but the data had values that were not in those categories. These extra values were replaced with NaN.

If we have ordered categories, we can do comparisons on them:

>>> s3 > 's'
0     True
1     True
2    False
3    False
4    False
dtype: bool

The prior example created a new Series from existing data that was not categorical. We can also add ordering information to categorical data. We just need to make sure that we specify all of the members of the category or pandas will throw a ValueError:

>>> s.cat.reorder_categories(['xs','s','m','l', 'xl'],
...                          ordered=True)
0     m
1     l
2    xs
3     s
4    xl
dtype: category
Categories (5, object): ['xs' < 's' < 'm' < 'l' < 'xl']

Note

String and datetime series have a str and dt attribute that allow us to perform common operations specific to that type. If we convert these types to categorical types, we can still use the str or dt attributes on them:

>>> s3.str.upper()
0      M
1      L
2    NaN
3      S
4    NaN
dtype: object

Summary

The Series object is a one-dimensional data structure. It can hold numerical data, time data, strings, or arbitrary Python objects. If you are dealing with numeric data, using pandas rather than a Python list will give you additional benefits. Pandas is faster, consumes less memory, and comes with built-in methods that are very useful to manipulate the data. Also, the index abstraction allows for accessing values by position or label. A Series can also have empty values and has some similarities to NumPy arrays. It is the basic workhorse of pandas; mastering it will pay dividends.

Pandas Data Structures

2021-06-02T00:00:00-06:00

One of the keys to understanding pandas is to understand the data model. At the core of pandas are two data structures:

Different dimensions of pandas data structures
Data Structure	Dimensionality	Spreadsheet Analog
`Series`	1D	Column
`DataFrame`	2D	Single Sheet

The most widely used data structures are the Series and the DataFrame that deal with array data and tabular data, respectively. An analogy with the spreadsheet world illustrates the basic differences between these types. A DataFrame is similar to a sheet with rows and columns, while a Series is similar to a single column of data.

Figure showing the relation between the main data structures in pandas. Namely, that a data frame can have multiple series.

Diving into these core data structures a little more is useful because a bit of understanding goes a long way towards better use of the library. We will spend a good portion of time discussing the Series and DataFrame. Both the Series and DataFrame share features. For example, they both have an index, which we will need to examine to understand how pandas works.

Also, because the DataFrame can be thought of as a collection of columns that are really Series objects, it is imperative that we have a comprehensive study of the Series first. Additionally (and perhaps odd to some), we will see this when we iterate over rows, and the rows are represented as Series.

Note

Some have compared the data structures to Python lists or dictionaries, and I think this is a stretch that doesn't provide much benefit. Mapping the list and dictionary methods on top of pandas' data structures just leads to confusion.

Summary

The pandas library includes three main data structures and associated functions for manipulating them. This book will focus on the Series and DataFrame. First, we will look at the Series as the DataFrame can be thought of as a collection of Series.

The Future of Corporate Training

2021-04-15T00:00:00-06:00

TLDR - I think in a post-Covid19 society, virtual training will be the default, but live training will return to office campuses and quarterly onsites.

Training Flipped

The last year threw corporate training for a loop. One week I was wrapping up a wonderful Machine Learning course for my biggest client and the next week they had dropped the dozen classes they had planned for the year.

It is never fun to lose your biggest client, but rather than look back, I doubled down and invested in upping my virtual training skills. I read a lot of material, bought equipment, and the feedback has been wonderful:

I have had many programming teachers who were super knowledgeable but didn't know much about pedagogy or vice versa and it was refreshing to be taught by someone who had both sets of skills.

Over all I think this was an excellent course and worthwhile. I like your teaching style and appreciate your willingness to answer questions, even when you had already explained it once before.

I really enjoyed it and learned a lot, thanks!

Default to Work from Home?

The jury is mixed on what the future of work will look like. Tech titans like Netflix and Google are insisted on a return to the office, while Twitter is leading the way pushing for permanent work from home. Between these extremes are the many companies that can operate remotely, but may do so only part time.

I asked my network on Twitter and LinkedIn and got very mixed results. The only take away is that a small percentage of my contacts will be going to the Netflix/Google model. The question is the degree from which they will work from home.

Where does Training Go?

My feeling is that much of training will remain virtual. However, I'm getting feeders from my network that in-person training is also desired. I suspect much of this may migrate to onsites.

I feel that even with my investment into virtual training, in-person is just a bit better. You get to see people (many are Zoom-shy or just Zoom-exhausted), you can help them easier, and they can help each other easier.

If you are looking for virtual or in-person (I'm vaccinated) training for Python or Data Science, reach out. Happy to help!

Best Way to Learn Python in 2021

2021-01-12T00:00:00-07:00

TLDR: There is a great demand for Python skills. There are also lots of ways to learn.

How does the saying go? The best time to learn Python was 20 years ago. The second best time is now.

That's not quite true, Python was a niche language 20 years ago. Today Python is ubiquitous and one of the top languages used to develop applications, analyze data, create predictive models, manage servers, and more.

When I learned Python in 2000, it was frowned upon as a less expressive Perl. It fit my brain, and I never looked back even though I have used Java, C#, Clojure, JavaScript, and other tools since.

A common question I get these days is: what is the best way to learn Python? I'm going to answer that with a staple of data science, a decision tree. The answer really depends, but after teaching Python to thousands over many years, I have a few opinions on the topic.

TLDR: Follow this for Python mastery

(Full disclosure: I make my living selling snake oil. I literally tell and enable people to do amazing things with Python that cure most ails.)

Are You a Corporation?

First things first. If you are a corporation, treat your employees well, and consider corporate training. There is something to be said about getting a team together and having them work through training together. (I said work because training can be intense). Your team will be on the same page at the end of the day. Training will fill in their knowledge gaps. They can address any issues or questions they have immediately with a knowledgeable instructor. Good trainers can facilitate discussions that bring out customers issues and solutions to those quickly.

Corporate Training is the quickest way to get your team on the same page.

Here is a quote from a manager at a large video company:

“Nice breadth, very practical, easy to follow along. Everybody loved it. We already used some of the skills to debug two different production issues. Awesome!”

Seriously, invest in your employees. Here is a common complaint. I talk to many companies that spend money on on-demand courses (full-disclosure: I have created content for many of these courseware providers), and while the content may be good, most employees are not motivated nor incentivized to go through it. The employers are not getting the value out of these courses. This is a typical criticism (in fact, most if not all of my clients have on-demand courses in addition to live training because they want their employees to have certain skills and most employees do not do on-demand.)

One more thing to consider with live corporate training. Not all corporate training is the same. Who would you rather have training you? Someone who is a professional slide reader? Or someone who has worked with the technology for 20+ years and also has had extensive experience tutoring, teaching, and writing?

Going with a quality trainer who has in-the-trenches experience and teaching skills can make a big difference. I generally only train on material that I have used as a consultant or as an employee.

If you are not interested in MetaSnake's offerings, reach out, I know a bunch of other quality trainers that also have more experience than just reciting Powerpoint presentations.

The Best for Individuals

How bad do you want to learn Python? I've come across countless people who claim they want to learn Python, but are not willing to invest anything to get results. The best results come from having a coach to identify weak spots and provide advice and practice to improve those faults.

Coach you probably don't recognize.

You have probably never heard of the coach above. His name is Brandon Payne. Still at a loss? His client is likely the best shooter in basketball history. Steph Curry has won NBA championships, MVPs, and made an amazing 100 three pointers in a row. Why would he need a coach? In Steph's own words:

I appreciate [Brandon's] perspective on things — everything from watching game film and cutting it up to advice and direction on reaching my full potential... And we created a vision for where I wanted to be in terms of ballhandling, footwork, balance and, of course, accountability.”

A coach is a partner who can help with reaching full potential and perhaps more importantly accountability.

Those who are at the top of their game use coaches: CEOs, athletes, entrepreneurs, and programmers. Why? Because they invest in their future. A coach can help identify and cure weaknesses and further strengthen existing skills. I could go on, but the point is that people who are serious about something invest in it, and coaching is probably the best way to improve.

A "free" option for coaching might be a more senior colleague who is willing to let you tag along and pair program. If this includes mentoring along the way, this can be invaluable. Some companies encourage this sort of behavior while others view it as "off task" and a waste of precious developer time.

Workshops & Live Courses

Similar to corporate training, workshops and live courses are the equivalents for an individual. Sometimes smaller companies or startups invest in one-off trainings for individuals when they don't have a whole team to train.

These courses lack many of the bonding and insight moments that come with team training, but if you want to grok something quickly, these are the way to go. Again, an experienced trainer can pinpoint weaknesses or gaps and develop them much quicker than going solo.

These may appear to cost more, but consider they can be like drinking from a firehose. You get comments like this:

I felt like I learned 3 months worth of material in 3 days!

After three days, most students are off and running with their new skills. Thrilled to be using thier new knowledge.

I was once in a training, and after demoing some code, a participant did a face palm. I asked what was the matter and they replied that the one line of code I had just showed them had taken them almost a month to implement. 🤦‍

Projects

Invariably, some are not sold on learning fast and thoroughly. (They have not allocated budget to invest in the future.)

At this point, there is a new requirement for learning Python (or any new skill), sustained motivation. Sitting through a couple of days of training is really easy compared to this, but I digress.

A great way to pick up Python is to choose a project and implement it in Python. In fact, this is how I learned Python. Was my code idiomatic Python when I was done? Nope, I was pretty clueless, but I enjoyed the language features that I did use. Did I have an expert helping me? No, it was the blind leading the blind. Was it quick? Somewhat, but again I wasn't using the tools how I should be. However, it did give me a good chance to play around with the language and compare it to other tools that I was used to.

If you have a hobby project that you are itching to do, or your boss is breathing down your neck about your next task, consider using this as a forcing function to get familiar with Python. There can be some inherent motivation here to learn.

Videos and On-demand Training

For those with sufficient self-motivation, on-demand video courses can be excellent. Again, the quality of the material can vary. You might want to consider if you are acquiring the courses from sources that focus on the topics at hand or if they are trying to be everything to everyone.

Over the years, many have asked me for training and I ask them how big their team is. When the response was "just me", I felt bad, but in the past year I have been developing a library of training for such individuals in addition to my corporate training. (Indeed, this can be complementary to corporate training as review material.)

I think my material is wonderful (but also realize that I'm heavily biased), and am willing to share my thoughts about other platforms and their pros and cons.

Again, the lack of budget might be a concern for some. I offer discounts to individuals not from the world's 30 wealthiest countries (or those laid off due to Covid19 and students). Still, for some, their willingness to invest in learning consists of spending no money but lots of time. (I hope you can see that I think this is the wrong tradeoff, especially for professionals).

In that case, most courses are paid courses. However, there are thousands if not millions of hours of free content on Youtube. These vary in quality of content, audio, and video. However, you can strike gold with these.

Unless you come across some University course (which might lean more theoretical and lacking application or at least experience in the battlefield), you will be picking up nuggets here and there without a coherent learning path. Again, not an effective use of time, in my opinion, but if the only currency you are willing to spend is time, this is an option.

Books

The final category of my learning options is the written word. You should ask yourself (or your company) "what are you willing to invest?". Many companies are willing to reimburse book purchases.

Some of the books that I have written

Why are books a good investment? Because the author was forced to fill out a proposal form that required that they think about taking the end-user, aka the reader, aka you, down a learning path. During the writing process, there are editors and reviewers whose purpose is to ensure that the learning happens.

Having written a few books (and seeing the reader feedback) as well as reading and reviewing many technical books, I'm a huge proponent of books. However, if you just read books, science tells us that you will not retain most of what you read. To retain information, you will need to apply it (or work on getting it into long term storage). This is probably my biggest beef with many books, they do not include exercises, or they are hard to apply (before they leave short-term memory).

Again, there are books of differing quality. I tend to read a lot of Python books, and am more than willing to proffer my opinion.

We are now back at the "lack of budget" folks. One option is the library. However most libraries do not have up-to-date material. Offering to be a technical reviewer might be a way to get your hands on newer material, but generally these are jobs that are requested from you rather than the other way around.

The cheap option, that sadly has suffered in the past few years with the rise of Twitter and blogging platforms like Medium, is the blog. This is the Youtube of written content. There is a lot out there. Some is gold, some is outdated. But, these posts generally lack a learning path and are not great for taking you along on a learning journey.

Get Started Learning

There you have it. My (obviously biased) thoughts around the most effective ways to learn Python after having been involved in almost every aspect of this for the past twenty years. Feel free to reach out on Twitter or LinkedIn to discuss.

Why Write a Book?

2020-11-13T00:00:00-07:00

The New Business Card

I never considered myself an author. But after answering many one-off calls asking for advice on book writing and realizing that I have published over a dozen books on Amazon, I guess I should admit it. I am an author!

Many of the books that I have written over the years.

Authoring has never been easier. Let's chat about some of the reasons for writing a book and see if they might resonate with you.

Note: I'm specifically talking about writing a non-fiction technical book. I've yet to tackle fiction (other than publishing a book of short stories from a book writing class I taught at the local elementary school.)

Lend me your ears and I'll sing you a song

Is there a story inside of you? I just said that I was not referring to fiction, so what does this even mean?

I started my writing journey after preparing and delivering a half-day Python tutorial. I was very excited about the tutorial's content and felt that it was "the tutorial that I wanted when I was learning Python." The tutorial feedback was extremely positive, and the tutorial had been accepted to be offered at another conference.

Rather than just rewriting my slides in preparation for the upcoming conference, I figured why not take this awesome content and turn it into a book? It should not be too hard, plus I had three months before the conference, so this would allow me to double-down and learn the material inside out.

Long story short, it took a bit more than three months to write the book, but this effort became the first version of what is now Illustrated Guide to Python 3. And this kickstarted my path as an author.

This seems to be a common occurrence. A (lead) engineer gets asked what resources they would recommend for ... and they come up empty. There are no good resources (or bikesheds that are their favorite color). So they set out to create their own. I have interviewed many authors who had this experience. They set out to create the resource they desired.

Recognition

After I published my book, I didn't want to make too much noise about it; because technical people realize that the value is in the product not the marketing or sales (how wrong I was).

Then I had the coolest experience, I was walking down the hallway at work, and a colleague approached me and said, "Hey, aren't you the author of Treading On Python?". I was dumbfounded. Without any mention on my part, the Amazon sales engine sold a book to someone who worked a few cubicles away from me. That felt awesome. But even better was that they had high praise for the book. Recognition is something that everyone craves.

Many potential authors want this recognition. Perhaps it can be better described as the author wants a "book with an animal on the cover" (along with their name).

The coveted animal cover.

I'm guilty of that. I have a framed copy of my book with a northern crested newt on the cover. (I wish it were a short-horned lizard, but I digress). This work has since been translated into seven languages and distributed around the world.

That experience of serendipitous recognition has been repeated over the years. I don't have a paparazzi following me, but...

Hustle

Recognition opens doors.

Many authors (or would-be authors) are interested in "the hustle." This could be looking for side gigs, augmenting income, looking for a better job, or starting consulting.

Book writing is an excellent business card.

One of my author friends, Mark Beckner, put it this way in his book The Coder's Path to Wealth and Independence:

If you publish intelligently, your business will be transformed. If you write with the intention that your publication will create unexpected opportunities, it will enable you to move to a new level in your business.

I have found this to be true. And I know and have spoken with many other authors who had this experience.

It is well known that the process of writing helps you to understand a subject better. You have literally written the book on the subject. With self-publishing (or even purchasing bulk copies from your publisher), you can use your book as a cost-effective business card booster. (In the current Covid-19 climate, this is a little harder. I have a bookshelf of books I meant to deliver to (potential) clients this year at conferences.)

This opens the doors to opportunities that would not come your way even if you were an expert in said technology.

Part of the hustle is money. While I know many authors who have sold six figures worth of books, these tend to be outliers. However, many authors I know have indicated that though they earned minimum wage from the writing, they have indirectly profited much more after the book from side gigs, job opportunities, and more.

Taking the Plunge

Having said this, writing a book is a marathon. Many have sprinted out of the gate only to fail. (I've been approached by a few publishers to "become a second author" on an in-progress book, which is code for "the author burned out").

Have you considered writing a book? I've created a course that walks from ideation to holding a book in your hand. It discusses the tradeoffs of self-publishing and going with a publisher, how to outline, what tools to use, how to get distribution, ebook and physical book tips, audiobook creation, and more. I've come full-circle. This is the course I wanted 10 years ago when I set out to write my first book.

Learn to use the assign method in Pandas (or how to deal with SettingWithCopyWarning issues)

2020-04-03T00:00:00-06:00

TLDR: Favor the .assign method, and you will not deal with SettingWithCopyWarning issues.

Master the .assign method and not index assignment!

Bad Code

Today I'm going to show you the solution to the dreaded SettingWithCopyWarning error that you eventually run into, but no one seems to explain it or give you a solution that you can understand.

I just saw another article discussing it today and thought I would share the best solution (that no one else talks about).

Here's the data referred to in the above article:

df = pd.DataFrame({
  'x':[1,5,4,3,4,5],
  'y':[.1,.5,.4,.3,.4,.5],
  'w':[11,15,14,13,14,15]})

The quest with this toy data is to change all values in the y column to 50 when the x column is greater than 3. Most try to do this by using indexing operations to filter out the appropriate rows and then doing an index assignment.

If you do an index operation directly on df, the update does not work. And you see the dreaded SettingWithCopyWarning warning:

df[df['x']>3]['y'] = 50

The dreaded warning!

The issue with this code is that df[df['x']>3] does not return the original dataframe. So the subsequent index assignment, ['y'] = 50, updates the new dataframe and not df. In effect, you create a new dataframe, assign to it, and do not capture it to a variable, so it is immediately available to be garbage collected.

The warning includes a tip to use .loc for indexing instead. Here is an example that works:

df.loc[df['x']>3,'y'] = 50

I recommend that you do not use this solution. I have a better solution below.

Three Rules for Better Pandas

However, I'll add three more rules of thumb that I have found to make pandas code easier to understand and less prone to errors:

Do not mutate objects
Favor chaining operations
Do not use assignment (the = operator) in pandas

The above code mutates df. However, most pandas operations do not mutate. For feeble-minded people like me, mixing mutation and immutability leads to confusing code and errors. My preferred solution to the above is this code:

df.assign(y=df.y.where(~(df.x > 3), 50))

In fact, whenever I create a new column (or update one) in pandas, I try to do it with the .assign method. This code also requires understanding the .where, method which takes some getting used to but obeys my three rules above.

A Chained Example

Here is an example from Chapter 5 of the The Pandas 1.x Cookbook. This code loads vehicle data from the fueleconomy.gov website. It applies a chain of operations to the dataframe when creating the data variable.

First, it filters out the makes that are Ford, Tesla, BMW, or Toyota with this line [fueleco.make.isin(makes)]. (It could have used .loc here as well). Next, it assigns a column, SClass, that is created from an inlined function (a lambda function) with this line .assign(SClass=.... If you use a function with the .assign method, you have access to the intermediate dataframe, referred to as df_ in the code. The lambda function dispatches to the generalize function.

The generalize function is a function I use all over the place when refining data. It takes a column and a list of tuples with match names and new values in it. It also accepts a default value. It loop over the list returns a new series where the match names are replaced with the new names. If there is no match, the default value is used. This function can be useful if you have a bunch of categorical data and want to limit it to a smaller subset:

url = 'https://github.com/mattharrison/datasets/raw/master/data/vehicles.csv.zip'
fueleco = pd.read_csv(url)

def generalize(ser, match_name, default):
    seen = None
    for match, name in match_name:
        mask = ser.str.contains(match)
        if seen is None:
            seen = mask
        else:
            seen |= mask
        ser = ser.where(~mask, name)
    ser = ser.where(seen, default)
    return ser

makes = ['Ford', 'Tesla', 'BMW', 'Toyota']
data = (fueleco
   [fueleco.make.isin(makes)]
   .assign(SClass=lambda df_: generalize(df_.VClass,
    [('Seaters', 'Car'), ('Car', 'Car'), ('Utility', 'SUV'),
     ('Truck', 'Truck'), ('Van', 'Van'), ('van', 'Van'),
     ('Wagon', 'Wagon')], 'other'))
)

data[['make', 'SClass', 'VClass']]

Master the .assign method and gain superpowers!

Conclusion

Try not to mutate data when using pandas. Even the pandas core developers say that you should not use mutation. If you use method chaining (note that index assignment does not work with chaining!), you will not mutate data. You will not have to deal with the SettingWithCopyWarning. Your code will be easier to read because it will read like processing steps.

If you want to see examples of pandas code that obey these rules see my latest book, Effective Pandas or follow me on twitter where I rant on such topics.

Setting up a Python Data Science Environment

2020-03-06T00:00:00-07:00

You can set up your environment using either method with three commands

Here are instructions for getting a Python stack installed on Windows, Mac, and Linux. These examples will use Python 3 (which I recommend as Python 2 is no longer being developed).

You have two main options for installing a Python environment. You can use the Python executable that you can get from the Python website, or you can use the Anaconda distribution. I will show you both. If you are new to Python, only doing "data science" work, or using Windows, I recommend using Anaconda.

Using Anaconda

TLDR: Use these to create an environment using Anaconda

Anaconda is both a product and a company. The company supports Python data tooling and makes the distribution available as a free product, but also as a supported enterprise paid for product. This page will demo the free option, but I'm assuming that the paid product is similar after install. There are versions for both Python 3.x and Python 2.7. Make sure you use the Python 3.x version.

The Anaconda distribution includes "Python" it just has some extras. It uses its own package system to make it simple to create working Python environments on most platforms. The steps you need to take if you want to use it are:

Install Anaconda (or Miniconda)
Create an environment
Install packages into the environment

Install Anaconda

You have two options for installing Anaconda (the product). The first is to install the whole kitchen sink, which is referred to as the "Anaconda Distribution" a 600MB+ download that includes many common libraries for machine learning, data analysis, and visualization. You can download it here:

The other option is to use Miniconda. This is a smaller download that does not include various libraries but rather lets you download and install them on demand. You can download it here:

(Many of my clients work inside a firewall and should use the Anaconda distribution that is available locally inside of their firewall.)

After you have installed Anaconda or Miniconda, you should have an executable called conda available. Test out that you have it. On UNIX platforms (Mac and Linux) open a terminal (if you are on Mac, you can hit Command-space and then type "Terminal" and hit enter). On Windows, go to the start menu and search for "Conda Prompt". Type into the terminal or command prompt:

conda --version

And it should print out the version, something like conda 4.7.5. If it does not do that, then you did not install Anaconda successfully.

Create an Environment

After you have installed the conda utility, you will need to make an "environment." An environment is a sandboxed installation of Python libraries and utilities. The purpose of an environment is to allow per project library dependencies.

For example, I consult with different clients. If I have multiple clients over the years, they might use different versions of libraries. Say, for example, client A is using Pandas 0.19, client B Pandas 0.25, and client C Pandas 1.0.1. Pandas in generally is pretty good about backward compatibility, but it does have breaking changes. Environments allow me to install specific packages for each client project. When I need to work on client A's work, I activate the environment for that project and have access to Pandas 0.19. When I need to shift to client C, I activate the corresponding environment and have access to Pandas 1.0.1.

To create an environment, you need to give it a name. For a machine learning class, you might create an environment named mlclass (you can specify another name if you like). You can also specify a Python version for the environment. To create an environment with Python 3.7 from a terminal or Anaconda Prompt (on Windows) type:

conda create --name mlclass python=3.7

You can create as many environments as you like. I create them on a per-project basis. That way, I can jump into a project and make sure that I can work on it without dependency conflicts.

After you create the environment, you should activate it. This updates the PATH environment variable so that when you invoke python or utilities, you use the executables found in the environment. Type:

conda activate mlclass

After running this, your prompt should indicate that you are in the environment, and it will change to look like this:

(mlclass)

Note that Anaconda, by default, stores its environments under your home directory. On my Mac, it would put it in ~/anaconda3/envs/mlclass.

Install Packages with Conda

At this point, you have installed conda, created an environment, and activated the environment. You are now ready to install some packages. Here is a command to install some of the common data analysis packages:

conda install notebook pandas scikit-learn seaborn xlrd districtdatalabs::yellowbrick py-xgboost notebook conda-forge::shap bokeh graphviz

The packages with colons in their names are specified as channel::package name. Anaconda ships with certain packages, but libraries can create their own "channel" to ship different versions of their library (or even package their library for Anaconda) on a different cadence than the Anaconda release cycle.

At this point, you should be ready to go with a basic installation. To launch Jupyter notebook type:

jupyter notebook

To learn more about the conda tool, check out the User Guide.

Installation with Python

TLDR: Use these commands on Linux or Mac if you do not want to use Anaconda. On Windows replace the second line with env\Scripts\activate

Note: If you are using Windows, it is recommended that you use Anaconda rather than this mechanism. Some of the scientific packages require installing and configuring a cross-compilation development environment, which can be annoying. If you are using a Mac and want to follow these instructions, it is recommended to install Python from Homebrew.

Install Python

If you are on Linux, use your package manager to install Python 3.x. On Mac, use Homebrew to install Python 3.x. If these instructions do not make sense to you, scroll up and use the instructions for Anaconda.

After Python is installed, open a terminal and check the version. Type:

python3 --version

It should say something like python 3.7.3. Make sure that you installed Python 3 and not Python 2.

Create a Virtual Environment

Python 3 ships with the ability to create environments that are very similar to Anaconda environments. However, they are not the same. Python calls its environments "Virtual Environments", so I will follow suit here. A virtual environment will allow you to create a sandbox for Python on a per-project basis.

To create a virtual environment named env in the current directory type:

python3 -m venv env

To activate this environment, type:

source env/bin/activate

Note that source is a UNIX Bash command (so this will not work on Windows. If you really want to use Windows, even though I suggested against it with this mechanism, you would type env\Scripts\activate.bat).

At this point, your prompt should change to something like this:

(env)

Install Packages with Pip

With Python installed and your virtual environment activated, you are ready to install packages. We use the pip tool to install packages from the Python Package Index. If your virtual environment is activated, then pip will install libraries into the virtual environment.

You should probably update pip with a new version. Type:

pip install -U pip

To install a base data science system type:

pip install notebook seaborn pandas scikit-learn xlrd yellowbrick xgboost shap bokeh graphviz

To launch Jupyter Notebook type the following:

jupyter notebook

There is a User Guide for pip as well.

Other Options

Can you use Docker? Probably. The above are the simplest ways of getting your environment set up. There may be an existing Docker container you can leverage. But Docker is usually orthogonal to what I'm teaching. Also, the Docker images will probably be using one of the two techniques I described above.

What about hosted services? Google COLAB is a great service (disclaimer: I'm a subscriber to their PRO account), but requires access to Google COLAB, which many of my clients do not have.

Summary

Setting up a Python development environment is three lines of code once you have Python 3 or Anaconda installed. If you have questions about this, feel free to tweet them. I try to keep this page up to date as I point my clients to it.

What I Use 2019 Edition

2019-11-20T00:00:00-07:00

It seems to be popular to share what gear you use. I'm pretty picky about some things, so I thought I would take a stab at it. Note that I'm just going to address physical hardware in this post, I'll save software for another post.

2015 Macbook Pro

I used Linux exclusively for over 15 years on ThinkPads, but due to the nature of my training and consulting got a Macbook in 2015. It is a decent piece of hardware. The screen went out on me a few years back, and now it seems the keyboard and trackpad are flaky sometimes. A few reboots seem to do the trick. Unless I'm training, I use an external keyboard and mouse, which oddly enough work even when the built-in devices do not. Feb 2020 Update - I have replaced the cable for the trackpad which seems to have solved the problem. Annoying, but easy if you have the correct tools and spend five minutes watching Youtube.

At the time it came out, the screen DPI was a wonder. Now it is a little more commonplace. I do most of my work on this machine. Again I will discuss software later. As of 2019, the 16inch version of this machine has reverted to using the same keyboard as I have. The butterfly keyboards of 2017-2019 were notoriously flaky, and I stayed away from those machines.

The battery life is still decent on this. I take it with me when traveling, and it performs well. I like the aluminum body, which is unparalleled among laptop manufacturers. I wish I could upgrade the SSD and RAM like I could with my ThinkPads. Se la vie.

My next laptop will probably be an Apple due to video editing software.

2017 Surface Book Pro

At some point, I realized that the vast majority of the world (and developers in said world) use Windows. I always missed my UNIX environment when forced to use Windows at work (in the NT days). Windows 10 is useable, and this device also travels with me (as both a backup and if I need to demo something on Windows).

The device comes in handy as a tablet as well. I've used the stylus and found it to be nice when editing books. I was a technical reviewer for Hands-On Unsupervised Learning Using Python and I liked being able to write on top of PDF's with the stylus.

Another area where the Mac excels is the trackpad. It is far superior to the MS version. However, I much prefer the ThinkPad's TrackPoint to either. I know ... gasp, the terror!

Standing Desk

I started standing at work around 2011. My job at the time was pre-standing desk movement. So I needed to use an adapter on top of my sitting desk. I used an Ergo Desktop Kangaroo Pro This worked ok. It was maybe not quite as stable but was able to transform from sitting to a standing position without falling over.

I mostly work from my home office these days and use a Tresanti Desk. I purchased mine at Costco, and I have few regrets. I'm 6'4", and the top hieght works for me. It has a glass top which my kids have used to write notes to me with whiteboard markers.

There are 3 USB outlets at the side, and no this is not a MIDI device, they are only used for charging. On that note, I have an IKEA Signum "Cable management" drilled into the bottom of the backside of the table. In the Signum is an Anker Dual Display Universal Docking Station. As you can tell from the photo, I'm not particularly concerned with eliminating wires. But it would be nice to have USB hub functionality in the desk.

Because it has a glass top mounting stands to it can prove challenging. My monitor stand works with it, but only because I could disassemble the clamp for it.

My other issue with this desk is the size. When I'm running a remote training and using the Mac and Windows, it can get a little crowded. I also use physical notes, so almost every bit of space is used.

I'm a huge fan of taking breaks during the day. Not only do they help your body; they help you with productivity. I recommend getting a standing desk. Do you have to stand eight hours a day? No. I used to stand most of the day. These days I sit most of the day because that feels better for my body. But I will still mix it up. I think the key is to be aware of your body and posture and take care of it.

Another thing I recommend if you are going the standing route is to pick up a padded mat to stand on.

ErgoDox Keyboard

I picked up an ErgoDox keyboard when Drop (the company formerly known as MassDrop) offered them initially. I was using a Unicomp (with a TrackPoint!), but my fingers were telling me that I needed to do something else. Come to think of it, that time probably correlated to when I was writing the first edition of what is now Illustrated Guide to Python 3.

I knew I wanted a keyboard that was ortholinear, mechanical, and split. Bonus points in that the firmware is opensource and you can program it how you want. This keyboard is an acquired taste, but I liked mine so much that when Drop offered them again, I picked up a second (partially out of fear that I would never be able to purchase one again, and partially because I wanted another for home use). Since that time, it appears that the DIY mechanical keyboard group has flourished. You can now purchase completely assembled models, and there are dozens of riffs on split keyboards.

I do not use QWERTY on this keyboard, but rather the Norman layout. Again, this is another acquired taste, but it has worked out for me. Also, I have keys that have a dual purpose. For example, I use emacs a lot. So my left thumb has Control under it. But if I tap it and release, it fires the Escape key.

I typically do not take this keyboard with me when traveling. I took it to PyCon once when it was novel, but it is not really portable (I have a travel case for it) in that it is bulky, and TSA gives you weird looks when you travel with it.

Again, you need to take care of yourself. Your fingers, wrists, and arms can suffer damage from too much use or non-ergonomic use. Take note and do what feels best.

Clearly Superior Technologies Trackball

Award for best-named company of 1995 goes to CST.

When I got my ErgoDox, I was considering modding it to add a TrackPoint. In the meantime, my previous blog had a couple of entries on the ErgoDox, and they were popular enough such that I sent Drop enough purchases of ErgoDoxen that they sent me a "Thank You" gift. The CST trackball mouse was in that gift box. It seemed to be the answer I was looking for. And it fits nicely in between the split keyboard.

I have also added additional "buttons" that plug into the mouse. I have them situated so that I can tap them with my palm. The left palm is a command modifier, and the right palm is the spacemacs leader key.

Apparently, the ball is the size of a billiard ball. I'm not a big pool player, but that seems like a fancy mod.

I also have two of these trackballs (one for "work" and one for home). As of 2019, CST seems to have retired and sold the business. Whether these high-quality devices will remain in production by the new owner remains to be seen. They are no longer available on Amazon. eBay lists one at over $400!

Again, listen to your body. If it tells you something when you are mousing about, maybe you should look into a change. A trackball certainly uses different body movements.

External Monitor

I bought an Auria 27 inch external monitor about ten years ago. At the time, it was a high DPI monitor that got good reviews for the price point. I will probably replace it with a 34 inch 4k monitor soon, but it has served me well.

Vivo Laptop Monitor Mount

I purchased the Vivo Laptop Monitor Mount in an attempt to get a little more real estate on top of my desk. This is a sturdy product that I enjoy, not in spark joy type of enjoy, but it does the job. It also makes standing more ergonomic as it enables me to raise my screen higher.

Audio-Technica BPHS1 Headset and Mic

I do up to a dozen virtual trainings per month, and this mic has served me well. The Audio-Technica BPHS1 has a long cable which allows me to walk around when teaching and not have to worry about standing in front of a mic. I also have a Blue Yeti, but this mic is more of a set it and forget it type. I feel like I'm always fussing with the levels on the Blue Yeti.

This is an XLR mic, so you will need a mic preamp like the Behringer Xenyx 302USB to use it with a USB interface.

I have a long power outlet attached to the Signum, and I hang the mic off of that when it is not in use.

MetaSnake

Your Corporate Training Shouldn't Be a Game of Telephone

Why MetaSnake Training

Exaggerate to Calibrate

Python for Finance: Pandas Resample, Groupby, and Rolling

Data Overview: FDIC BankFind Suite

Data Dictionary

Counts of failures by year

Using Offset Aliases with Pandas Resample

Smoothing By Rougher Aggregations Using Pandas Resample

Resolution Type Over Time

Now You Have Pandas Resample, Groupby, and Rolling Superpowers

I Wrote A Book

Why Write a Book?

You Made More in Two Days Then I Make in...

Audience

Physical Books

Traditional Publishing?

AI and the Death of Books

Will Writing A Book Help You?

Professional Pandas: Indexing with Pandas Loc

Introduction to Pandas Loc

The Data for Our Examples

The Basics on Indexing and Slicing with Pandas

What Is Pandas Loc, and What Inputs Does it Accept?

What the Loc Documentation Says About Allowed Inputs

How to Use Pandas Loc with Each Input Type

Part 1: Using Pandas Loc to Index with Scalars

Part 2: Using Pandas Loc to Index with Lists

Part 3: Using Pandas Loc to Index with Slices

Part 4: Using Pandas Loc to Index with Boolean Arrays

Part 5: Using Pandas Loc to Index with Indexes

Part 6: Using Pandas Loc with Functions as Indexers

A Practical Example of Pandas Loc

Using Loc with Modin

Conclusion

Professional Pandas: The Pandas Merge Method

Intro

Joining Data

Terminology Refresher: Method vs. Function

A Quick Aside on the Python Language

End of the Aside

A Short Overview of Joining

Left Join

Right Join

Inner Join

Outer Join

Cross Join

Validation Using the Pandas Merge Method

Example: Joining Dirty Devil Flow and Weather Data Using the Pandas Merge Method

Joining Data

Validating Joined Data

Visualization of Data Joined with the Pandas Merge Method

Summary

Professional Pandas: The Pandas Assign Method and Chaining

Intro

Load data

Create a new column

Using Pandas assign instead

Chaining interlude

A false positive issue

Real issues with Pandas assign

Further refactoring

Create a plot

Final code

Pandas Series Introduction

The index abstraction

The pandas Series

The NaN value

Optional Integer Support for NaN

Similar to NumPy

Categorical Data

Summary

Pandas Data Structures

Summary

The Future of Corporate Training

Training Flipped

Default to Work from Home?

Where does Training Go?

Best Way to Learn Python in 2021

The pandas `Series`

The `NaN` value

Optional Integer Support for `NaN`