Extracting and parse baseball events data

retrosheetpy is the command-line utility to extract, and parse Retrosheet events data. Under the hood it is a wrapper of Chadwick command-line utlities.

Prerequisites

First, install chadwick tools. You can find instructions how to do so, here.

Second, create a virtual Python environment. I use pyenv to manage my virtual environments, but you can use whichever tool you prefer. With pyenv you would do something like:

pyenv versions # Check which Python versions you have installed. This package uses Python 3.11
pyenv virtualenv 3.11.8 <name of environment> # Create an environment
pyenv activate <name of environment>

How it works

The utility works in a directory, which you have created for retrosheet data. You would do something like this:

cd ~
mkdir retrosheet
cd retrosheet

Utility creates .config.json file, which tracks your working directory structure and paths to chadwick utilities. To initiate this process, simply run:

retrosheetpy init

This will check if you have chadwick utilities installed, and will create the directory structure for you.

After that, you can start extracting and parsing events files.

Available commands

I will describe my process, but you can do it slightly differently. Here is what I normally do.

Get events files

Run (in your dedicated directory and with virtual environemnt activated)

retrosheetpy get-events --start 2023 --end 2023

This will download zip file from Retrosheet for 2023 season, unpack it and put it into raw_data directory, creating a dedicated 2023 folder. Events files can be used for two purposes:

• Parse games logs
• Parse events

So once you've downloaded the files, you can

Parse Games Logs and save to csv file

You will notice that csv_output folder has already been created. That's where the csv files will be kept. In order to parse games logs, run:

retrosheetpy logs --start 2023 --end 2023

Once finished running, you will see a new folder callled 2023 in csv_output folder, and 2023-games-logs.csv file there. Parsing may sometimes fail, or throw out warnings, like, the data type cannot be read. Unfortunately, what I've noticed is that sometimes there maybe a mistyping in source events files. For example, I was running a parsing of 1950-2023 seasons, and all of them run well except for 1960 season, where in the numeric column there was a string. I am not sure it is possible to catch such random errors programmatically, or I haven't figure out a way to do so. If something like this happens (and older seasons may be prone to this), you will have to figure out the problem and re-run retrosheetpy.

Parse Events Logs and save to csv file

If you want to parse events logs, then you will need to run:

retrosheetpy events --start 2023 --end 2023

This will work similar to games logs, you will just see a different csv file in csv_output folder, 2023-events-logs.csv

You're done now! You can use python and pandas for further analysis of the data.

(Experimental) Dump data into a database

I've developed this mainly to support my own setup. I might or might not try to develop a more generic solution. If you use PostgreSQL, like I do, you might want to tweek models/db.py file to create your own connection string. Before doing anything here, you would need to create two tables in your database. Scripts are provided in the sql/ folder. I have figured out column types, so that you don't have to, but feel free to adjust to your needs. After tables are created, and connection string is according to your setup, simply run:

retrosheetpy db -g --start 2023 --end 2023 # to dump game logs to database, or
retrosheetpy db -e --start 2023 --end 2023 # to dump events logs

It will append data to your tables, so be careful, you might need to track what seasons you already have in your database, so that you don't append duplicates. Again, I might find the time to develop a more generic solution, but as quick and dirty solution, it works for me now (and I am trying to fight my perfectionism!).

That's it! You have 2023 events data in your database and you can start your analysis.

Good luck!

P.S.

The utility is not perfect, and many things can be improved, but I have so little time to do this. If you like it, and feel like you want to collaborate on this, I am more than happy to do this.