Introduction¶
We want to analyze the commit history of a small demo Git repository. As analysis task, we want to know who are the Top 10 committers and how the commits are distributed among them. This could be handy if you want to identify the main committers of a project e. g. to send them a gift at Christmas 😉 .
The Git repository itself already contains all the information we need for this. We use GitPython to talk to the repository directly and hand the result straight to Pandas – no extra tooling or server required.
So let’s go!
Preparation¶
This notebook assumes that
- you checked out the demo repository https://github.com/feststelltaste/demo-repo next to this project (as a sibling directory, e. g. via
git clone https://github.com/feststelltaste/demo-repo.git ../demo-repo); it contains a small, synthetic commit history (2014-2016) with intentionally messy committer names/emails, generated with the demo-repo-gen scripts - you use a standard Anaconda installation with Python 3+
- you installed GitPython (
pip install gitpython) to access the Git repository
If everything is set up, we just import the usual suspects: git (GitPython) for connecting to the Git repository and Pandas for data analysis. We also want to plot some graphics later on, so we import matplotlib accordingly as the convention suggests.
import git
import pandas as pd
from io import StringIO
import matplotlib.pyplot as plt
# display graphics directly in the notebook
%matplotlib inline
Data input¶
We need some data to get started. We simply ask Git itself. GitPython gives us access to the underlying Git installation via repo.git, so we can call any Git subcommand as if it were a regular Python method and capture its output as a plain string.
We only need the raw committer’s name (%cn) and raw email address (%ce) of every commit, separated by a tabulator. The --all option includes commits reachable from all local refs, rather than only the currently checked-out branch. We read the resulting Git log directly from memory with StringIO – no intermediate file needed.
GIT_REPO_PATH = r'../../demo-repo/'
repo = git.Repo(GIT_REPO_PATH)
git_log = repo.git.log('--all', '--pretty=format:%cn\t%ce')
# just show the first 80 characters
git_log[:80]
'rfalk\trfalk@example.com\nRobin Falk\trfalk@example.org\nrfalk\trfalk@example.com\nRob'
The query returns all commits reachable from the repository’s refs, together with their committer names and email addresses. We put this tabular data into a Pandas DataFrame via read_csv and the in-memory buffer.
commits = pd.read_csv(StringIO(git_log),
sep="\t",
header=None,
names=['name', 'email']
)
commits.head()
| name | ||
|---|---|---|
| 0 | rfalk | rfalk@example.com |
| 1 | Robin Falk | rfalk@example.org |
| 2 | rfalk | rfalk@example.com |
| 3 | Robin Falk | rfalk@example,org |
| 4 | rfalk | rfalk@example.com |
Familiarization¶
First, I like to check the raw data a little bit. I often do this by first having a look at the data types the data source is returning. It’s a good starting point to check that Pandas recognizes the data types accordingly. You can also use this approach to check for skewed data columns very quickly (especially necessary when reading CSV or Excel files): If there should be a column with a specific data type (e. g. because the documentation of the dataset said so), the data type should be recognized automatically as specified. If not, there is a high probability that the imported data source isn’t correct (and we have a data quality problem).
commits.dtypes
name str email str dtype: object
That’s OK for our simple scenario. The two columns with texts are objects – nothing spectacular.
In the next step, I always like to get a “feeling” of all the data. Primarily, I want to get a quick impression of the data quality again. It could always be that there is “dirty data” in the dataset or that there are outliers that would screw up the analysis. With such a small amount of data we have, we can simply list all unique values that occur in the columns. I just list the top 10’s for both columns.
commits['name'].value_counts()[0:10]
name rfalk 29 Priya Chandran 27 Tomasz Kowalski 18 Erin Novak 17 Robin Falk 15 Sam Whitfield 12 Devon Okafor 12 R. Falk 5 tkowalski 5 swhitfield 4 Name: count, dtype: int64
OK, at first glance, something seems awkward. Let’s have a look at the email addresses.
commits['email'].value_counts()[0:10]
email rfalk@example.com 30 erin.novak@example.com 17 priya.chandran@example.com 17 devon.okafor@example.org 12 tomasz.kowalski@example.org 12 rfalk@example.org 10 priya.chandran@example.net 10 tkowalski@example.com 9 sam.whitfield@example.com 7 rfalk@example,org 5 Name: count, dtype: int64
OK, the bad feeling is strengthening. We might have a problem with committers using multiple names and email addresses. We take a look at this later on.
Let’s also check for other data-quality issues that could affect later analyses of this Git repository. One useful check is to look for malformed or non-standard email addresses.
email_pattern = r'^[^@\s]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$'
non_standard_emails = commits.loc[
~commits['email'].str.fullmatch(email_pattern), 'email'
].value_counts().rename_axis('email').to_frame(name='commits')
non_standard_emails
| commits | |
|---|---|
| rfalk@example,org | 5 |
| rfalk@DESKTOP-7GK3QAF | 4 |
| jenkins@build-server-03 | 2 |
| tkowalski@localhost | 2 |
| root@ip-10-0-1-23 | 2 |
| vagrant@vagrant-ubuntu-trusty-64 | 1 |
The output contains one clearly malformed address (rfalk@example,org) and several machine-local addresses whose domains have no public suffix, such as tkowalski@localhost. These values should not simply be discarded: they still identify commits and can help connect aliases during cleansing. The check is deliberately pragmatic rather than a complete validation of every address allowed by the email standards.
Interlude – begin¶
In the interlude section, I take you to a short, mostly undocumented excursion with probably messy code (don’t do this at home!) to make a point. If you like, you can skip that section.
Goal: Create a diagram that shows the relationship between committer names and email addresses.
I need a unique index for each committer name and have to calculate the number of different email addresses per name.
grouped_by_committers = commits[['name', 'email']]\
.drop_duplicates().groupby('name').count()\
.sort_values('email', ascending=False).reset_index().reset_index()
grouped_by_committers.head()
| index | name | ||
|---|---|---|---|
| 0 | 0 | Robin Falk | 2 |
| 1 | 1 | Sam Whitfield | 2 |
| 2 | 2 | rfalk | 2 |
| 3 | 3 | tkowalski | 2 |
| 4 | 4 | Priya Chandran | 2 |
Same procedure for the email addresses.
grouped_by_email = commits[['name', 'email']]\
.drop_duplicates().groupby('email').count()\
.sort_values('name', ascending=False).reset_index().reset_index()
grouped_by_email.head()
| index | name | ||
|---|---|---|---|
| 0 | 0 | rfalk@example.com | 2 |
| 1 | 1 | tkowalski@example.com | 2 |
| 2 | 2 | 84213+quietdev@users.noreply.example.com | 1 |
| 3 | 3 | ada.sorensen@example.com | 1 |
| 4 | 4 | 11009+coderabbit99@users.noreply.example.com | 1 |
Then I merge the two DataFrames with a subset of the original data. I get an index and occurrence count for each committer name and email address. I only need values that occur in multiple relationships, so I check for counts greater than one.
plot_data = commits.drop_duplicates()\
.merge(grouped_by_committers, left_on='name', right_on="name", suffixes=["", "_from_committers"], how="outer")\
.merge(grouped_by_email, left_on='email', right_on="email", suffixes=["", "_from_emails"], how="outer")
plot_data = plot_data[\
(plot_data['email_from_committers'] > 1) | \
(plot_data['name_from_emails'] > 1)]
plot_data
| name | index | email_from_committers | index_from_emails | name_from_emails | ||
|---|---|---|---|---|---|---|
| 19 | Priya Chandran | priya.chandran@example.com | 4 | 2 | 19 | 1 |
| 20 | Priya Chandran | priya.chandran@example.net | 4 | 2 | 23 | 1 |
| 22 | rfalk | rfalk@DESKTOP-7GK3QAF | 2 | 2 | 25 | 1 |
| 23 | Robin Falk | rfalk@example,org | 0 | 2 | 24 | 1 |
| 24 | R. Falk | rfalk@example.com | 18 | 1 | 0 | 2 |
| 25 | rfalk | rfalk@example.com | 2 | 2 | 0 | 2 |
| 26 | Robin Falk | rfalk@example.org | 0 | 2 | 29 | 1 |
| 28 | Sam Whitfield | sam.whitfield@example.com | 1 | 2 | 27 | 1 |
| 31 | Sam Whitfield | swhitfield@example.org | 1 | 2 | 31 | 1 |
| 33 | Tomasz Kowalski | tkowalski@example.com | 5 | 2 | 1 | 2 |
| 34 | tkowalski | tkowalski@example.com | 3 | 2 | 1 | 2 |
| 35 | tkowalski | tkowalski@localhost | 3 | 2 | 33 | 1 |
| 36 | Tomasz Kowalski | tomasz.kowalski@example.org | 5 | 2 | 34 | 1 |
I just add some nicely normalized indexes for plotting (note: there might be a method that’s easier)
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(plot_data['index'])
plot_data['normalized_index_name'] = le.transform(plot_data['index']) * 10
le.fit(plot_data['index_from_emails'])
plot_data['normalized_index_email'] = le.transform(plot_data['index_from_emails']) * 10
plot_data.head()
| name | index | email_from_committers | index_from_emails | name_from_emails | normalized_index_name | normalized_index_email | ||
|---|---|---|---|---|---|---|---|---|
| 19 | Priya Chandran | priya.chandran@example.com | 4 | 2 | 19 | 1 | 40 | 20 |
| 20 | Priya Chandran | priya.chandran@example.net | 4 | 2 | 23 | 1 | 40 | 30 |
| 22 | rfalk | rfalk@DESKTOP-7GK3QAF | 2 | 2 | 25 | 1 | 20 | 50 |
| 23 | Robin Falk | rfalk@example,org | 0 | 2 | 24 | 1 | 0 | 40 |
| 24 | R. Falk | rfalk@example.com | 18 | 1 | 0 | 2 | 60 | 0 |
Plot an assignment table with the relationships between committer names and email addresses.
fig1 = plt.figure(facecolor='white')
ax1 = plt.axes(frameon=False)
ax1.set_frame_on(False)
ax1.get_xaxis().tick_bottom()
ax1.axes.get_yaxis().set_visible(False)
ax1.axes.get_xaxis().set_visible(False)
# simply plot all the data (imperfection: duplicated will be displayed in bold font)
for data in plot_data.iterrows():
row = data[1]
plt.text(0, row['normalized_index_name'], row['name'], fontsize=15, horizontalalignment="right")
plt.text(1, row['normalized_index_email'], row['email'], fontsize=15, horizontalalignment="left")
plt.plot([0,1],[row['normalized_index_name'],row['normalized_index_email']],'grey', linewidth=1.0)
Here we see that committers can use multiple names and email addresses. There is also a pattern we can use to improve this synthetic data.
Interlude – end¶
If you skipped the interlude section: I just demonstrated that a committer can use different email addresses, and that a single email address can occur with different committer names.
Data Wrangling¶
The situation above is a typical case of a little data messiness and – to demotivate you – absolutely normal. So we have to do some data correction before we start our analysis. Otherwise, we would ignore reality completely and deliver wrong results. This could damage our reputation as a data analyst and is something we have to avoid at all costs!
We want to reduce the duplicate identities caused by the same committer using multiple names or email addresses. In this synthetic dataset, the part of the email address before @ follows a useful pattern, so we use it as a tentative person identifier. We also remove optional +... address tags.
This is deliberately a demo heuristic, not general-purpose identity resolution: different people can have the same email local part on different domains, and one person can use unrelated addresses. For a real repository, prefer a reviewed .mailmap or an explicit identity mapping.
commits['nickname'] = commits['email'].apply(
lambda email: email.split("@", 1)[0].split("+", 1)[0]
)
commits.head()
| name | nickname | ||
|---|---|---|---|
| 0 | rfalk | rfalk@example.com | rfalk |
| 1 | Robin Falk | rfalk@example.org | rfalk |
| 2 | rfalk | rfalk@example.com | rfalk |
| 3 | Robin Falk | rfalk@example,org | rfalk |
| 4 | rfalk | rfalk@example.com | rfalk |
Next, we choose a display name for each nickname group. The heuristic prefers names containing whitespace (treated here as full names), then the most frequent candidate, and finally the longest candidate as a deterministic tie-breaker. These assumptions work for the synthetic demo data but can be ambiguous in real data.
def determine_real_name(names):
counts = names.value_counts()
full_names = [name for name in counts.index if " " in name]
candidates = full_names or list(counts.index)
return min(
candidates,
key=lambda name: (-counts[name], -len(name), name.casefold())
)
commits_grouped = commits[['nickname', 'name']].groupby(['nickname']).agg(determine_real_name)
commits_grouped = commits_grouped.rename(columns={'name' : 'real_name'})
commits_grouped.head()
| real_name | |
|---|---|
| nickname | |
| 11009 | coderabbit99 |
| 55210 | octobuild |
| 84213 | quietdev |
| ada.sorensen | Ada Sorensen |
| bram.jansen | Bram Jansen |
That looks great! Now we switch back to our previous DataFrame by joining in the new information.
commits = commits.merge(commits_grouped, left_on='nickname', right_index=True)
# drop duplicated for better displaying
commits.drop_duplicates().head()
| name | nickname | real_name | ||
|---|---|---|---|---|
| 0 | rfalk | rfalk@example.com | rfalk | Robin Falk |
| 1 | Robin Falk | rfalk@example.org | rfalk | Robin Falk |
| 3 | Robin Falk | rfalk@example,org | rfalk | Robin Falk |
| 5 | Erin Novak | erin.novak@example.com | erin.novak | Erin Novak |
| 10 | R. Falk | rfalk@example.com | rfalk | Robin Falk |
That is sufficient identity normalization for this synthetic dataset.
Analysis¶
Now that we have cleaned the demo data, we can produce some new insights.
Top 10 committers¶
First, we group by the normalized committer name, count the rows (one row per commit), sort the result, and display the Top 10 committers.
committers = commits.groupby('real_name').size()\
.to_frame(name='commits')\
.sort_values('commits', ascending=False)
committers.head(10)
| commits | |
|---|---|
| real_name | |
| Robin Falk | 49 |
| Priya Chandran | 27 |
| Tomasz Kowalski | 23 |
| Erin Novak | 17 |
| Sam Whitfield | 16 |
| Devon Okafor | 12 |
| Wendell Cho | 4 |
| Casper Voss | 3 |
| coderabbit99 | 3 |
| Talia Brooks | 3 |
Commit distribution¶
Next, we create a pie chart to get an impression of how commits are distributed among committers.
committers['commits'].plot(kind='pie')
<Axes: >
That chart is difficult to read because it contains many committers with only a few commits. We use the 75th percentile of the per-committer commit counts as a display threshold: committers above it remain separate, while committers at or below it are combined as Others. This does not mean that the selected committers created 75% of the commits or code.
committers_description = committers.describe()
committers_description
| commits | |
|---|---|
| count | 30.000000 |
| mean | 6.400000 |
| std | 10.558866 |
| min | 1.000000 |
| 25% | 1.250000 |
| 50% | 2.000000 |
| 75% | 3.000000 |
| max | 49.000000 |
We take the 75th percentile of the per-committer commit counts as the threshold.
threshold = committers_description.loc['75%', 'commits']
threshold
np.float64(3.0)
Committers with commit counts at or below this threshold will be grouped into Others.
minor_committers = committers[committers['commits'] <= threshold]
minor_committers.head()
| commits | |
|---|---|
| real_name | |
| Casper Voss | 3 |
| coderabbit99 | 3 |
| Talia Brooks | 3 |
| Marta Kowal | 3 |
| Owen Michaud | 3 |
These are the entries we want to combine into the new Others section. We preserve their total number of commits.
others_commit_count = minor_committers['commits'].sum()
others_commit_count
np.int64(44)
We select the committers whose commit counts are above the threshold.
main_committers = committers[committers['commits'] > threshold].copy()
main_committers
| commits | |
|---|---|
| real_name | |
| Robin Falk | 49 |
| Priya Chandran | 27 |
| Tomasz Kowalski | 23 |
| Erin Novak | 17 |
| Sam Whitfield | 16 |
| Devon Okafor | 12 |
| Wendell Cho | 4 |
The resulting table contains only committers above the display threshold.
main_committers
| commits | |
|---|---|
| real_name | |
| Robin Falk | 49 |
| Priya Chandran | 27 |
| Tomasz Kowalski | 23 |
| Erin Novak | 17 |
| Sam Whitfield | 16 |
| Devon Okafor | 12 |
| Wendell Cho | 4 |
Finally, we add the combined Others row.
main_committers.loc["Others", "commits"] = others_commit_count
main_committers
| commits | |
|---|---|
| real_name | |
| Robin Falk | 49.0 |
| Priya Chandran | 27.0 |
| Tomasz Kowalski | 23.0 |
| Erin Novak | 17.0 |
| Sam Whitfield | 16.0 |
| Devon Okafor | 12.0 |
| Wendell Cho | 4.0 |
| Others | 44.0 |
We redraw the chart with some styling and minor adjustments.
# some configuration for displaying nice diagrams
plt.style.use('fivethirtyeight')
plt.figure(facecolor='white')
ax = main_committers['commits'].plot(
kind='pie', figsize=(6,6), title="Main committers",
autopct='%.2f', fontsize=12)
# get rid of the distracting label for the y-axis
ax.set_ylabel("")
Text(0, 0.5, '')
Summary¶
This example illustrates some of the difficulties in working with committer identity data. For the synthetic dataset, we reduced duplicate identities with a documented heuristic and transformed a crowded pie chart into a more readable one.
This shows that a local Git clone and a few lines of GitPython/Pandas code are already enough to get meaningful insights into your project’s committers – no graph database or extra scanning tool required.
Really cool blog post. In Neo4j you could have done.
MATCH (author:Author)-[:COMMITED]-> (commit:Commit)
RETURN author.name as name, collect(distinct author.email) as email, count(*) as commits
you could then either marked one of the autors as :Person and connected the others to it or create a new “Person” node and connect all of them to it
MATCH (author:Author)-[:COMMITED]-> (commit:Commit)
WITH author, count(*) as commits order by commits desc // ordered
WITH author.name as name, collect(distinct author) as emails
WITH head(emails) as main, tail(emails) as rest
SET main:Person
FOREACH (a in rest | MERGE (a)-[:ALIAS_FOR]->(main) )
Thanks for the tip! I like the idea of the “higher level concept” of a person. I have to admit that it felt a little bit wrong to do it in Pandas. I think it’s time to read a Neo4j book 😉