Introduction

In the last notebook, I showed you how easy it is to connect jQAssistant/neo4j with Python Pandas/py2neo. In this notebook (also available on GitHub), I show you a (at first glance) simple analysis of the Git repository https://github.com/feststelltaste/spring-petclinic. This repository is a fork of the demo repository for jQAssistant (https://github.com/feststelltaste/spring-petclinic) therefore it integrates jQAssistant already.

As analysis task, we want to know who are the Top 10 committers and how the distribution of the commits is. This could be handy if you want to identify your main contributors of a project e. g. to send them a gift at Christmas šŸ˜‰ .

But first, you might ask yourself: “Why do I need a fully fledged data analysis framework like Pandas for such a simple task? Are there no standard tools out there?” Well, I’ll show you why (OK, and you got me there: I needed another reason to go deeper with Python, Pandas, jQAssistant and Neo4j to get some serious software data analysis started)

So let’s go!

Preparation

This notebook assumes that

  • there is a running Neo4j server with the default configuration.
  • the graph database is filled with the scan results of jQAssistant (happens for the repository above automatically with an mvn clean install)
  • you use a standard Anaconda installation with Python 3+
  • you installed the py2neo connector

If everything is set up, we just import the usual suspects: py2neo for connecting to Neo4j and Pandas for data analysis. We also want to plot some graphics later on, so we import matplotlib accordingly as the convention suggests.

InĀ [1]:
import py2neo
import pandas as pd
import matplotlib.pyplot as plt
# display graphics directly in the notebook
%matplotlib inline

Data input

We need some data to get started. Luckily, we have jQAssistant at our hand. It’s integrated into the build process of Spring PetClinic repository above and scanned the Git repository information automatically with every executed build.

So let’s query our almighty Neo4j graph database that holds all the structural data about the software project.

InĀ [2]:
graph = py2neo.Graph()
query = """
MATCH (author:Author)-[:COMMITED]-> (commit:Commit)
RETURN author.name as name, author.email as email
"""
result = graph.data(query)
# just how first three entries
result[0:3]
Out[2]:
[{'email': 'feststelltaste@googlemail.com', 'name': 'Markus Harrer'},
 {'email': 'feststelltaste@googlemail.com', 'name': 'feststelltaste'},
 {'email': 'feststelltaste@googlemail.com', 'name': 'feststelltaste'}]

The query returns all commits with their authors and the author’s email addresses. We get some nice, tabular data that we put into Pandas’s DataFrame.

InĀ [3]:
commits = pd.DataFrame(result)
commits.head()
Out[3]:
emailname
0feststelltaste@googlemail.comMarkus Harrer
1feststelltaste@googlemail.comfeststelltaste
2feststelltaste@googlemail.comfeststelltaste
3feststelltaste@googlemail.comfeststelltaste
4feststelltaste@googlemail.comfeststelltaste

Familiarization

First, I like to check the raw data a little bit. I often do this by first having a look at the data types the data source is returning. It’s a good starting point to check that Pandas recognizes the data types accordingly. You can also use this approach to check for skewed data columns very quickly (especially necessary when reading CSV or Excel files): If there should be a column with a specific data type (e. g. because the documentation of the dataset said so), the data type should be recognized automatically as specified. If not, there is a high probability that the imported data source isn’t correct (and we have a data quality problem).

InĀ [4]:
commits.dtypes
Out[4]:
email    object
name     object
dtype: object

That’s OK for our simple scenario. The two columns with texts are objects ā€“ nothing spectacular.

In the next step, I always like to get a “feeling” of all the data. Primarily, I want to get a quick impression of the data quality again. It could always be that there is “dirty data” in the dataset or that there are outliers that would screw up the analysis. With such a small amount of data we have, we can simply list all unique values that occur in the columns. I just list the top 10’s for both columns.

InĀ [5]:
commits['name'].value_counts()[0:10]
Out[5]:
Mic                   211
Antoine Rey           112
michaelisvy            87
Dirk Mahler            50
Keith Donald           35
Costin Leau            28
feststelltaste         26
Cyrille Le Clerc        5
Thibault Duchateau      5
Dapeng                  5
Name: name, dtype: int64

OK, at first glance, something seems awkward. Let’s have a look at the email addresses.

InĀ [6]:
commits['email'].value_counts()[0:10]
Out[6]:
misvy@vmware.com                 224
misvy@gopivotal,com               63
antoine.rey@gmail.com             59
antoine.rey@free.fr               53
dirk.mahler@buschmais.com         46
kdonald@vmware.com                35
cleau@vmware.com                  28
feststelltaste@googlemail.com     27
misvy@gopivotal.com               11
verydapeng@gmail.com               5
Name: email, dtype: int64

OK, the bad feeling is strengthening. We might have a problem with multiple authors having multiple email addresses. Let me show you the problem by better representing the problem.

Interlude – begin

In the interlude section, I take you to a short, mostly undocumented excursion with probably messy code (don’t do this at home!) to make a point. If you like, you can skip that section.

Goal: Create a diagram that shows the relationship between the authors and the emails addresses.

(Note to myself: It’s probably better to solve that directly in Neo4j the next time šŸ˜‰ )

I need a unique index for each name and I have to calculate the number of different email addresses per author.

InĀ [7]:
grouped_by_authors = commits[['name', 'email']]\
   .drop_duplicates().groupby('name').count()\
    .sort_values('email', ascending=False).reset_index().reset_index()
grouped_by_authors.head()
Out[7]:
indexnameemail
00michaelisvy3
11Ameya Pandilwar2
22Dirk Mahler2
33Antoine Rey2
44Rossen Stoyanchev1

Same procedure for the email addresses.

InĀ [8]:
grouped_by_email = commits[['name', 'email']]\
   .drop_duplicates().groupby('email').count()\
    .sort_values('name', ascending=False).reset_index().reset_index()
grouped_by_email.head()
Out[8]:
indexemailname
00feststelltaste@googlemail.com2
11misvy@vmware.com2
22Andrej11
33jdubois@ippon.fr1
44kdonald@vmware.com1

Then I merge the two DataFrames with a subset of the original data. I get each author and email index as well as the number of occurrences for author respectively emails. I only need the ones that are occurring multiple times, so I check for > 2.

InĀ [9]:
plot_data = commits.drop_duplicates()\
   .merge(grouped_by_authors, left_on='name', right_on="name", suffixes=["", "_from_authors"], how="outer")\
    .merge(grouped_by_email, left_on='email', right_on="email", suffixes=["", "_from_emails"], how="outer")
plot_data = plot_data[\
                        (plot_data['email_from_authors'] > 1) | \
                        (plot_data['name_from_emails'] > 1)]
plot_data
Out[9]:
emailnameindexemail_from_authorsindex_from_emailsname_from_emails
0feststelltaste@googlemail.comMarkus Harrer21102
1feststelltaste@googlemail.comfeststelltaste14102
3dirk.mahler@buschmais.comDirk Mahler22391
4dirk.mahler@asml.comDirk Mahler22381
7antoine.rey@gmail.comAntoine Rey32281
8antoine.rey@free.frAntoine Rey32271
13ameya@pandilwar.comAmeya Pandilwar12251
14ameya@ccs.neu.eduAmeya Pandilwar12231
22misvy@gopivotal,commichaelisvy0391
23misvy@gopivotal.commichaelisvy03101
24misvy@vmware.commichaelisvy0312
25misvy@vmware.comMic22112

I just add some nicely normalized indexes for plotting (note: there might be a method that’s easier)

InĀ [10]:
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(plot_data['index'])
plot_data['normalized_index_name'] = le.transform(plot_data['index']) * 10
le.fit(plot_data['index_from_emails'])
plot_data['normalized_index_email'] = le.transform(plot_data['index_from_emails']) * 10
plot_data.head()
Out[10]:
emailnameindexemail_from_authorsindex_from_emailsname_from_emailsnormalized_index_namenormalized_index_email
0feststelltaste@googlemail.comMarkus Harrer21102500
1feststelltaste@googlemail.comfeststelltaste14102400
3dirk.mahler@buschmais.comDirk Mahler223912090
4dirk.mahler@asml.comDirk Mahler223812080
7antoine.rey@gmail.comAntoine Rey322813070

Plot an assignment table with the relationships between authors and email addresses.

InĀ [11]:
fig1 = plt.figure(facecolor='white')
ax1 = plt.axes(frameon=False)
ax1.set_frame_on(False)
ax1.get_xaxis().tick_bottom()
ax1.axes.get_yaxis().set_visible(False)
ax1.axes.get_xaxis().set_visible(False)

# simply plot all the data (imperfection: duplicated will be displayed in bold font)
for data in plot_data.iterrows():
    row = data[1]
    plt.text(0, row['normalized_index_name'], row['name'], fontsize=15, horizontalalignment="right")
    plt.text(1, row['normalized_index_email'], row['email'], fontsize=15, horizontalalignment="left")
    plt.plot([0,1],[row['normalized_index_name'],row['normalized_index_email']],'grey', linewidth=1.0)

Alright! Here we are! We see that multiple authors use multiple email addresses. And I see a pattern that could be used to get better data. Do you, too?

Interlude – end

If you skipped the interlude section: I just visualized / demonstrated that there are different email addresses per author (and vise versa). Some authors choose to use another email address and some choose a different name for committing to the repositories (and a few did both things).

Data Wrangling

The situation above is a typical case of a little data messiness and ā€“ to demotivate you ā€“ absolutely normal. So we have to do some data correction before we start our analysis. Otherwise, we would ignore reality completely and deliver wrong results. This could damage our reputation as a data analyst and is something we have to avoid at all costs!

We want to fix the problem with the multiple authors having multiple email addresses (but are the same persons). We need a mapping between them. Should we do it manually? That would be kind of crazy. As mentioned above, there is a pattern in the data to fix that. We simply use the name of the email address as an identifier for a person.

Let’s give it a try by extracting the name part from the email address with a simple split.

InĀ [12]:
commits['nickname'] = commits['email'].apply(lambda x : x.split("@")[0])
commits.head()
Out[12]:
emailnamenickname
0feststelltaste@googlemail.comMarkus Harrerfeststelltaste
1feststelltaste@googlemail.comfeststelltastefeststelltaste
2feststelltaste@googlemail.comfeststelltastefeststelltaste
3feststelltaste@googlemail.comfeststelltastefeststelltaste
4feststelltaste@googlemail.comfeststelltastefeststelltaste

That looks pretty good. Now we want to get only the person’s real name instead of the nickname. We use a little heuristic to determine the “best fitting” real name and replace all the others. For this, we need group by nicknames and determine the real names.

InĀ [13]:
def determine_real_name(names):
    
    real_name = ""
    
    for name in names:
        # assumption: if there is a whitespace in the name, 
        # someone thought about it to be first name and surname
        if " " in name:
            return name
        # else take the longest name
        elif len(name) > len(real_name):
            real_name = name
            
    return real_name
        
commits_grouped = commits[['nickname', 'name']].groupby(['nickname']).agg(determine_real_name)
commits_grouped = commits_grouped.rename(columns={'name' : 'real_name'})
commits_grouped.head()
Out[13]:
real_name
nickname
Andrej1AndrejGajdos
ameyaAmeya Pandilwar
angel.aguileraAngel Aguilera
antoine.reyAntoine Rey
armagan.ersozkadinyazilimci

That looks great! Now we switch back to our previous DataFrame by joining in the new information.

InĀ [14]:
commits = commits.merge(commits_grouped, left_on='nickname', right_index=True)
# drop duplicated for better displaying
commits.drop_duplicates().head()
Out[14]:
emailnamenicknamereal_name
0feststelltaste@googlemail.comMarkus HarrerfeststelltasteMarkus Harrer
1feststelltaste@googlemail.comfeststelltastefeststelltasteMarkus Harrer
27feststelltaste@users.noreply.github.comMarkusfeststelltasteMarkus Harrer
29dirk.mahler@buschmais.comDirk Mahlerdirk.mahlerDirk Mahler
75dirk.mahler@globalfoundries.comdmahlerdirk.mahlerDirk Mahler

That should be enough data cleansing for today!

Analysis

Now that we have valid data, we can produce some new insights.

Top 10 committers

Easy tasks first: We simply produce a table with the Top 10 committers. We group by the real name and count every commit by using a subset (only the email column) of the DataFrame to only get on column returned. We rename the returned columns to commits for displaying reasons (would otherwise be email). Then we just list the top 10 entries after sorting appropriately.

InĀ [15]:
committers = commits.groupby('real_name')[['email']]\
  .count().rename(columns={'email' : 'commits'})\
    .sort_values('commits', ascending=False)
committers.head(10)
Out[15]:
commits
real_name
michaelisvy298
Antoine Rey112
Dirk Mahler51
Keith Donald35
Markus Harrer29
Costin Leau28
Tomas Repel5
Thibault Duchateau5
Cyrille Le Clerc5
Dapeng5
InĀ [16]:
committers.head(10)
Out[16]:
commits
real_name
michaelisvy298
Antoine Rey112
Dirk Mahler51
Keith Donald35
Markus Harrer29
Costin Leau28
Tomas Repel5
Thibault Duchateau5
Cyrille Le Clerc5
Dapeng5

Committer Distribution

Next, we create a pie chart to get a good impression of the committers.

InĀ [17]:
committers['commits'].plot(kind='pie')
Out[17]:
<matplotlib.axes._subplots.AxesSubplot at 0x24319d65d30>

Uhh…that looks ugly and kind of weird. Let’s first try to fix the mess on the right side that shows all authors with minor changes by summing up their number of commits. We will use a threshold value that makes sense with our data (e. g. the committers that contribute more than 3/4 to the code) to identify them. A nice start is the description of the current data set.

InĀ [18]:
committers_description = committers.describe()
committers_description
Out[18]:
commits
count37.000000
mean16.540541
std51.868742
min1.000000
25%1.000000
50%1.000000
75%5.000000
max298.000000

OK, we want the 3/4 main contributors…

InĀ [19]:
threshold = committers_description.loc['75%'].values[0]
threshold
Out[19]:
5.0

…that is > 75% of the commits of all contributors.

InĀ [20]:
minor_committers = committers[committers['commits'] <= threshold]
minor_committers.head()
Out[20]:
commits
real_name
Tomas Repel5
Thibault Duchateau5
Cyrille Le Clerc5
Dapeng5
Gordon Dickens5

These are the entries we want to combine to our new “Others” section. But we don’t want to loose the number of changes, so we store them for later usage.

InĀ [21]:
others_number_of_changes = minor_committers.sum()
others_number_of_changes
Out[21]:
commits    59
dtype: int64

Now we are deleting all authors that are in the author_minor_changes‘s DataFrame. To not check on the threshold value from above again, we reuse the already calculated DataFrame.

InĀ [22]:
main_committers = committers[~committers.isin(minor_committers)]
main_committers.tail()
Out[22]:
commits
real_name
Colin ButNaN
Craig DennisNaN
Mike EltsufinNaN
Faisal HameedNaN
thinkshNaN

This gives us for the contributors with just a few commits missing values for the changes column, because these values were in the author_minor_changes DataFrame. We drop all Nan values to get only the major contributors.

InĀ [23]:
main_committers = main_committers.dropna()
main_committers
Out[23]:
commits
real_name
michaelisvy298.0
Antoine Rey112.0
Dirk Mahler51.0
Keith Donald35.0
Markus Harrer29.0
Costin Leau28.0

We add the “Others” row by appending to the DataFrame

InĀ [24]:
main_committers.loc["Others"] = others_number_of_changes
main_committers
Out[24]:
commits
real_name
michaelisvy298.0
Antoine Rey112.0
Dirk Mahler51.0
Keith Donald35.0
Markus Harrer29.0
Costin Leau28.0
Others59.0

Almost there, you redraw with some styling and minor adjustments.

InĀ [25]:
# some configuration for displaying nice diagrams
plt.style.use('fivethirtyeight')
plt.figure(facecolor='white')

ax = main_committers['commits'].plot(
    kind='pie', figsize=(6,6), title="Main committers", 
    autopct='%.2f', fontsize=12)
# get rid of the distracting label for the y-axis
ax.set_ylabel("")
Out[25]:
<matplotlib.text.Text at 0x2431ae79c18>

Summary

I hope you saw that there are some minor difficulties in working with data. We got the big problem with the authors and email addresses that we solved by correcting the names. We also transformed an ugly pie chart into a management-grade one.

This analysis also gives you some handy lines of code for some common data analysis tasks.

print
Committer Distribution

2 thoughts on “Committer Distribution

  • March 15, 2017 at 11:52 PM
    Permalink

    Really cool blog post. In Neo4j you could have done.

    MATCH (author:Author)-[:COMMITED]-> (commit:Commit)
    RETURN author.name as name, collect(distinct author.email) as email, count(*) as commits

    you could then either marked one of the autors as :Person and connected the others to it or create a new “Person” node and connect all of them to it

    MATCH (author:Author)-[:COMMITED]-> (commit:Commit)
    WITH author, count(*) as commits order by commits desc // ordered
    WITH author.name as name, collect(distinct author) as emails
    WITH head(emails) as main, tail(emails) as rest
    SET main:Person
    FOREACH (a in rest | MERGE (a)-[:ALIAS_FOR]->(main) )

    Reply
    • March 16, 2017 at 5:37 AM
      Permalink

      Thanks for the tip! I like the idea of the “higher level concept” of a person. I have to admit that it felt a little bit wrong to do it in Pandas. I think it’s time to read a Neo4j book šŸ˜‰

      Reply

Leave a Reply

Your email address will not be published.