Statistical Models to Estimate

Software Evolution Data From Repositories

By David A. Nader

Agenda

  • Background
    • terminology
    • goal
  • Exploratory Analysis
  • Empirical Analysis
    • Software Metrics
    • Time Metrics
  • Simulation
    • ssq
    • multi-server
      • Original Traces
      • Fitted
      • non-stationary

Background

Software Repositories

 

  • Issue: activity tracker (bug-req-task)
  • Stakeholder: a person working for the project or issue
  • Commit: tracking of changes of source code

Open

Close

Assigned

  • Time Metrics
    • issue-open: the creation of an issue
    • issue-close: the timestamp of the finalization of an issue
    • issue-wait: the difference between issue-open and issue-close
    • issue-delay: the time of the issue before being assigned
    • issue-service: the time of the issue after being assigned

Open

Close

Assigned

  • Time Metrics
    • issue-open: the creation of an issue
    • issue-close: the timestamp of the finalization of an issue
    • issue-wait: the difference between issue-open and issue-close
    • issue-delay: the time of the issue before being assigned
    • issue-service: the time of the issue after being assigned
  • Software Management Metrics
    • stakeholders: people involved in the issue realization
    • effort: the measure of days-persons per issue-wait

Open

Close

Assigned

The goal of this project is to simulate the software evolution process from repositories meta-data

There are two approaches ​to this problem

  • Simulation for Optimization 🡒  Software Management
  • Simulation for Automation 🡒  Software Maintenance

Simulation for Optimization 🡒  Software Management

Three software engineering factors (Honsel, et al. 2014):

  • Project Growth Trends
  • Bug Appearance
  • Development Strategies

Simulation for Optimization 🡒  Software Management

Three software engineering factors (Honsel, et al. 2014):

  • Project Growth Trends
  • Bug Appearance
  • Developer Activity

Proposed Metrics:

  • Project Growth Trends: Number of Commits
  • Bug Appearance: Number of Issues (Jobs)
  • Developer Activity: Effort 

The goal of this project is to simulate the software evolution process from repositories meta-data to minimize the effort

Datasets

GHTorrent monitors the Github public event time line. For each event, it retrieves its contents and their dependencies, exhaustively. It then stores the raw JSON responses to a MongoDB database, while also extracting their structure in a MySQL database.

MSR2014 Challenge Dataset

This dataset includes data from the top-10 starred software projects for the top programming languages deployed in gitHub.

 

In total, there are 90 different projects with their respective derivations (forks).

 

I utilized the datasets with the relational DB schema

Exploratory Analysis

  • Analogies
    • job: issue
    • server: teams or software groups
  • Time Metrics
    • a_i: the creation of an issue
    • c_i: the timestamp of the finalization of an issue
    • w_i: the difference between issue-open and issue-close
    • d_i: the time of the issue before being assigned
    • s_i: the time of the issue after being assigned
  • Software Management Metrics
    • stakeholders: people involved in the issue realization
    • effort: the measure of days-persons per issue-wait
    • size: commits per trace
  • Analogies
    • job: issue
    • server: teams or software groups

Software Evolution Trace Example - Timestamps

Computing Arrivals and Completions times

Exploratory Analysis of the Features

Cumulative Software Metrics

Stakeholders and Effort across the time

PDFs Arrivals and Services

Fitted Arrivals and Services

histograms main metrics

CDFs main metrics

effort-w_i [r = 0.64]

effort-d_i: [r = 0.51]

Auto-correlation in metrics of time

Empirical Analysis

Software and Time Metrics

First Order Statistics from Datasets

   average service time .... =  31.97
   average interarrival time =   0.21
   arrival rate ............ =   4.68
   average delay ........... =  32.35
   average wait ............ =  64.32

# of jobs in the node across the time

approx_servers:	         1176.58
approx_servers_max:	 38.29

number of servers?

approx_servers = max stakeholders across time / avg-stk

approx_servers_max = max stakeholders across time / max-stk

# of jobs in the Q across the time

Time-Averaged Statistics

 avg # in node 	= 305.36
 avg # in queue = 155.9
 utilization 	= 149.46

Time-Averaged Statistics

 avg # in node 	= 305.36
 avg # in queue = 155.9
 utilization 	= 149.46

Consistency Metrics

avg wait 	= 64.32 64.32
avg # in node 	= 300.88 305.36
avg # in queue 	= 151.32 155.9
utilization 	= 149.55 149.46

Time-Averaged Statistics

 avg # in node 	= 305.36
 avg # in queue = 155.9
 utilization 	= 149.46

Consistency Metrics (Little's Law)

avg wait 	= 64.32 64.32
avg # in node 	= 300.88 305.36
avg # in queue 	= 151.32 155.9
utilization 	= 149.55 149.46

Approx Utilization per server

avg utilization[server] 	= 0.13
max utilization[server] 	= 3.91

Simulation

ssq

Simulating with ssq providing arrivals and services

for 7326 jobs
   average interarrival time =   0.21
   average service time .... =  31.97
   average delay ........... = 22838.20
   average wait ............ = 22870.17

Simulating with ssq providing arrivals and services

for 7326 jobs
   average interarrival time =   0.21
   average service time .... =  31.97
   average delay ........... = 22838.20
   average wait ............ = 22870.17

   average service time .... =  31.97
   average interarrival time =   0.21
   arrival rate ............ =   4.68
   average delay ........... =  32.35
   average wait ............ =  64.32

Simulating with ssq providing arrivals and services

for 7326 jobs
   average interarrival time =   0.21
   average service time .... =  31.97
   average delay ........... = 22838.20
   average wait ............ = 22870.17

Approximation Error

   avg wait error ............ 	= 354.59
   avg delay error ............ 	= 705.04
   avg interarrival error ............ 	=   0.00
   avg service error ............ 	=   0.00

   average service time .... =  31.97
   average interarrival time =   0.21
   arrival rate ............ =   4.68
   average delay ........... =  32.35
   average wait ............ =  64.32

multi-server

Simulating with msq providing arrivals and exponential service times with 150 servers

for 7325 jobs and 150 servers the service node statistics (for 35 runs) are:


  avg interarrivals .. =   0.21
  avg wait ........... = 283.65
  avg # in node ...... = 864.21
  avg delay .......... = 251.73
  avg # in queue ..... = 766.94

 

  sim utilization	= 97.26
  sim service	= 32.54

Confidence Values [c=0.95]

  avg interarrivals .. = [0.21 0.21 0.21]
  avg wait ........... = [283.97 280.97 286.97]
  avg # in node ...... = [864.23 855.9  872.55]
  avg delay .......... = [251.98 249.09 254.86]
  avg # in queue ..... = [766.86 758.82 774.9 ]
  sim utilization	= [97.37 96.94 97.79]
  sim service	= [32.64 32.5  32.78]

Avg. Approximation Error for msq

metric with exponential services ...
avg wait 3.42
avg delay 6.79
avg interarrival 0.00
avg service 0.02
utilization 0.35
avg # in node 1.87
avg # in q 4.07

Avg. Approximation Error for msq providing fitted time services 

metric with exponential services with fitted services
avg wait 3.42 0.87
avg delay 6.79 1.00
avg interarrival 0.00 0.00
avg service 0.02 0.73
utilization 0.35 0.77
avg # in node 1.87 0.89
avg # in q 4.07 1.00

Avg. Approximation Error for msq providing fitted services and non-stationary arrivals (thinning method) 

metric with exponential services with fitted services with fitted services and non stationary arrivals (λ_max)
avg wait 3.42 0.87 0.87
avg delay 6.79 1.00 1.00
avg interarrival 0.00 0.00 3.70
avg service 0.02 0.73 0.74
utilization 0.35 0.77 0.95
avg # in node 1.87 0.89 0.97
avg # in q 4.07 1.00 1.00

Avg. Approximation Error for msq providing non-stationary arrivals and exponential services

metric with exponential services with fitted services with fitted services and non stationary arrivals (λ_max) with exponential services and non stationary arrivals (λ_max)
avg wait 3.42 0.87 0.87 0.50
avg delay 6.79 1.00 1.00 1.00
avg interarrival 0.00 0.00 3.70 3.70
avg service 0.02 0.73 0.74 0.02
utilization 0.35 0.77 0.95 0.80
avg # in node 1.87 0.89 0.97 0.90
avg # in q 4.07 1.00 1.00 1.00

Simulating with msq providing arrivals and exponential service times with approx. servers

for 7325 jobs and 1176 servers the service node statistics (for 35 runs) are:


  avg interarrivals .. =   1.00
  avg wait ........... =  31.85
  avg # in node ...... =  29.19
  avg delay .......... =   0.00
  avg # in queue ..... =   0.00

 

  sim utilization	= 29.19
  sim service	= 32.2

Confidence Values [c=0.95]

  avg interarrivals .. = [1. 1. 1.]
  avg wait ........... = [31.85 31.58 32.12]
  avg # in node ...... = [29.19 28.87 29.52]
  avg delay .......... = [ 0. -0.  0.]
  avg # in queue ..... = [ 0. -0.  0.]
  sim utilization	= [29.19 28.87 29.52]
  sim service	= [32.2 31.9 32.5]

Avg. Approximation Error for msq providing non-stationary arrivals and exponential services

metric with exponential services with fitted services with fitted services and non stationary arrivals (λ_max) with exponential services and non stationary arrivals (λ_max) with approx servers
avg wait 3.42 0.87 0.87 0.50 0.50
avg delay 6.79 1.00 1.00 1.00 1.00
avg interarrival 0.00 0.00 3.70 3.70 3.69
avg service 0.02 0.73 0.74 0.02 0.01
utilization 0.35 0.77 0.95 0.80 0.80
avg # in node 1.87 0.89 0.97 0.90 0.90
avg # in q 4.07 1.00 1.00 1.00 1.00

Summary

Simulation for Optimization 🡒  Software Management

Simulation for Optimization 🡒  Software Management

Accumulated Effort Across the Time

Simulation for Optimization 🡒  Software Management

Accumulated Effort Across the Time

effort-w_i [r = 0.64]

effort-d_i: [r = 0.51]

Effort vs {w_i, d_i}

Simulation for Optimization 🡒  Software Management

Accumulated Effort Across the Time

effort-w_i [r = 0.64]

effort-d_i: [r = 0.51]

metric with exponential services with fitted services with fitted services and non stationary arrivals (λ_max) with exponential services and non stationary arrivals (λ_max) with approx servers
avg wait 3.42 0.87 0.87 0.50 0.50
avg delay 6.79 1.00 1.00 1.00 1.00
avg interarrival 0.00 0.00 3.70 3.70 3.69
avg service 0.02 0.73 0.74 0.02 0.01
utilization 0.35 0.77 0.95 0.80 0.80
avg # in node 1.87 0.89 0.97 0.90 0.90
avg # in q 4.07 1.00 1.00 1.00 1.00

Avg. Approx.  Error

Effort vs {w_i, d_i}

State-of-the-art

Previous research can be classified into two categories

  • Simulations about Software Engineering
  • Data-mining techniques on the MSE2014 dataset

Simulations about Software Engineering (2014)

Simulations about Software Engineering (2015)

Simulations about Software Engineering (2018)

Data-mining techniques on the MSE2014 dataset

Data-mining techniques on the MSE2014 dataset

Data-mining techniques on the MSE2014 dataset

Data-mining techniques on the MSE2014 dataset

create or replace view v_processed_issues as select FirstSet.PR, FirstSet.ID, 
		FirstSet.open_issue, 
        FirstSet.wait_issue, 
        SecondSet.delay_issue, 
		(FirstSet.wait_issue-SecondSet.delay_issue) as service_issue, 
        ThirdSet.stakeholders,
		ceil(FirstSet.wait_issue/ThirdSet.stakeholders) as effort,
        ceil((FirstSet.wait_issue-SecondSet.delay_issue)
            /ThirdSet.stakeholders) as service_effort
from   
	(select distinct ISU.id as ID, ISU.repo_id as PR, 
	ISU.created_at as open_issue, ISUE.action as AC,
	TIMESTAMPDIFF(DAY, ISU.created_at, ISUE.created_at) as wait_issue
	from msr14.issues as ISU join msr14.issue_events as ISUE on ISU.id = ISUE.issue_id
    where ISUE.action='closed') as FirstSet
inner join
	(select distinct ISU.id as ID, ISU.repo_id as PR, 
	ISU.created_at as open_issue, ISUE.action as AC,
	TIMESTAMPDIFF(DAY, ISU.created_at, ISUE.created_at) as delay_issue
	from msr14.issues as ISU join msr14.issue_events as ISUE on ISU.id = ISUE.issue_id
    where ISUE.action='assigned') as SecondSet
on FirstSet.PR = SecondSet.PR and FirstSet.ID = SecondSet.ID   
inner join v_issues as ThirdSet
on FirstSet.ID = ThirdSet.IS_ID
having service_issue >= 0 
and FirstSet.wait_issue >= 0 
and SecondSet.delay_issue >= 0
order by FirstSet.open_issue;

An sql view to create the "Software Evolution Traces" 

There are 90 different SW projects

One project has multiple issues​

A pull-request has multiple issues