A Survey of Source Code Summarization
Devjeet Roy
QE Oral Presentation
Introduction
Software Developers spend a significant portion of their time comprehending source code
Proper documentation and inline comments in source code is vital to the comprension process.
Maintaining documentation is resource intensitve
Automatic Source Code Summarization
The automatic generation of source code documentation without human interventation
Approaches exist that apply to a wide range of source code artifacts:
* methods, classes, source code segments
* version control commits
* release notes
This survey
+ Abstractive Summarization only
+ Source code intrinsic documentation only (excludes generation of release notes, commit messages and other extrinsic documentation)
- Excludes crowd-sourcing based approaches (which are sometimes included in literature reviews)
Methodology
Template Based Approaches
Relies on pre-defined templates for summary generation.
Templating systems range from simple, text templates to complex Natural Language Generation (NLG) systems.
Typically use information retrieval (IR) and/or program analysis to fill templates
Deep Learning
Machine learning technique
Generates full sentences directly without the use of templates
Typically trained on large amounts of data for state-of-the-art performance
Templates vs Deep Learning
Template based approaches require significant amounts of human effort to build/maintain templates
Deep learning based approaches are expensive computationally and require large amounts of data
However, deep learning generalizes better, and can often be more natural.
Templates vs Deep Learning
The first deep learning based summarization approaches appeared in 2016.
Ever since 2018, most summarization approaches are based on deep learning
Evaluation Methods
Manual Verification
Researchers manually verify correctness/quality of their summarization approach.
Used more in earlier works in summarization, not commonly used any more.
Large potential for bias
Difficult to replicate results
Human Study
The summarization approach is evaluated by external human participants
Gold standard for evaluation
Resource intensive
Automatic Evaluation
The summarization approach is evaluated using automatic metrics that serve as a proxy for human judgement.
Commonly used metrics: BLEU, ROUGE, METEOR.
Can be easily applied to large datasets.
Unclear as to how effective they are at approximating human judgement
Change in Evaluation Methods
Gradual shift from manual verification/human studies to automatic evaluation metrics
- largely due to the adoption of deep learning techniques
Only 4/14 deep learning based papers from the last 4 years include human study to augment automatic evaluation
Datasets in Summarization
Datasets
Easily available from publicly hosted repositories on services like GitHub, BitBucket etc.
The landscape is fairly fragmented; there are no standard benchmark datasets, which makes comparing performance across approaches difficult
Changed significantly since the early days of summarization:
from small(1-10 projects) to extremely large (> 1000 projects)
Limitations of Existing Literature
Evaluation Problem
The Evaluation Problem
Currently, the majority of summarization papers use automatic metrics as their evaluation method
The metrics used are adapted from Neural Machine Translation (NMT)
However, the validity of these metrics in the context of code summarization remains unclear.
There are significant differences between datasets used in machine natural language processing, and datasets in code summarization (Gros et al)
On comprehension tasks, developers perform significantly better when using human written summaries vs automatically generated summaries (Stapleton et al)
The Evaluation Problem
The Evaluation Problem
Do metrics effectively approximate human judgmement in summarization?
Do improvements in metric scores translate to tangible improvements in real world performance?
Proposed Solution
Investigate the validity of the use of automatic evaluation metrics in code summarization
There is a rich body of literature in Neural Machine Translation (NMT) on how to conduct these studies and analyze the data collected
- NMT has a yearly conference known as WMT dedicated to this
Proposed Solution
Establish the efficacy of automatic evaluation metrics in their ability to serve as approximations of human judgement
- conduct human studies comparing summarization approaches, and use human judgement as gold standard to evaluate metrics
- using the human studies, establish guidelines for interpretation of these automatic evaluation metrics
- identify shortcomings of these metrics and draft measures to mitigate them
Dataset Problem
Heterogeneity
There are no standard datasets that are used as benchmarks to evaluate summarization techniques.
Makes it difficult to compare performance of different approaches if they don't use the same datasets for evaluation
Compounded by the fact that automatic evaluation metrics are highly sensitive to data preprocessing steps, such as tokenization (same dataset preprocessed differently can lead to high variance in results)
The Dataset Problem: Heterogeneity
The lack of standard benchmark datasets impedes research
- Deep learning based summarization approaches routinely take > 50 hours to train on specialized hardware
- this computational cost is often prohibitive when it comes to replication, reproduction and comparison
The Dataset Problem: Quality
It is unclear whether summarization datasets are representative of real world scenarios.
Papers generally report high level metrics like # of methods, projects, avg. loc etc.
- but don't take into account what proportion of the dataset is actually useful in the context of summarization
The Dataset Problem: Quality
There is no literature on what should and shouldn't be summarized. eg. short methods might not need be summarized.
Without this knowledge, it is unclear whether improvements in performance on these datasets would translate to real world performance improvements
Outside of basic quality control measures, there is no explicit filtering of poor quality projects that are not appropriate for summarization
Proposed Solution
The research community must build high quality, easy to use benchmark datasets, such as the ones used in natural language processing (GLUE, bABI etc)
The first step is to establish desiderata for such a dataset using human studies, focusing on the needs of software developers
- what artifacts do they want automatically summarized and what documentation would they rather themselves maintain?
Conclusion
Source code summarization has seen major changes and significant progress over the years
However, need to step back address some fundamental questions regarding evaluation/datasets.