The code review study has ended, further info for the participants is now available.
My main research interest and the topic of my doctoral research is improving code reviews through computer support. I'm also interested in ways to improve software development with a focus on small and medium software companies with agile development processes.
Code review has been known to be an effective quality assurance technique for decades. In the last years, industrial code review practices were observed to converge towards “change-based/modern code review”, but with a lot of variation in the details of the processes. Recent research also proposed hypotheses on factors that influence the choice of process. However, all current research in this area is based on small and largely non-random samples of cases. Therefore, we set out to assess the current state of the practice and to test some of these hypotheses with a survey among commercial software development teams. We received responses from 240 teams. They support many of the stated hypotheses, e.g., that change-based code review is the dominating style of code review in the industry, and that teams doing change-based code review have a lower risk that review use fades away. However, other hypotheses could not be confirmed, mainly that the balance of effects a team tries to reach with code reviews acts as a mediator in determining the details of the review process. Apart from these findings, we contribute the survey data set as a foundation for future research.
Change-based code review, e.g., in the form of pull requests, is the dominant style of code review in practice. An important option to improve review’s efficiency is cognitive support for the reviewer. Nevertheless, review tools present the change parts under review sorted in alphabetical order of file path, thus leaving the effort of understanding the construction, connections, and logic of the changes on the reviewer. This leads to the question: How should a code review tool order the parts of a code change to best support the reviewer? We answer this question with a middle-range theory, which we generated inductively in a mixed methods study, based on interviews, an online survey, and existing findings from related areas. Our results indicate that an optimal order is mainly an optimal grouping of the change parts by relatedness. We present our findings as a collection of principles and formalize them as a partial order relation among review orders.
Tool support for change-based code review is gaining widespread acceptance in the industry. This indicates that the current generation of tools is well-aligned to current code review practices. Nevertheless, we believe that further improvements in code review tooling can lead to increased review efficiency and effectiveness. In this paper, we combine results from a qualitative study and results from the literature to substantiate this claim. We derive promising improvement areas and provide an overview of existing research in these areas. A common attribute of these improvements is that they trade flexibility for reviewer support. As flexibility is one of the main characteristics of the current generation of code review tools in Hedberg's classification of review tool generations, we regard these coming tools as part of a new generation of code review tools.
Code review in the industry today is different to code review twenty years ago. The process has become more lightweight, reviews are performed frequently and change-based and the use of specialized tools is increasing. An accurate view of the current state of the industrial practice is an indispensable foundation for improving it. Most recent descriptions of review practices come from a limited population of large high-tech companies. Therefore, we used interviews with software engineering professionals from a broad sample of 19 companies to gain insight into their code review practices. We augmented our findings with data for 11 companies found through a semi-systematic literature review. There are many commonalities in the code review processes of these companies, but also a lot of variation in the details. A simple process taxonomy cannot describe these variations adequately. Therefore, we present a faceted classification scheme that is grounded in our observations.
Code review is known to be an efficient quality assurance technique. Many software companies today use it, usually with a process similar to the patch review process in open source software development. However, there is still a large fraction of companies performing almost no code reviews at all. And the companies that do code reviews have a lot of variation in the details of their processes. For researchers trying to improve the use of code reviews in industry, it is important to know the reasons for these process variations. We have performed a grounded theory study to clarify process variations and their rationales. The study is based on interviews with software development professionals from 19 companies. These interviews provided insights into the reasons and influencing factors behind the adoption or non-adoption of code reviews as a whole as well as for different process variations. We have condensed these findings into seven hypotheses and a classification of the influencing factors. Our results show the importance of cultural and social issues for review adoption. They trace many process variations to differences in development context and in desired review effects.
Code review in practice is often performed change-based, i.e. using the code changes belonging to a task to determine which code to review. In previous studies, it was found that two variations of this process are used in industry: Pre commit review (review-then-commit) and post commit review (commit-then-review). The choice for one of these variants has implications not only for practitioners deciding on a code review process to use, but also for the development of review tools and for experimentation with review processes. In some situations, a specific variant is clearly preferable due to the nature of the development process or team. In other situations, there are conflicting opinions, and both variants have proponents arguing for their method of choice. So we asked: Are there practically relevant performance differences between pre commit and post commit reviews, and how are these differences influenced by contextual factors? To assess this question, we designed a parametric discrete event simulation model of certain agile development processes. We validated this model with practitioner's feedback and in part also with empirical data from industry. Our analysis of the simulation results indicates that the best choice does depend on the context, but also that there are many situations with no practically relevant difference between both choices. We identified the main influencing factors and underlying effects and condensed our findings into heuristic rules.
The need to increase the efficiency of software development creates a demand for unobtrusive in-process means of software quality assurance that align well with agile processes. Many version control system clients provide facilities for running scripts or other executables before a change is committed to the repository. These ”pre-commit hooks” can be client-side or server-side, both having distinct advantages. Client-side pre-commit hooks are well suited to implement context-sensitive checklists and automatic quality checks, but their potential has not been fully realized in practice and corresponding research is missing. To narrow this gap, we did an industrial case study on this use of client-side pre-commit hooks. The case study used the tool ”TortoiseChecklist” and was focused on increasing continuous integration build stability. The results were positive, with a notable increase in stability and no significant disadvantages noticed.