5.
DISCUSSION
5.1 Contributions and
implications
Process Debt is a phenomenon that has not been studied as much as
Technical Debt. Through a 3-step investigation, we have collected
qualitative data from five companies and quantitative data from four of
them for validation purposes.
The main contributions are:
- a comprehensive framework to practically and theoretically reason
about PD, including:
- causes, consequences and types of PD
- occurrence patterns and evolution of PD over time
- a survey of the state of practice for PD management in five companies
- evidence of concrete instances covering each type of PD
- mitigation strategies to manage PD
- extensive validation of results by triangulating methods and sources
across contexts
We discuss these points in the following sections.
5.1.1 A framework to reason about Process
Debt
We propose an overall PD conceptual framework (Figure 2) where the key
elements and their relationships are visible. We also give detailed
taxonomies of types of PD, occurrence patterns, causes, effects and
mitigation strategies.
Our framework can support reasoning about PD for practitioners who would
like to systematically manage PD and need a comprehensive reference
point. First and foremost, TD research has shown how the first important
step to managing debt is to build awareness around it and to start a
discussion in the organizations.
From the theoretical point of view, our framework is not complete, as
the taxonomies can be extended with additional evidence collected from
new and specific case studies. However, our results show that, when
extending our initial framework with new, in-depth evidence from a
longitudinal case study with a different context from the initial four
companies, the framework was substantially validated with some little
additions. This gives confidence that the framework represents a robust
first step towards building a comprehensive theory of PD and is already
usable as a key theoretical reference point.
Our findings show that the PD lens can enrich the knowledge related to
Software Process Improvement with a new perspective on how to prioritize
SPI work. Much of the existing literature is dedicated to assessing SPI
or to improving specific processes: however, little is found about how
to prioritize if an SPI intervention is actually more important than
another one or even if improving a process is more important than
developing features and fixing bugs. The debt perspective and our
characterization using entities such as principal and interest borrowed
from the financial domain, can help to reason about the prioritization
of SPI to avoid or repay an existing PD that is particularly disruptive,
while avoiding SPI work and spending resources that would reduce PD with
limited negative effects.
5.1.2 Process Debt state of
practice
Our investigation shows that PD is an existing phenomenon that can cause
a huge amount of negative effects, among which mentioned the limitation
of applying modern practices to support vital business approaches such
as continuous delivery. PD is also recognized as generating Technical
Debt.
Although a quantification of the (negative) effects of PD is not
provided in this study, according to the interviewees the magnitude of
its impact is comparable to the TD one. Further studies and surveys
should be conducted on the PD phenomenon to assess its actual impact.
Our taxonomies can help create a solid instrument to investigate such a
phenomenon.
Throughout our report, we have mentioned several concrete instances
mentioned by the practitioners supporting our taxonomies. Besides
underpinning our framework, such instances constitute a concrete list of
examples of PD that can occur in organizations. Practitioners can also
use our taxonomy and even such concrete list of instances to identify
and differentiate across PD types in their context, to recognize their
causes and effects even before PD becomes disruptive and be able to
reason about prioritizing PDs’ avoidance and removal.
A key point revealed by observing PD in the longitudinal study, is that
removing PD often introduces new PD. This is different from TD, where
often repaying TD implies that the new solution is without better
(excluding a few corner cases). This means that evaluating if repaying
PD is worth it, a more complex assessment is needed, and more than one
“repayment” might be included in the equation to reach a substantial
improvement. More research is needed to understand such evaluation and
assessment. This is in line with Argyris and Schön [21] and the use
of double-loop learning, where additional evaluation is done after a
solution is applied.
The most valuable part of our reported PD state of practice is perhaps
related to the reported mitigation strategies used by the practitioners
to manage PD, which can also be applied in other contexts (if not
precisely, at least can give inspiration for new practices). In
addition, besides what is used today, we show what mitigation strategies
are in practitioners’ plans for better managing PD in the future. This
can give key targets for companies developing tools for SPI and to
researchers to investigate novel approaches along the directions
mentioned by our informants.
5.1.3 Validity, reliability and limitations
Our study relies on interviews, observations and quantitative data
collection with a limited set of organizations. This means that our
results cannot be considered general and suffer from external validity
threats [25]. For example, additional organizations with different
domains and contexts should be interviewed and surveyed to reach a
complete conceptual framework. Some of our results would hold in
different settings (e.g., the types of PD), but, for example, some of
the causes and consequences might not apply or be different in such
contexts. We, therefore, acknowledge that further research is required.
However, we argue that our exploratory effort is already valuable, in
that we included different companies with different contexts and
different stakeholders, ranging from process designers to software
developers, to team leaders and practitioners in other organizations,
such as hardware designers. To mitigate the threats to external
validity, we have also applied two additional validation steps by
running a workshop and collecting additional qualitative and
quantitative data, and comparing our results to the in-depth findings
from a longitudinal case study in a new and different company E. We
argue that it would not be possible to cover the phenomenon in its
entirety with just one study, but we call for the software engineering
community to contribute to studying PD with additional investigations.
As for reliability, although it’s often the case that qualitative
studies rely heavily on the researchers’ interpretation, we have used
several mitigation strategies to limit the bias introduced by the
authors. First, we made sure that in the majority, at least two
researchers were present at the interviews, and two researchers analyzed
the interviews in parallel and then discussed and agreed on the
resulting coding. At least two researchers have discussed the findings
and definitions across studies to ensure that the data from the
longitudinal study would be correctly interpreted and mapped to the
framework obtained from the first phase.
One limitation is the lack of input for the workshop run in phase 3 from
the practitioners in the longitudinal case study related to phase 2, but
only related to the first four companies investigated in the first
phase. However, similar insights were collected and analyzed in the
longitudinal case study with a different method, the only difference is
the data collection format.