A Status Report on the Evaluation of Variability Management Approaches

OBJECTIVE – The objective of this study is to review the status of evaluation of Variability Management (VM) approaches in Software Product Line Engineering (SPLE). METHOD – We have carried out a systematic review of studies of VM in SPLE reported in any publication venue and published before September 2007. RESULTS – 97 papers were selected according to our inclusion and exclusion criteria. The selected papers appeared in 56 publication venues and the dominance of well-known publication venues of SPLE is not very clear. Only a small portion of the identified approaches were evaluated using rigorous scientific methods. A detailed investigation of the studies employing empirical research methods revealed significant quality deficiencies in various quality assessment aspects. Even more disappointing finding is that the trends of scientific evaluation of VM approaches do not appear to be improving based on the analysis of the data extracted from the reviewed studied. CONCLUSIONS – The status of evaluation of VM approaches in SPLE is poor. Most of the approaches proposed are amenable to empirical evaluation, but the available evidence is sparse and the quality of the presented evidence is very low. The findings highlight the areas of improvement for rigorous evaluation of VM approaches.