Assessing the value of Architectural Information Extracted from Patterns for Architecting

Background: We have developed an approach to identifying and capturing architecturally significant information from patterns (ASIP), which can be used to improve architecture design and evaluation. Goal: Our goal was to evaluate whether the use of the ASIP provides more effective support in understanding or designing software architecture composed of the software design patterns which are the source of the ASIP compared with the original design pattern documentation. Experimental design: Our subjects were 20 experienced software engineers who had returned to University for a post graduate course. All participants were taking a course in software architecture. The participants were randomly assigned to two groups of equal size. Both groups performed two tasks: understanding the use of J2EE design pattern in a given architecture based on the quality requirements the architecture was supported to satisfy, and designing software architecture to satisfy a given set of quality requirements using J2EE design patterns. For the first task, one group (treatment group) was given ASIP information the other (control group) was given the standard J2EE pattern documentation. For the second task, treatment group became the control group and vice versa and the type of support information was kept constant. The outcome variables were the number of correctly identified design patterns. The participants also completed a post-experiment questionnaire. Result: The average score for the first task for the treatment group was 23.90 and for the control group was 13.80. The difference between the groups was significant using Mann-Whiney test (p=0.0375). The average score for the second task for the treatment group was 26.85 and for the control group was 19.60. Mann-Whitney test revealed that the difference between the groups was again significant at (p=0.035). Post-study questionnaire revealed that 18 of the 20 participants believed that the ASIP was more helpful than pattern documentation for understanding and designing architectures. Conclusion: Our results support the hypotheses that ASIP information is more helpful in understanding or designing software architectures using software design patterns than pattern documentation itself.