A lesson learned from the black-box perspective
On behalf of the German Gymnastics Association, an exercise program for community-dwelling persons aged over 70 was developed [8]. The exercise program consisted of 12 exercise sessions. The concept of each session was based on four different modules, namely strength, mobility, flexibility, and cognition/safety. Conducting a formative evaluation, the program was carried out by three exercise groups. At first, the pre-test and post-test results of motor abilities were studied: one remarkable result was that flexibility of the test persons showed clear improvements in all groups. If there had been a black-box perspective as in many evaluations [3], the module conception and the exercise selection would have been regarded as confirmed. However, studying the exercise documentation as well as the instructor observation protocols opened new perspectives: The flexibility module only had been conducted three out of 12 times! Thus, flexibility improvements could hardly be ascribed to the program conception and the flexibility module.
Although the outcomes were expected and welcomed, it must be emphasized that they did not produce evidence for the implicit assumptions of the new program (certain exercise modules within each session lead to corresponding improvements). The reason was that the instructor did not carry out the lessons as proposed in the manual. In fact, due to time constraints, the instructor had to drop the last module and therefore tried to include some flexibility exercises into the other modules. Using a black-box evaluation, this would not have been discovered. This example clearly shows how important the explicit declaration of the assumptions of a program is. Furthermore, it underlines the relevance of controlling how program plans are implemented and executed.