Theory-driven evaluation of exercise programs: often recommended but still only a few hits in literature databases

Among the papers submitted to and published by the European Review of Aging and Physical Activity, there are usually some dealing with exercise programs. In this editorial, some thoughts on the evaluation of such programs are shared. The editorial closes with some news regarding the journal.

Research-based activities in order to develop, to design or to change some real-world object or aspect (e.g., by an exercise program), are considered technological science, as expanded on in a recent editorial of this journal [1]. The outcome of this kind of research should and usually is in fact assessed and rated (evaluation). Researchers, funders, representatives of the target group, and other individuals being affected by such a program (stakeholders) increasingly call for evaluation. Often, the term evaluation is used as a synonym for assessment of effectivity. Compared with that, theory-driven evaluation includes also planning and implementation of a program. Theory-driven evaluation is a framework (in fact, there are several frameworks) with rising acceptance during the last 20 years—and seems to be helpful for health-oriented exercise programs.

In the following sections, the reader will find

• (a) an example showing, that it is important not only to assess the outcomes of a program, but also to carefully examine all constituents of the program, too,

• (b) a sketch on theory-driven evaluation as a tool to analyze complex programs in this way, especially if they deal with exercise and physical activity,

• (c) first results of an ongoing systematic literature research showing, however, that this kind of evaluation framework scarcely seems to be applied for health-oriented exercise programs yet.

A lesson learned from the black-box perspective

On behalf of the German Gymnastics Association, an exercise program for community-dwelling persons aged over 70 was developed [8]. The exercise program consisted of 12 exercise sessions. The concept of each session was based on four different modules, namely strength, mobility, flexibility, and cognition/safety. Conducting a formative evaluation, the program was carried out by three exercise groups. At first, the pre-test and post-test results of motor abilities were studied: one remarkable result was that flexibility of the test persons showed clear improvements in all groups. If there had been a black-box perspective as in many evaluations [3], the module conception and the exercise selection would have been regarded as confirmed. However, studying the exercise documentation as well as the instructor observation protocols opened new perspectives: The flexibility module only had been conducted three out of 12 times! Thus, flexibility improvements could hardly be ascribed to the program conception and the flexibility module.

Although the outcomes were expected and welcomed, it must be emphasized that they did not produce evidence for the implicit assumptions of the new program (certain exercise modules within each session lead to corresponding improvements). The reason was that the instructor did not carry out the lessons as proposed in the manual. In fact, due to time constraints, the instructor had to drop the last module and therefore tried to include some flexibility exercises into the other modules. Using a black-box evaluation, this would not have been discovered. This example clearly shows how important the explicit declaration of the assumptions of a program is. Furthermore, it underlines the relevance of controlling how program plans are implemented and executed.

Program theory: meeting the complexity of exercise programs

A program, simply defined as an entity of planned activities in order to achieve aims, is based on certain assumptions. Generally, it is assumed that a program is implemented as it was planned, i.e., that the activities are executed. Moreover, it is assumed that the activities of a program are able to establish the determinants of a certain law of nature or a recognized theory, which itself will yield the desired outcomes. In our field, exercises should be done as they have been planned (implementation). In addition, the exercises should establish a stimulus exceeding a minimum threshold in order to cause biological adaptation. Every program implicates many assumptions for different stakeholders on the individual and institutional level.

The explication of all these assumptions is called the program theory. The main feature of theory-driven evaluation is to consider program theory during the evaluation of a program [2, 3].

There are several pleas for applying program theory in different domains. A corresponding development in evaluation research can also be detected [1, 4]. It can be said that this trend applies for health-oriented programs in general, and for complex programs in particular. Exercise programs encompass a special complexity which goes well beyond, e.g., using medical drugs. However, in a certain manner, exercise programs also supersede other behavior change programs within health promotion. This can be ascribed to the special nature of an exercise:

• It includes more than just compliance, as for example in taking a medical drug or undergoing a certain treatment. Positive effects of exercises only emerge from regular, even straining, efforts of the individual.

• It comprises even more than other health-related behaviors because the behavior level including sport participation, regularity, and learning of new behaviors is a necessary but not a sufficient condition for exercise success.

• In order to assure positive effects and to prevent from harm, each exercise requires prescribed movements which should be controlled for correct performance at all times. Furthermore, extent and intensity of each exercise have to be changed over time in a well-balanced manner in order to meet training principles.

The expressions “it includes more” or “it comprises even more” do not state that exercise participation would generally be more difficult for the individual compared with other medical or health behavior. However, these expressions should underline that as a subject of research and evaluation, exercise programs are often more complex to describe and to investigate. In conclusion, this supports the arguments for applying program theory in evaluation of programs containing exercise and physical activity.

Searching for theory-driven evaluation of exercise programs

Reviewing theory-driven evaluation practice over the last 20 years, Coryn et al. [4] recently described the development of this framework and discussed certain methodological aspects of 45 evaluations found by a multi-step sampling and reviewing process. The titles of the publications reviewed imply that the main subjects of health promotion are covered, but none of the reviewed programs dealt distinctly with physical activity or exercise.

In order to verify the hypothesis that the advantages of theory-driven evaluation are scarcely utilized in the area of physical activity and exercise programs, a systematic literature research was conducted.

Relevant databases were Pubmed (health sciences) and Sport-Discus (sport science). For the comparison of hit numbers, Scopus and Google Scholar were also queried. The following key words were taken from the overview section of the paper written by Coryn et al. [4]: program theory, realist evaluation, theory-driven evaluation, theory of change. Table 1 shows the number of hits.

Table 1 Hits in different online databases (search performed on March 28, 2011)

A first attempt was made to identify different subject fields of the Pubmed hits by carefully reading the abstracts. Out of the 78 articles, 14 were excluded for formal reasons (no abstract available, four; colon between the terms program and theory, one) and for multiple use of the term program theory out of the present scope of evaluation research (biology of aging, three; motor programs, five; disease screening, one). Table 2 shows the identified subject fields.

Table 2 Subject fields of theory-driven evaluation

Only three papers were found that dealt with physical activity: an analysis of general assumptions about using pedometers for activity enhancement [9], a paper on the development of a daily activity program for individuals with type 2 diabetes [10], and a qualitative study of adaptations made on the institutional level during a transfer of successfully proved programs into real-world settings [6]. By now, even this small number of studies shows the broad range of opportunities to utilize program theory and theory-driven evaluation in physical activity and exercise programs. More applications are expected to be found during the ongoing literature analysis. In addition, the program theory framework has recently been used to compile a literature review in preparation of a new program [5].

News from EURAPA

Regarding the European Review of Aging and Physical Activity, the following news can be reported:

This is the first special issue of EURAPA, i.e., an issue with papers focusing on a special subject and guest edited by renowned experts in the field:

A life course approach to physical activity, health, and aging

Harri Suominen and Eino Heikkinen from the University of Jyväskylä, Finland, worked on this important subject. They won several colleagues to author papers for this issue. On behalf of the editorial board, I would like to thank the guest editors as well as all authors for their contributions.

The editors are open for proposals on other remarkable subjects as well as for new candidates for guest editors. Editor-in-chief Heinz Mechling already announced the changes starting with 2011. A call was published for application of future guest editors, for special issues and new editorial board members [7]. Everyone interested should feel free to send a proposal or to apply for a task or position.

As hitherto, in the first issue of the New Year, EURAPA's reviewers in 2010 are acknowledged:

Ian D. Cameron, Australia

Kay Cox, Australia

Jerri Edwards, USA

Sebastião Gobbi, Brazil

Eino Heikkinen, Finland

Timo Hinrichs, Germany

Darko Jekauc, Germany

Urho M. Kujala, Finland

Andrea Lamont-Mills, Australia

Markus Lamprecht, Switzerland

Babette Lobinger, Germany

Lee-Fay Low, Australia

Yael Netz, Israel

Jean Petermans, Belgium

Peter Reaburn, Australia

Annelie Reicherz, Germany

Jörg Schorer, Germany

Harri Suominen, Finland

Risto Telama, Finland

Claudia Voelcker-Rehage, Germany

Wiebren Zijlstra, The Netherlands

They all helped to maintain the high quality of the European Review of Aging and Physical Activity. Thank you very much for your efforts.

The present issue is the first online-only issue. Consequently, from now on EURAPA will no longer be available in a printed version. We have to admit that the editorial board was not happy when we were told about this change. However, we had to understand that the economical situation has recently been difficult. Furthermore, we also had to accept that today the common way of working with academic literature is the digital way. From the submission to the publication process, from literature research to reading and excerpting academic literature—everything is done in front of the computer. If a reader wants to read outside of the office or to put manual comments between the lines, he/she easily would print the paper instead of walking to the library. For promotional reasons, e.g., on occasion of EGREPA conferences, it will still be possible to print and to distribute single issues of EURAPA.