Theory-driven evaluation of exercise programs: often recommended but still only a few hits in literature databases
European Review of Aging and Physical Activity volume 8, pages 1–4 (2011)
Among the papers submitted to and published by the European Review of Aging and Physical Activity, there are usually some dealing with exercise programs. In this editorial, some thoughts on the evaluation of such programs are shared. The editorial closes with some news regarding the journal.
Research-based activities in order to develop, to design or to change some real-world object or aspect (e.g., by an exercise program), are considered technological science, as expanded on in a recent editorial of this journal . The outcome of this kind of research should and usually is in fact assessed and rated (evaluation). Researchers, funders, representatives of the target group, and other individuals being affected by such a program (stakeholders) increasingly call for evaluation. Often, the term evaluation is used as a synonym for assessment of effectivity. Compared with that, theory-driven evaluation includes also planning and implementation of a program. Theory-driven evaluation is a framework (in fact, there are several frameworks) with rising acceptance during the last 20 years—and seems to be helpful for health-oriented exercise programs.
In the following sections, the reader will find
an example showing, that it is important not only to assess the outcomes of a program, but also to carefully examine all constituents of the program, too,
a sketch on theory-driven evaluation as a tool to analyze complex programs in this way, especially if they deal with exercise and physical activity,
first results of an ongoing systematic literature research showing, however, that this kind of evaluation framework scarcely seems to be applied for health-oriented exercise programs yet.
A lesson learned from the black-box perspective
On behalf of the German Gymnastics Association, an exercise program for community-dwelling persons aged over 70 was developed . The exercise program consisted of 12 exercise sessions. The concept of each session was based on four different modules, namely strength, mobility, flexibility, and cognition/safety. Conducting a formative evaluation, the program was carried out by three exercise groups. At first, the pre-test and post-test results of motor abilities were studied: one remarkable result was that flexibility of the test persons showed clear improvements in all groups. If there had been a black-box perspective as in many evaluations , the module conception and the exercise selection would have been regarded as confirmed. However, studying the exercise documentation as well as the instructor observation protocols opened new perspectives: The flexibility module only had been conducted three out of 12 times! Thus, flexibility improvements could hardly be ascribed to the program conception and the flexibility module.
Although the outcomes were expected and welcomed, it must be emphasized that they did not produce evidence for the implicit assumptions of the new program (certain exercise modules within each session lead to corresponding improvements). The reason was that the instructor did not carry out the lessons as proposed in the manual. In fact, due to time constraints, the instructor had to drop the last module and therefore tried to include some flexibility exercises into the other modules. Using a black-box evaluation, this would not have been discovered. This example clearly shows how important the explicit declaration of the assumptions of a program is. Furthermore, it underlines the relevance of controlling how program plans are implemented and executed.
Program theory: meeting the complexity of exercise programs
A program, simply defined as an entity of planned activities in order to achieve aims, is based on certain assumptions. Generally, it is assumed that a program is implemented as it was planned, i.e., that the activities are executed. Moreover, it is assumed that the activities of a program are able to establish the determinants of a certain law of nature or a recognized theory, which itself will yield the desired outcomes. In our field, exercises should be done as they have been planned (implementation). In addition, the exercises should establish a stimulus exceeding a minimum threshold in order to cause biological adaptation. Every program implicates many assumptions for different stakeholders on the individual and institutional level.
There are several pleas for applying program theory in different domains. A corresponding development in evaluation research can also be detected [1, 4]. It can be said that this trend applies for health-oriented programs in general, and for complex programs in particular. Exercise programs encompass a special complexity which goes well beyond, e.g., using medical drugs. However, in a certain manner, exercise programs also supersede other behavior change programs within health promotion. This can be ascribed to the special nature of an exercise:
It includes more than just compliance, as for example in taking a medical drug or undergoing a certain treatment. Positive effects of exercises only emerge from regular, even straining, efforts of the individual.
It comprises even more than other health-related behaviors because the behavior level including sport participation, regularity, and learning of new behaviors is a necessary but not a sufficient condition for exercise success.
In order to assure positive effects and to prevent from harm, each exercise requires prescribed movements which should be controlled for correct performance at all times. Furthermore, extent and intensity of each exercise have to be changed over time in a well-balanced manner in order to meet training principles.
The expressions “it includes more” or “it comprises even more” do not state that exercise participation would generally be more difficult for the individual compared with other medical or health behavior. However, these expressions should underline that as a subject of research and evaluation, exercise programs are often more complex to describe and to investigate. In conclusion, this supports the arguments for applying program theory in evaluation of programs containing exercise and physical activity.
Searching for theory-driven evaluation of exercise programs
Reviewing theory-driven evaluation practice over the last 20 years, Coryn et al.  recently described the development of this framework and discussed certain methodological aspects of 45 evaluations found by a multi-step sampling and reviewing process. The titles of the publications reviewed imply that the main subjects of health promotion are covered, but none of the reviewed programs dealt distinctly with physical activity or exercise.
In order to verify the hypothesis that the advantages of theory-driven evaluation are scarcely utilized in the area of physical activity and exercise programs, a systematic literature research was conducted.
Relevant databases were Pubmed (health sciences) and Sport-Discus (sport science). For the comparison of hit numbers, Scopus and Google Scholar were also queried. The following key words were taken from the overview section of the paper written by Coryn et al. : program theory, realist evaluation, theory-driven evaluation, theory of change. Table 1 shows the number of hits.
A first attempt was made to identify different subject fields of the Pubmed hits by carefully reading the abstracts. Out of the 78 articles, 14 were excluded for formal reasons (no abstract available, four; colon between the terms program and theory, one) and for multiple use of the term program theory out of the present scope of evaluation research (biology of aging, three; motor programs, five; disease screening, one). Table 2 shows the identified subject fields.
Only three papers were found that dealt with physical activity: an analysis of general assumptions about using pedometers for activity enhancement , a paper on the development of a daily activity program for individuals with type 2 diabetes , and a qualitative study of adaptations made on the institutional level during a transfer of successfully proved programs into real-world settings . By now, even this small number of studies shows the broad range of opportunities to utilize program theory and theory-driven evaluation in physical activity and exercise programs. More applications are expected to be found during the ongoing literature analysis. In addition, the program theory framework has recently been used to compile a literature review in preparation of a new program .
News from EURAPA
Regarding the European Review of Aging and Physical Activity, the following news can be reported:
This is the first special issue of EURAPA, i.e., an issue with papers focusing on a special subject and guest edited by renowned experts in the field:
A life course approach to physical activity, health, and aging
Harri Suominen and Eino Heikkinen from the University of Jyväskylä, Finland, worked on this important subject. They won several colleagues to author papers for this issue. On behalf of the editorial board, I would like to thank the guest editors as well as all authors for their contributions.
The editors are open for proposals on other remarkable subjects as well as for new candidates for guest editors. Editor-in-chief Heinz Mechling already announced the changes starting with 2011. A call was published for application of future guest editors, for special issues and new editorial board members . Everyone interested should feel free to send a proposal or to apply for a task or position.
As hitherto, in the first issue of the New Year, EURAPA's reviewers in 2010 are acknowledged:
Ian D. Cameron, Australia
Kay Cox, Australia
Jerri Edwards, USA
Sebastião Gobbi, Brazil
Eino Heikkinen, Finland
Timo Hinrichs, Germany
Darko Jekauc, Germany
Urho M. Kujala, Finland
Andrea Lamont-Mills, Australia
Markus Lamprecht, Switzerland
Babette Lobinger, Germany
Lee-Fay Low, Australia
Yael Netz, Israel
Jean Petermans, Belgium
Peter Reaburn, Australia
Annelie Reicherz, Germany
Jörg Schorer, Germany
Harri Suominen, Finland
Risto Telama, Finland
Claudia Voelcker-Rehage, Germany
Wiebren Zijlstra, The Netherlands
They all helped to maintain the high quality of the European Review of Aging and Physical Activity. Thank you very much for your efforts.
The present issue is the first online-only issue. Consequently, from now on EURAPA will no longer be available in a printed version. We have to admit that the editorial board was not happy when we were told about this change. However, we had to understand that the economical situation has recently been difficult. Furthermore, we also had to accept that today the common way of working with academic literature is the digital way. From the submission to the publication process, from literature research to reading and excerpting academic literature—everything is done in front of the computer. If a reader wants to read outside of the office or to put manual comments between the lines, he/she easily would print the paper instead of walking to the library. For promotional reasons, e.g., on occasion of EGREPA conferences, it will still be possible to print and to distribute single issues of EURAPA.
Brach M (2009) Research on exercise programs—an approach of technological science. Eur Rev Aging Phys Activ 6(2):63–65. doi:10.1007/s11556-009-0053-x
Chen HT (1990) Theory-driven evaluations. Sage, London
Chen HT (2005) Practical program evaluation. Assessing and improving planning, implementation, and effectiveness. Sage, London
Coryn CLS, Noakes LA, Westine CD, Schröter DC (2010) A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. doi:10.1177/1098214010389321 (published online 12 November 2010)
Hinrichs T, Brach M (2011) The general practitioner’s role in promoting physical activity to older adults: a review based on program theory. Current Aging Science (in press)
Lattimore D, Griffin SF, Wilcox S, Rheaume C, Dowdy DM, Leviton LC, Ory MG (2010) Understanding the challenges encountered and adaptations made by community organizations in translation of evidence-based behavior change physical activity interventions: a qualitative study. Am J Health Promot 24:427–434. doi:10.4278/ajhp.081024-QUAL-252
Mechling H (2010) Five years of cooperation: EURAPA and Springer Publishers—state of the art, changes, and future development. Eur Rev Aging Phys Activ 7(2):59–60. doi:10.1007/s11556-010-0067-4
Regelin P, Winkler J, Nieder F, Brach M (2008) Fit bis ins hohe Alter. Eine Kurskonzeption zur Erhaltung von Selbständigkeit und Verhütung von Stürzen im Alter (2. Aufl.). Meyer & Meyer: Aachen
Tudor-Locke C, Lutes L (2009) Why do pedometers work?A reflection upon the factors related to successfully increasing physical activity. Sports Med 39:981–993. doi:10.2165/11319600-000000000-00000
Tudor-Locke CE, Myers AM, Rodger NW (2001) Development of a theory-based daily activity intervention for individuals with type 2 diabetes. Diab Educ 27:85–93
About this article
Cite this article
Brach, M. Theory-driven evaluation of exercise programs: often recommended but still only a few hits in literature databases. Eur Rev Aging Phys Act 8, 1–4 (2011). https://doi.org/10.1007/s11556-011-0079-8