Skip to main content

Advertisement

Small samples and increased variability – discussing the need for restricted types of randomization in exercise interventions in old age

Article metrics

Abstract

Background

Randomization provides an equal chance for participants to be allocated to intervention groups, in order to create an equal distribution of all variables at baseline in all groups. However, this is not guaranteed, particularly if the groups are too small, or if the researched groups consist of older adults. The aims of this commentary are to discuss the increased variability in old age which intensifies the risk of baseline inequalities, to elaborate on the need to estimate potential baseline group differences in small samples of older participants in exercise intervention, to discuss alternative procedures for creating equal groups at baseline and to provide specific guidelines for selecting the design of small studies.

Main body

Small groups with increased inter-individual differences may lead to reduced power, thus differences that truly exist may not be detected, or false group differences may appear in the outcome following the treatment. Studies that focused exclusively on older adults have found increased variability in advanced age. Therefore, baseline group differences are more common in older adults as compared to younger persons, and may lead to misinterpretation of the intervention′s results. Imbalances can be reduced by covariate-adaptive randomization procedures, such as stratified permuted-block randomization or minimization. Specific guidelines are provided for selecting a randomization procedure by assessing the probability of unequal groups at baseline in typical, widely used functional tests in old age. A calculation of the required number of participants for creating equal groups for these functional tests is provided, and can be used when increasing the number of participants is possible. R-scripts specifically created for assessing the probability of unequal groups, or for determining the sample size assuring equal groups, are recommended.

Conclusions

In exercise interventions assessing older adults, it is recommended to have a sample large enough for creating equal groups. If this is not possible, as is the case quite often in intervention studies in old age, it is recommended to assess the probability of inequality in the study groups and to apply an alternative randomization.

Background

Randomization does not imply equivalence in small samples

The allocation of participants to experimental conditions via a random procedure is considered to be a fundamental statistical technique in biomedical research. In fact, statistical experiments, including experiments in exercise sciences [1], are defined by the use of randomization [2]. This means that each participant has an equal chance to be assigned to any of the study groups, and this is assured by such means as a coin toss or a table of random numbers.

The value of a control group for determining that the intervention is the only explanation for the change in the experimental group depends on its similarity to the experimental group [1]. Thus, in order to protect against bias, the objective of randomization is to have two groups that are as equal as possible [3]. This helps to ensure that the treatment groups being compared are similar in both measured and unmeasured participant characteristics [4]. It is therefore not surprising that the randomized experiment is often referred to as the gold standard of research – specifically clinical research, including exercise training research [1]. Indeed, many applied fields, for example evidence-based medicine, draw a basic distinction between randomized and non-randomized evidence [5].

However, an equal chance to be allocated to each of the experimental groups does not ensure group equality [4, 6]. That is, it does not guarantee balance in covariate distributions across treatment groups [6]. Imbalanced groups may occur particularly when there are relatively few participants (e.g. 15 to 20 participants per group) enrolled in a trial [4], with one paper even claiming that less than 100 participants in a group can be a problem for achieving equality [7].

In a previous article, we examined the use of stratified permuted-block randomization as an alternative procedure for simple randomization in the case of unequal groups at baseline in motor learning research [8]. The aims of the present commentary are: 1) to discuss the increased variability in old age, which intensifies the risk of baseline inequalities; 2) to elaborate on the need to estimate potential baseline group differences in small samples of older participants in exercise interventions; 3) to discuss alternative procedures for creating equal groups at baseline in these interventions; and 4) to provide specific guidelines for the design of small samples in these interventions.

Main text

Increased variability in old age

In exercise sciences, age is a moderating variable in almost all areas of research, primarily in relation to physical variables but also to psychological, social and behavioral variables, which are often studied in relation to exercise or fitness. One study examining the relationship between chronological and biological age showed that the variability in physical deterioration starts as early as age 38 [9]. Advancing age is typified by increasing variability in both physical [e.g. 10] and mental [e.g. 11] fitness between individuals. Therefore, in clinical interventions in exercise science age should be clearly defined, and if the range of the participants’ ages is large, then the participants should be divided into age groups and the intervention will be assessed per age group.

Even studies focusing exclusively on older adults have found that inter-individual variability shows a quantitative increase in advanced age in both physical and behavioral aspects. For example, a study on gait measures in people aged 60–86 reported a relationship between age and gait variability after adjustment for height, weight and chronic disease. Older age was associated with greater variability in all gait measures that are commonly associated with falls [10]. Another study assessing postural stability in older adults reported an age variability within this age group on static and dynamic balance, concluding that balance performance is task-specific in older adults. Thus, it was recommended that various dynamic and static balance tests be used for assessing postural balance ability in old age [11]. As postural stability is known to be specifically sensitive to age, normative data provided for a unipedal balance test (one-leg stance), for example, were divided into subgroups of 5 years each (65–69, 70–74, 75–79, etc.) [12].

Age variability in older adults was also indicated in the relationship between postural control and specific aspects of cognition, such as attention control [13]. Increased variability in old age has also been reported in response to aerobic exercise interventions [14], executive functioning [15] and reaction time [16]. Specifically, increased variability in reaction time was described in relation to aerobic fitness [17], falls and gait [18], and even to mortality in old age [19]. It has been argued that both inter- and intra-variability in reaction time was greater in older as compared with younger adults [20].

The need to examine potential baseline group differences in small samples of aging populations

Given the increased variability in old age, it may be possible that after a procedure of simple randomization, participants in controlled studies will have baseline differences in the outcome measures or in other relevant covariates. This is particularly important in small groups of participants (e.g. 15–30 participants in each group), in which the number of covariates that can be balanced in stratified randomization is largely limited [6]. That is, in addition to the increased variability in old age, a small number of participants may indicate less statistical power. A study with high statistical power has a greater probability of detecting a specified treatment effect at any level of statistical significance. Statistical power is inversely related to variance of the difference between two means [7]. A study comprised of small groups with increased inter-individual differences may have a reduced power, and thus fail to detect differences that actually exist.

Furthermore, a bias that may occur in small groups with unequal baseline scores may lead to false group differences, in favor of the experimental group, in the outcome following the treatment. This may happen when the randomization process allocates participants in a way that the active treatment group has a better chance to improve than participants in the control group [7]. Specifically, it may occur when the baseline scores of the experimental group are lower than those of the control group – thus giving the experimental group “more room” for improvement.

Examples from the literature are presented in Table 1. In these examples participants were randomly allocated to experimental and control groups. However, the scores were not equal at baseline, with the experimental group scoring lower than the control group. In two studies [21, 23], the post-test scores of the experimental group were equal to the pre-test scores of the control group. The baseline differences between the experimental and control groups may have indicated the possibility that the controls did not improve because they were performing at ceiling level in terms of the potential range of improvement, whereas the exercise group had some room for improvement [23].

Table 1 Examples of small groups with unequal baseline scores with experimental groups’ scores lower than the control group

In a previous article [8] we provided a flowchart and R scripts for examining the probability of inequality between groups at baseline, and for selecting an effective randomization strategy. Based on these R scripts, we calculated the chances for baseline group differences in the Gomes et al. [22] and Netz et al. [23] studies (this was not possible in the Rehfeld et al. study [21], as no SDs were provided). Assuming a mean of 16, a SD of 5 and 15 participants in each experimental group, we found a 66% chance for a 5% baseline group difference, a 38% chance for a 10% difference, and an 8% chance for a 20% difference (see Table 1). These calculations suggest that it is quite possible that the groups may have not been equal at baseline.

Thus, although these studies reported that the treatment was beneficial, it is likely that had the participants in the experimental groups performed better at baseline, their performance following the exercise treatment would have shown less improvement. Hence, it is possible that the treatment only helped those who had poor scores at baseline. In other words, the interaction indicating greater improvement for the experimental than the control group may not indicate that the treatment was actually beneficial.

Alternative procedures for creating equal groups at baseline

Imbalances between groups can be reduced in small sample-size studies by restricting the randomization procedure. Restricted randomization means that simple randomization is applied within defined groups of participants [4, 24]. Covariate-adaptive randomization is the most commonly used procedure for creating balance in relevant covariates across treatment groups [6, 24].

The two leading techniques of covariate–adaptive randomization are stratified (block) randomization and minimization [1, 6, 24]. Stratified randomization controls treatment imbalances within each covariate stratum [24]. It creates a separate randomization process, usually a permuted block design, for each specific stratum formed by a combination of the levels of the relevant covariates [6]. For example, in studies assessing the effect of a certain exercise intervention on cognitive functioning, fitness level and gender will be typical strata. The limitation of this technique is that if the sample size is small, the number of strata is very limited. Too many covariates (strata) means a small number of participants in each stratum or even empty strata. If only a few strata are allowed the imbalance could be reduced, but would still be present [7].

While stratified randomization is a predetermined technique, with participants being assigned to groups in advance, minimization enables an on-going process of group allocation. Minimization achieves balance in treatment assignments across factor levels, by choosing the allocation for the new participant that would lead to the smallest possible degree of imbalance across the set of his or her baseline characteristics [6]. More specifically, in stratified randomization an algorithm is applied to distribute participants to each stratum. Minimization, on the other hand, enables the control of imbalance in covariates not included in the stratification algorithm, such as unknown baseline differences in the outcome measures [24]. In cases of unplanned changes in the study, even re-randomization is recommended before breaking the blind method of the treatment in clinical trials [25].

Balanced treatment groups should be comparable in most relevant aspects except for the specific component applied in the intervention group. However, comparability between the study treatment groups depends on the phenomenon studied. Gender, for example, is important with respect to cardiovascular fitness and to other physical measurements such as static and dynamic balance [13], but is independent of global cognitive functioning. That is, experimental groups do not need to be identical in any respect; it suffices that they are alike with respect to the outcome variable under study [5]. On the other hand, more so than gender, age is quite often considered a moderating variable in clinical trials.

Assessing the probabilities of inequalities after simple randomization in typical functional tests in small samples of aging populations – practical examples

Based on the R scripts proposed in our previous article [8], we calculated the probabilities of inequalities after simple randomization in selected typical functional tests in small samples of aging populations. The tests included were: Timed Up and Go, Walking Speed, Functional Reach, Sit-to-Stand, Handgrip Strength and Unipedal Balance (one-leg stance). We based our calculations on means and SDs reported in the literature for these tests. Tables 2 and 3 present our calculations for men and women, respectively. It should be noted that in these calculations, we suggested a threshold for accepting the probability of inequality (10% or 15% or 20%), above which we think it may be too risky to use simple randomization. However, such a threshold should be determined by researchers based on the specific research domain, previous experience and previous results.

Table 2 The probabilities of inequalities after simple randomization in groups of older men, based on scores of selected functional tests
Table 3 The probabilities of inequalities after simple randomization in groups of older women, based on scores of selected functional tests

For example, in samples of 15 participants per group there is a probability of 24% that there will be a 10% difference between groups in Timed Up and Go in both women and men. In Sit-to-Stand there is a probability of almost 40% (Tables 2 and 3). Special attention should be attributed to balance measurements known to produce large standard deviations, indicating substantial significant variability. These large standard deviations are clearly demonstrated in a wide range of normative data on the unipedal balance test [12]. Therefore, it is not surprising that we calculated a probability of 23% for having a 20% difference between groups of 25 participants in both genders (Tables 2 and 3), and a probability of 64% (!) for having a 10% difference in groups of 15 participants (Tables 2 and 3).

Clearly, it can be argued that research should be conducted with large samples to ensure equal groups. However, in most cases this is quite difficult, especially in intervention studies in older populations. The recruitment is difficult and the rate of attrition is high. Consequently, intervention studies published in journals with a relatively high impact factor are based on small groups. For example, Voelcker-Rehage et al. [32] based their study on two groups of 16 and 17 participants, Rehfeld et al. [21] on 12 and 14, and Eggenberger et al. [33] on 19 and 14. Furthermore, this last study provided support for the relatively small sample by conducting a power analysis. This analysis had revealed a sample size of 17 participants per group based on α-level of 0.05 and effect size of 0.25. It should be noted that a power analysis is quite acceptable in research as a criterion for determining samples size. On the other hand, this criterion does not guarantee equal groups.

A researcher can calculate the number of participants required for creating equal groups for a specific study. In order to illustrate these calculations, we used the same examples provided in Tables 2 and 3, which included the expected means (±SDs) in typical, widely used functional tests for older adults. Table 4 presents our calculations. In the Time-Up-and Go test, for example, in order to allow for no more than 10% difference 44 women and 42 men are recommended in a group, and in the unipedal balance test, 264 women and 262 men in a group, which is clearly unrealistic. If creating equal groups by increasing the sample size is unachievable, it is our recommendation to examine the probability of inequality and possibly acquire an alternative type of randomization.

Table 4 Number of participants required to reach equal groups at baseline in simple randomization (values are based on the expected mean ± SD for men and women presented in Tables 2 and 3)

Summary and recommendations

An experiment with random allocation to groups based on equal chance is often referred to as the gold standard of research. However, an equal chance to be allocated to any group does not guarantee group equality. Imbalanced groups may occur, particularly when there are relatively few participants enrolled in a trial.

In the exercise sciences, age is a moderating variable in almost all areas of research, primarily in relation to physical variables but also to psychological, social and behavioral variables, which are often studied in relation to exercise or fitness. Studies focusing exclusively on older adults have found that variability shows a quantitative increase in advanced age. Given this increased variability in old age, it may be possible that experimental groups of a small sample size (e.g. 15–30 participants per group) will have baseline differences in the outcome measures as well as in relevant covariates. Evidently, a researcher should strive for a sample size large enough to warrant equality. However, this is quite challenging in intervention studies in aging populations. It is therefore recommended to examine the probability that simple randomization will lead to group differences at baseline.

Imbalances between groups can be reduced in studies with a small sample size by restricting the randomization procedure, which means that that simple randomization is applied within defined groups of participants. Covariate-adaptive randomization is the most commonly used procedure for creating balance in relevant covariates across treatment groups, and the two leading techniques of covariate-adaptive randomization are stratified (block) randomization and minimization. R scripts for calculating probabilities of inequalities, and the number of participants required for creating equal groups by simple randomization have been proposed in a previous article [8]. Additional practical information on assessing and implementing randomization techniques has recently been published [34].

And last but not least – in this commentary we discussed the randomization process in interventional studies, elaborating on inter-individual variability typical to aging, which may cause unequal groups at baseline. However, as age increases, both inter- and intra-individual variability are increasing [20]. The intra-individual variability may affect the relationship between pre- and post-tests in interventional studies as well. If participants perform differently in a single task on different occasions, the chances to observe true changes as a result of an intervention is reduced. Strategies to deal with this issue should be discussed in future research.

Conclusion

Given this increased variability in old age – it is recommended in small groups of older adults to examine the probability of baseline differences before conducting a simple randomization and, if necessary, to apply a restricted randomization technique.

Availability of data and materials

Not Applicable

References

  1. 1.

    Hecksteden A, Faude O, Meyer T, Donath L. How to construct, conduct and analyze an exercise training study? Front Physiol. 2018;9:1007. https://doi.org/10.3389/fphys.2018.01007.

  2. 2.

    Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin Company; 2002.

  3. 3.

    Hilgers RD, Uschner D, Rosenberger WF, Heussen N. ERDO-a framework to select an appropriate randomization procedure for clinical trials. BMC Med Res Methodol. 2017;17(1):159. https://doi.org/10.1186/s12874-017-0428z.

  4. 4.

    Broglio K. Randomization in clinical trials: permuted blocks and stratification. JAMA. 2018;319(21):2223–4.

  5. 5.

    Saint-Mont U. Randomization does not help much, comparability does. PLoS One. 2015;10(7):e0132102.

  6. 6.

    Lin Y, Zhu M, Su Z. The pursuit of balance: an overview of covariate-adaptive randomization techniques in clinical trials. Contemp Clin Trials. 2015;45:21–5.

  7. 7.

    Kernan WN, Viscoli CM, Makuch RW, Brass LM, Horwitz RI. Stratified randomization for clinical trials. J Clin Epidemiol. 1999;52(1):19–26.

  8. 8.

    Ziv G, Lidor R, Netz Y. Dealing with possible baseline inequalities between experimental groups – the case of motor learning. J Mot Behav. 2019;7:1–12. https://doi.org/10.1080/00222895.2019.1649996 https://doi.org/10.1080/00222895.2019.1649996.

  9. 9.

    Belsky DW, Caspi A, Houts R, Cohen HJ, Corcoran DL, Danese A, et al. Quantification of biological aging in young adults. Proc Natl Acad Sci. 2015;112(30):E4104–10. https://doi.org/10.1073/pnas.1506264112/-/DCSupplemental.

  10. 10.

    Callisaya ML, Blizzard L, Schmidt MD, McGinley JL, Srikanth VK. Ageing and gait variability—a population-based study of older people. Age Ageing. 2010;39(2):191–7. https://doi.org/10.1093/ageing/afp250.

  11. 11.

    Dunsky A, Zeev A, Netz Y. Balance performance is task specific in older adults. Biomed Res Int. 2017. https://doi.org/10.1155/2017/6987017.

  12. 12.

    Bohannon RW, Tudini F. Unipedal balance test for older adults: a systematic review and meta-analysis of studies providing normative data. Physiotherapy. 2018;104(4):376–82.

  13. 13.

    Netz Y, Zeev A, Dunsky A. Postural control and posture-unrelated attention control in advanced age—An exploratory study. Maturitas. 2018;116:130–6. https://doi.org/10.1016/j.maturitas.2018.08.003.

  14. 14.

    Whipple MO, Schorr EN, Talley KM, Lindquist R, Bronas UG, Treat-Jacobson D. Variability in individual response to aerobic exercise interventions among older adults. J Aging Phys Act. 2018;26(4):655–70. https://doi.org/10.1123/japa.2017-0054.

  15. 15.

    West R, Murphy KJ, Armilio ML, Craik FI, Stuss DT. Lapses of intention and performance variability reveal age-related increases in fluctuations of executive control. Brain Cogn. 2002;49(3):402–19. https://doi.org/10.1006/brcg.2001.1507.

  16. 16.

    Bielak AA, Cherbuin N, Bunce D, Anstey KJ. Intraindividual variability is a fundamental phenomenon of aging: evidence from an 8-year longitudinal study across young, middle, and older adulthood. Dev Psychol. 2014;50(1):143.

  17. 17.

    Bauermeister S, Bunce D. Aerobic fitness and intraindividual reaction time variability in middle and old age. J Gerontol B Psychol Sci Soc Sci. 2014;71(3):431–8. https://doi.org/10.1093/geronb/gbu152.

  18. 18.

    Graveson J, Bauermeister S, McKeown D, Bunce D. Intraindividual reaction time variability, falls, and gait in old age: a systematic review. J Gerontol B Psychol Sci Soc Sci. 2015;71(5):857–64. https://doi.org/10.1093/geronb/gbv027.

  19. 19.

    Kochan NA, Bunce D, Pont S, Crawford JD, Brodaty H, Sachdev PS. Is intraindividual reaction time variability an independent cognitive predictor of mortality in old age? Findings from the Sydney Memory and Ageing Study. PLoS One. 2017;12(8):e0181719.

  20. 20.

    Hultsch DF, MacDonald SW, Dixon RA. Variability in reaction time performance of younger and older adults. J Gerontol B Psychol Sci Soc Sci. 2002;57(2):P101–15.

  21. 21.

    Rehfeld K, Müller P, Aye N, Schmicker M, Dordevic M, Kaufmann J, et al. Dancing or fitness sport? The effects of two training programs on hippocampal plasticity and balance abilities in healthy seniors. Front Neurosci. 2017;11:305. https://doi.org/10.3389/fnhum.2017.00305.

  22. 22.

    Gomes GC, do Socorro Simões M, Lin SM, Bacha JM, Viveiro LA, Varise EM, et al. Feasibility, safety, acceptability, and functional outcomes of playing Nintendo Wii Fit PlusTM for frail older adults: A randomized feasibility clinical trial. Maturitas. 2018;118:20–8.

  23. 23.

    Netz Y, Tomer R, Axelrad S, Argov E, Inbar O. The effect of a single aerobic training session on cognitive flexibility in late middle-aged adults. Int J Sports Med. 2007;28(01):82–7. https://doi.org/10.1055/s-2006-924027.

  24. 24.

    Zhao W. A better alternative to stratified permuted block design for subject randomization in clinical trials. Stat Med. 2014;33(30):5239–48. https://doi.org/10.1002/sim.6266.

  25. 25.

    Proschan MA. Re-randomization tests for unplanned changes in clinical trials. Clin Trials. 2017;14(5):425–31. https://doi.org/10.1177/1740774517710657.

  26. 26.

    Ibrahim A, Singh DK, Shahar S. ‘Timed Up and Go’test: Age, gender and cognitive impairment stratified normative values of older adults. PLoS One. 2017;12(10):e0185641.

  27. 27.

    Atkinson HH, Rosano C, Simonsick EM, Williamson JD, Davis C, Ambrosius WT, et al. Cognitive function, gait speed decline, and comorbidities: the health, aging and body composition study. J Gerontol Series A: Biological Sci Med Sci. 2007;62(8):844–50.

  28. 28.

    Bohannon RW, Wolfson LI, White WB. Functional reach of older adults: normative reference values based on new and published data. Physiotherapy. 2017;103(4):387–91.

  29. 29.

    Rikli RE, Jones CJ. Functional fitness normative scores for community-residing older adults, ages 60-94. J Aging Phys Act. 1999;7:162–81.

  30. 30.

    Jansen CW, Niebuhr BR, Coussirat DJ, Hawthorne D, Moreno L, Phillip M. Hand force of men and women over 65 years of age as measured by maximum pinch and grip force. J Aging Phys Act. 2008;16(1):24–41.

  31. 31.

    Lohne-Seiler H, Kolle E, Anderssen SA, Hansen BH. Musculoskeletal fitness and balance in older individuals (65–85 years) and its association with steps per day: a cross sectional study. BMC Geriatr. 2016;16(1):6.

  32. 32.

    Voelcker-Rehage C, Godde B, Staudinger UM. Cardiovascular and coordination training differentially improve cognitive performance and neural processing in older adults. Front Hum Neurosci. 2011;5:26.

  33. 33.

    Eggenberger P, Wolf M, Schumann M, de Bruin ED. Exergame and balance training modulate prefrontal brain activity during walking and enhance executive function in older adults. Front Aging Neurosci. 2016;8:66.

  34. 34.

    Uschner D, Schindler D, Hilgers RD, Heussen N. RandomizeR: An R package for the assessment and implementation of randomization in clinical trials. J Stat Softw; 2017.

Download references

Acknowledgements

The authors extend their deepest thanks to the Academic College at Wingate for their willing to cover the article processing charges. In addition, the authors thank Ms. Dinah Olswang for the language corrections.

Funding

No funding

Author information

The three authors collaborated on the concept of the paper. YN did the writing and contributed specifically to the specific section on the increased variability in old age. RL contributed to the general background on the importance of randomization in studies in general and specifically on exercise studies. GZ did the calculations presented in the table and elaborated on restricted randomization techniques. All three authors reviewed the manuscript and edited it. All authors read and approved the final manuscript.

Correspondence to Yael Netz.

Ethics declarations

Ethics approval and consent to participate

Not Applicable

Consent for publication

Not Applicable

Competing interests

The author declares that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Netz, Y., Lidor, R. & Ziv, G. Small samples and increased variability – discussing the need for restricted types of randomization in exercise interventions in old age. Eur Rev Aging Phys Act 16, 17 (2019) doi:10.1186/s11556-019-0224-3

Download citation

Keywords

  • Unequal groups
  • Biased results
  • Stratified randomization
  • Small study groups
  • Probability of baseline differences