mixed methods research case study

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

79k Accesses

34 Citations

16 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• General enquiries: [email protected]

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

A mixed methods case study investigating how randomised controlled trials (RCTs) are reported, understood and interpreted in practice

Ben e byrne, leila rooshenas, helen s lambert, jane m blazeby.

• Author information
• Article notes
• Copyright and License information

Corresponding author.

Received 2019 Nov 26; Accepted 2020 May 6; Collection date 2020.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

While randomised controlled trials (RCTs) provide high-quality evidence to guide practice, much routine care is not based upon available RCTs. This disconnect between evidence and practice is not sufficiently well understood. This case study explores this relationship using a novel approach. Better understanding may improve trial design, conduct, reporting and implementation, helping patients benefit from the best available evidence.

We employed a case-study approach, comprising mixed methods to examine the case of interest: the primary outcome paper of a surgical RCT (the TIME trial). Letters and editorials citing the TIME trial’s primary report underwent qualitative thematic analysis, and the RCT was critically appraised using validated tools. These analyses were compared to provide insight into how the TIME trial findings were interpreted and appraised by the clinical community.

23 letters and editorials were studied. Most authorship included at least one academic (20/23) and one surgeon (21/23). Authors identified wide-ranging issues including confounding variables or outcome selection. Clear descriptions of bias or generalisability were lacking. Structured appraisal identified risks of bias. Non-RCT evidence was less critically appraised. Authors reached varying conclusions about the trial without consistent justification. Authors discussed aspects of internal and external validity covered by appraisal tools but did not use these methodological terms in their articles.

Conclusions

This novel method for examining interpretation of an RCT in the clinical community showed that published responses identified limited issues with trial design. Responses did not provide coherent rationales for accepting (or not) trial results. Findings may suggest that authors lacked skills in appraisal of RCT design and conduct. Multiple case studies with cross-case analysis of other trials are needed.

Keywords: Randomised controlled trial, surgery, Methods, Translational medical research, Health services research, Evidence-based medicine

It is widely recognised that clinical practice is often not in line with the best available evidence. This is the so-called ‘gap’ between research and practice [ 1 , 2 ]. Best evidence predominantly comes from well designed and conducted randomised controlled trials (RCTs) [ 3 ]. However, RCTs are often complex and challenging. Surgical RCTs present specific issues with recruitment, blinding of patients and surgeons, and intervention standardisation [ 4 ]. Many of these issues have been clarified with methodological research [ 5 – 10 ]. Such work has led to improvements in trial quality over time [ 11 , 12 ]. However, the gap between trials and implementation of their results in practice persists [ 13 ], potentially compromising patient care and wasting resources. Reasons for the disconnect are myriad.

Trial findings that report putative evidence for a change in clinical practice may not be implemented because of poor conduct and reporting [ 14 ], limitations in generalisation and applicability [ 15 ], cost, and unacceptability of new interventions. Clinical culture may emphasise the importance of experience over evidence [ 16 ], and some clinicians may have limited numeracy skills required to understand and apply quantitative results from trials [ 17 ]. Appropriate understanding of RCTs is critical to implementation and of vital importance to clinicians, researchers and funders. We have previously described a novel approach to explore understanding and interpretation of RCT evidence, by examining writings about individual surgical trials [ 18 ]. The present study aims to apply this new method to a single case study: the TIME (Traditional Invasive versus Minimally invasive Esophagectomy) RCT [ 19 ]. The purpose is to better understand how this trial has been interpreted and to illustrate the potential of this novel approach.

The methodology used in this study has been described in detail elsewhere [ 18 ] and will be summarised here. The approach represents a form of case-study research, comprising mixed methods analysis of documentary evidence relating to a published RCT [ 20 ]. Case-study approaches have been defined in various ways and used across numerous disciplines. Their central tenet is to explore an event or phenomenon in depth and in its natural context [ 21 ]. The ‘real-world context’ in this study was the landscape of published articles that interpreted, appraised and discussed implementation of the TIME trial’s findings. Our approach aligned with Stake’s ‘instrumental case-study’ [ 22 ], using a particular case (the TIME RCT’s outcomes paper) to gain a broader appreciation of the issue or phenomenon of interest (in this case, interpretation and appraisal of RCTs in the clinical community, and implications for implementation). We conducted qualitative analysis of selected published articles citing this RCT’s primary report and compared this with structured critical appraisal of the RCT using established tools. We also sought to demonstrate the utility of this novel approach, which we intend to apply in future case studies.

Identify and analyse articles citing a trial

Purposefully select a major surgical rct.

An index RCT was identified and summarised as the case of interest. We sought a highly cited trial report, published in a high-impact journal within the last 10 years. The TIME trial [ 19 ], comparing open and minimally invasive surgical access for removal of oesophageal cancer, was selected as it met these criteria and was within our area of expertise.

Identify and systematically sample articles citing the RCT

All articles citing this RCT were identified using Web of Science and Scopus citation tracking tools. Letter, editorial and discussion article types were included. On-line comments were identified using the Altmetric.com bookmarklet. Non-English language articles were excluded. Searches were conducted in October 2017.

Undertake in-depth qualitative analysis and identify relevant themes

Included articles were thematically analysed using the constant comparison technique, adopted from grounded theory [ 23 , 24 ]. Articles were read in detail, with no a priori coding framework. Text was considered against the research topic, which focused on understanding how the authors interpreted, appraised and/or applied the findings of the trial. New findings or interpretations were continuously related to existing findings to develop the data set as a whole (i.e. the constant comparison technique). Coding was not constrained by pre-defined boundaries defining relevance. Rather, this was guided by the content of the articles being analysed. During analysis, it transpired that understanding authors’ interpretations of the RCT required examination of their discussion of evidence from other studies. Therefore, other articles cited by the authors were sought to determine the types of evidence being referenced. The designs of these additional studies were ascertained based on the descriptions in those articles (rather than our assessment).

Analysis was performed by BEB and LR. BEB is a senior surgical trainee and postdoctoral researcher with previous experience of qualitative research. LR is a Lecturer in Qualitative Health Science with an interest in trial recruitment issues, implementation of trial evidence, and experience of working on multiple surgical RCTs. Both researchers work within a department with expertise in trials methodology and have detailed knowledge in this field which is likely to have influenced their identification and coding of relevant themes.

Two rounds of double coding of five articles were performed by BEB and LR. Further coding was conducted by BEB and reviewed among the team to revise coded themes. Descriptive data on authorship and origins of the articles were collected.

Summarise validity and reporting of the RCT

The RCT was assessed by BEB using a range of critical appraisal tools commonly used to appraise RCTs. These included two of the most commonly used tools to assess RCTs: one examining trial reporting in a broad sense (Consolidated Standards of Reporting Trials for Non-Pharmacological Treatments (CONSORT-NPT) [ 5 ]), and another focusing on internal validity as commonly assessed in systematic reviews of trials (the updated Cochrane Risk of Bias Tool (ROBT 2.0) [ 7 ]). In addition, the Pragmatic Explanatory Continuum Indicator Scale (PRECIS-2) tool [ 8 ] was included, to examine domains associated with the broad applicability and utility of the trial, and the Context and Implementation of Complex Interventions (CICI) framework [ 25 ] was included on an exploratory basis to identify broader contextual factors that could be relevant. JMB contributed to assessment during piloting of the tools and in discussion with BEB where there was uncertainty.

Broad comparison of all results to develop deeper understanding of how trials are understood and relationship with trial quality

The results of both qualitative analysis and structured critical appraisal were considered side-by-side, with the overall aim of better understanding how other authors’ interpretations of the TIME trial compared with the critical appraisal guided by the above tools. The qualitative analysis of the authors’ interpretations was conducted before the structured critical appraisal to ensure the coding/themes were grounded in authors’ writings, rather than our experience of conducting the structured appraisals. The final step aimed to draw together both analyses, to see whether authors discussing the trial raised concerns across similar domains to the areas covered by the critical appraisal tools, or whether their topics of discussion addressed other considerations.

Ethical considerations

This study involved secondary use of publicly available written material and did not require ethical review.

Patient and public involvement

Patients and members of the public were not involved in any aspect of the design of this study.

Summary of index RCT

The TIME trial was a two-group, multicentre randomised trial comparing a minimally invasive approach to the surgical removal of oesophageal cancer with an open approach to the abdomen and chest. It was conducted in five centres across four European countries from 2009 to 2011 and is summarised in Table  1 .

Summary of TIME trial

Characteristics of articles

Searches identified 26 articles, and 23 were included (exclusions: an incorrectly classified case report and two articles in German). Summary characteristics are provided in Table  2 . Most articles (18/23, 78%) originated from Europe or the United States. The majority (20/23, 87%) included at least one author holding an academic position; 18/23 (78%) included at least one professor or associate professor (as defined within their own institution). Nearly all included at least one consultant or trainee surgeon (21/23, 91%).

Summary of characteristics of included articles

Altmetric.com identified several references to the TIME trial, detailed in Table  3 . Only one, part of the British Medical Journal blog series, included text discussing the trial, rather than simply restating its results or directing readers to the study report.

Summary of references to TIME trial identified using Altmetric.com (searched 10 September 2018)

Themes identified

Qualitative analysis resulted in description of three key themes: identification of wide-ranging issues with the RCT; limited appraisal of non-RCT studies; and variable recommendations for future practice and research. Codes linking quotes to articles and bibliographic data are provided in supplementary Table  1 .

Identification of wide-ranging issues with the RCT

Authors extensively discussed and critiqued several features of trial design and conduct. These included the population, intervention and outcomes of the trial.

If the author’s primary outcome was focused on pulmonary infection, perhaps other patient associated inclusion / exclusion criteria may have been of value. These would include patients with poor pulmonary function parameters … patients with major organ disease … and recent history of prior malignancy. (E2).

In the present [TIME] trial, the difference between minimally invasive and open oesophagectomy was maximised with a purely thoracoscopic (prone position) and laparoscopic technique. (E1).

The primary outcome … was pulmonary infection within the first 2 weeks after surgery and during the whole stay in hospital. This cannot be considered as the relevant primary outcome with reference to the decision problem outline by the authors … (E5).

Beyond these basic trial design parameters, authors of the citing articles also highlighted important confounding variables.

Many non-studied variables, including malnutrition, previous and current smoking, pulmonary comorbidities, functional status, and clinical TNM (tumour, node, metastasis) staging, have all been shown to strongly affect the primary endpoint of this trial – postoperative pulmonary infection. (L2).

Several correspondents suggest that lower rates of respiratory infection might have been achieved by use of alternative strategies for preoperative preparation, patient positioning, ventilator settings, anaesthetic agents, or postoperative care. (L6).

The articles also covered other potential problems with the trial, such as sample size and learning curve effects.

The sample size for sufficient statistical power for major morbidity, survival, total morbidity and other similarly important outcomes may actually be larger. (E2).

The inclusion criteria for participating surgeons appears to have the performance of a minimum of only 10 MIOs and this low level of experience may be reflected in relatively high conversion rate of 13%. (E4).

Only one article (E2) made clear statements praising aspects of the trial:

‘…The protocols for the RCT appear sound with randomization, intention to treat, PICO … and bias elimination.’

The next sentence of this article balanced these positive comments with discussion of limits due to the lack of blinding and other potential confounding variables.

Limited appraisal of non-RCT studies

Authors often cited other types of evidence in the same field to support their views without discussing their methodological limitations. Types of evidence included single-surgeon series, non-randomised comparative studies, systematic reviews (SRs) and meta-analyses (MAs).

Luketich et al. , one of the earlier pioneers of MIE, reported their extensive experience of 1033 consecutive patients undergoing MIE with acceptable lymph node resection, postoperative outcomes, and a 1.7% mortality rate. (L8).

In a population-based national study, … the incidence of pneumonia was 18.6% after open oesophagectomy and 19.9% after minimally invasive oesophagectomy … (L3).

Although systematic reviews and a large comparative study of minimally invasive oesophagectomy have not shown this technique to be beneficial as compared with open oesophagectomy, some meta-analyses have suggested specific advantages. (E1).

The existing SRs and MAs were discussed in relation to the intervention and its outcomes, without directly relating them to the TIME trial itself. The implications for authors’ impressions of the TIME trial findings were generally unclear.

There was limited appraisal of these SRs and MAs, especially when contrasted with discussion of the TIME trial. Several authors referred to the large, single-surgeon series of MIO by Luketich, but only one author described limits of this single-institution non-comparative study.

We must not rely on the limitation of single-institution studies and historical data. This procedure must be broadly applicable and not the domain of a few experts for it to become the new gold standard. (E12).

A few others highlighted the limits of other study designs, but there was a striking disparity in the level of critique, when compared with that of the TIME trial.

In their systematic review … Uttley et al. correctly conclude that due to factors such as selection bias, sufficient evidence does not exist to suggest the MIO is either equivalent to or superior to open surgery. (E6).

All these studies however, concede that due to a lack of feasible evidence by way of prospective randomized controlled trials (RCT), no definitive statement of MIE ‘superiority’ over standard open techniques can be made. (E2).

Although several authors referred to the existing SRs and MAs, none reported the design of the included primary studies, which were largely retrospective and non-randomised.

Variable recommendations for future practice and research

The authors had differing interpretations and recommendations for implementation based on the TIME trial. Some articles discussed issues with the trial and did not make recommendations for future practice, in some cases asking for additional information to better understand or interpret the trial.(L1, L3–5) For example, one simply wrote that the authors ‘have several concerns’, before reporting differences in outcomes between TIME and other studies, and describing practice in their own institution. (L1) Others reported that more work was required, such as further analysis of long-term results of patients included in TIME, or called for further trials in different patient populations.

However, the main issue which this study [TIME] does not address is that of long-term survival. … If the authors can indeed demonstrate at least equivalent long-term oncological outcome for MIO and open oesophagectomy, then this paper should provide an impetus for driving forward the widespread adoption of MIO. (E4).

Of interest will be whether similar results can be repeated in patients in Asia, with mainly squamous cell cancers that are proximally located. … The substantial benefit shown in this trial [TIME] … might encourage investigators to do further randomised studies at other centres. If these results can be confirmed in other settings, minimally invasive oesophagectomy could truly become the standard of care. (E1).

One article (E6) considered the evidence for MIO, discussed this against methodological aspects of a colorectal trial evaluating a minimally invasive approach, before restating the findings of TIME, opining that:

‘This study confirms that RCT [sic] for open versus MIO is indeed possible, but further larger trials are required.’

Later in that article, the authors suggested extensive control of wide-ranging aspects of perioperative care would be important for future trials.

Authors of three articles (E7, E9, E11) suggested that the available evidence was enough for increasing adoption of MIO.

…The available evidence increasingly favors a prominent role for minimally invasive approaches in the management of esophageal cancer. Endoscopic therapies and minimally invasive approaches offer at least equivalent oncologic outcomes, with reduced complications and improved quality of life compared with maximal surgery. (E11).

We are close to a situation in which one can argue that MIE is ready for prime time in the curative treatment of invasive esophageal cancer. If we critically analyse the level and grading of evidence, the current situation concerning MIE and hybrid MIE is far better than was the case when laparoscopic cholecystectomy, anti-reflux surgery, and bariatric surgery were introduced into clinical practice. (E9).

No authors called for the cessation of MIO, although one referred to some centres stopping ‘their MIE [minimally invasive esophagectomy] program due to safety reasons’. (E13).

Assessment of RCT using validated tools

The TIME trial results and protocol papers [ 19 , 26 ] were examined to assess the trial and its reporting. Assessment using CONSORT-NPT demonstrated reporting shortfalls in several areas (full notes in supplementary Table  2 ). These included: lack of information on adherence of care providers and patients to the treatment protocol; discrepancies between the primary outcomes proposed in the protocol (3 pulmonary outcomes) and the trial report (one pulmonary result); no information on interim analyses or stopping criteria; a lack of information regarding statistical analysis to allow for clustering of patients by centre; and absence of discussion of the trial limitations or generalisability.

Risk of bias was assessed as shown in Table  4 . Overall, the TIME trial was considered at high risk of bias.

Risk of bias determined using the Cochrane Risk of Bias Tool 2.0

Assessment using the PRECIS-2 tool is shown in Table  5 . Overall, TIME had features in keeping with a more pragmatic rather than explanatory trial. This suggested a reasonable degree of applicability and usefulness to wider clinical practice.

Assessment of the TIME trial using the PRECIS-2 tool

Application of the CICI framework highlighted several higher-level considerations relevant to the applicability of the TIME trial not described in the protocol or study report (see Table  6 ). These included lack of detail on the setting, as well as epidemiological and socio-economic information.

Notes on domains relevant to the implementation of MIO based upon the CICI checklist

Overall, these tools suggested that TIME had several limitations. These included issues with standardisation and monitoring of intervention adherence, lack of blinding, failure to use hierarchical analysis and a lack of information on provider volume. The risk of bias was high, limiting confidence attributing outcomes to the allocated interventions. Broad applicability was considered reasonable, though study utility was compromised by a short-term clinical outcome, rather than longer term or patient-reported outcomes. While TIME may have provided early evidence for benefit of MIO to reduce pulmonary infection within 2 weeks of surgery, the appraisal suggested more evidence was needed before considering wider adoption of MIO.

Broad comparison of all results to develop deeper understanding

We considered the findings from the qualitative analysis in relation to those of the critical appraisal. In doing so, broad domains of internal and external validity seemed a useful system to bring together results of both analyses. While the ROBT was described by its creators as focused on internal validity, the PRECIS-2 and CICI tools were not described in terms of validity. Rather, their authors referred to applicability and reproducibility in other settings, which may also be described as external validity. CONSORT-NPT is a tool focused on reporting of trials, and its authors referred to both domains, with some duplication of factors covered in the other tools. However, authors of the articles included in the qualitative analysis did not adopt such methodological terminology when expressing concerns about these aspects of the index RCT’s conduct or reporting.

Robust internal validity allows confident attribution of treatment effects to the experimental intervention. The ROBT identified high risk of bias in the TIME trial. Qualitative analysis revealed discussion of various aspects relevant to internal validity. For example, several authors discussed differences in patient positioning and anaesthetic techniques. These confounding variables may have introduced systematic differences in care between groups, aside from the allocated intervention, resulting in bias. However, the article authors did not articulate the implications of their concerns in such terms and did not consider whether these problems rendered the trial fatally biased.

Sound external validity suggests similar treatment effects may be achieved by other clinicians in other settings for other patients. Pragmatic trials have broad applicability, with wide inclusion criteria, and patient-centred outcomes. The PRECIS-2 describes domains relevant to this applicability. TIME had several features of a pragmatic trial, suggesting relatively broad applicability. The qualitative analysis showed authors were concerned about these issues. For example, several discussed the appropriateness and utility of 2-week and in-hospital pulmonary infection rates as the primary outcome measure. However, authors did not directly relate such concerns to external validity or generalisability, to reach a conclusion about whether the trial should influence practice.

While many authors identified issues relevant to internal and external validity, the lack of clear explanation of their implications meant it was difficult to determine whether they thought the trial justified a change in practice. This contrasts with the structured assessments, which defined clear problems with the trial and limits to its usefulness.

This study presents the first application and results of a new method to generate insights into how evidence from a trial was understood, contextualised and related to practice. Qualitative analysis of letters and editorials, largely written by academic surgeons, documented extensive discussion of problems with the trial, but without clear formulation of the implications of these concerns for its internal or external validity and applicability. These authors reached a variety of conclusions about the implications of the trial for surgical practice. A separate assessment using structured tools defined specific weaknesses in trial methodology. Whilst this new approach yielded useful findings in this single case study, the method should be further tested using multiple trials and cross-case analyses. The initial findings based on this single case study suggest a need to clarify standards against which a trial may be assessed to guide decisions about its role in changing practice, and potentially also to guide efforts to influence practitioners to implement change if appropriate. Within this, our findings suggest a need to focus efforts on educating surgeons about trial design and quality, which may contribute to implementation science-based efforts to inform clinical decision-making and implementation of trial results.

This study contributes to the wider literature showing that evidence does not speak for itself. New evidence is often considered alongside competing bodies of existing evidence that may support different ideas, theories or interventions [ 27 , 28 ]. When a study is published, this new evidence is assimilated into the wider scientific context. Its strengths, weaknesses and overall contribution are debated and disputed. Through the lens of Latour’s actor-network theory [ 29 , 30 ], the new trial can be considered a novel actor within the wider network of actors that includes other trials and studies of the intervention, as well as the consumers of this evidence. Those commenting on the trial have an important role in how different features of the trial are identified, discussed and debated, and how its findings are framed. This agency may be influenced by their own clinical experience, education, skill set, work environment and colleagues, amongst other factors. Given these complexities, it is not surprising to find that different authors reached different conclusions about the TIME trial.

The way authors of the included articles used and appraised different types of study raises questions about how the hierarchy of evidence, and the primacy of the RCT, is applied to routine clinical practice. We found extensive criticism of the TIME trial. Article authors described several limitations relating to its population, intervention, associated co-interventions and confounding variables, as well as the outcomes selected. Certainly, the authors presented valid criticisms that limited the trial’s validity, as identified by structured critical appraisal. Over recent years, trials methodologists have worked to better understand and optimise many such aspects of trial conduct. The development of the CONSORT reporting standards promotes detailed description of key methods, such as random sequence generation and allocation concealment, that allow critical judgements about internal validity to be made [ 5 ]. The growth of pragmatic trials, featuring wide inclusion criteria, conducted across multiple sites, with clinically meaningful outcomes, reflects a concerted effort to improve applicability or external validity of RCTs [ 8 , 31 ]. It may never be possible to conduct a ‘perfect’ trial, but improvements in the rigor and transparency of design hopefully ensure that RCTs can provide sufficiently robust evidence that is useful to the broad population of patients and clinicians within a healthcare system. Whether these developments, designed to address valid criticisms of RCTs, are widely understood outside the sphere of trials methodologists is unclear.

Conversely, the authors of the included articles were far less critical of non-RCT evidence. For example, several authors referred to the single-surgeon case-series of Luketich [ 32 ]. Only one author discussed its limitations for generalisation. Surgical skill and performance vary [ 33 ]; what is possible for a single surgeon cannot be generalised to what is usual for most. Similarly, authors cited systematic reviews and meta-analyses without clear description of the original study designs. Evidence synthesis cannot eliminate biases in retrospective, non-randomised studies using statistical techniques. Failure to clearly articulate limitations of these different studies may support our contention that the authors lacked appropriate appraisal skills. Alternatively, it may suggest bias in favour of the intervention, such that the authors understood, but did not want to articulate its limitations.

While RCTs have not been toppled from their position at the top of the hierarchy of evidence about the efficacy of interventions, developments in other areas have seen increasingly sophisticated use of observational data to better understand the effects of treatments. Researchers have taken advantage of increasing availability of vast quantities of genetic data. In epidemiology, the concept of Mendelian randomisation has been used to try and unpick causal relationships from non-causative correlations [ 34 ]. At the patient level, genetic testing of different types of cancer has allowed targeting of treatments according to cellular sensitivities [ 35 ]. The development of such markers by which to tailor treatment have led to proposals of an idealised future whereby individual treatments are entirely personalised according to a panel of markers that accurately predict treatment response and prognosis. These different research approaches are inevitably competing for resources and intellectual priority. However, as has been argued by Backmann, for these other study types to take priority, “what needs to be shown is not only that RCTs might be problematic …, but that other methods such as cohort studies actually have better external validity.” [ 36 ]

Evidence-based medicine aims to apply the best available evidence to individual patients [ 37 ]. This aim, by its very nature, creates a disconnect between evidence from RCTs, which are aggregated studies of groups of patients to determine average effects, and clinical decision-making at the individual level [ 38 ]. This could be considered to represent an insurmountable ‘get-out’ clause, whereby a clinician may always justify deviation from ‘the evidence’ due to differences between the patient in front of them and those included in the relevant study. It may also prove very difficult to allow the theory-based weight of a journal article to over-ride an individual clinician’s personal lived experience of different interventions and their efficacy. This may be particularly problematic in surgical practice [ 16 ] where the practitioner is usually physically connected with the intervention. This may increase the importance attached to experience, even if that experience is at odds with large-scale studies. We do not disagree that clinicians must treat individual patients according to their specific condition and their wishes. However, it may be considered that aggregate practice, across a surgeon’s cases or across a department, should fall roughly in line with an appropriate body of suitably valid and relevant evidence.

Implementation science research has illuminated many factors affecting implementation beyond knowledge of the evidence. Damschroder et al. described the Consolidated Framework for Implementation Research (CFIR) to identify real-world constructs influencing implementation, relating to the intervention, individuals, organisations and systems [ 39 ]. These included ‘evidence strength and quality’ as well as ‘knowledge and beliefs about the intervention’, constructs readily identified within the present study. Their framework also highlights many other important factors such as cost, patient needs and resources, peer pressure, external policies and incentives, and organisational culture. Surgical research has demonstrated wide variation in practice, even in the presence of high quality evidence [ 40 ], and the broad range of factors affecting implementation of interventions, such as Enhanced Recovery After Surgery [ 41 ]. Our approach may contribute as another tool to understand barriers and facilitators to evidence implementation. It may prove particularly useful in conjunction with other methods such as interviews and observations, informed by a relevant framework, such as the Theoretical Domains Framework [ 42 , 43 ].

The early promise of our new method needs further work to conduct multiple case studies of different RCTs to allow cross-case analyses and a more thorough understanding of how RCTs are interpreted and appraised in the landscape of written commentaries. Examination of further case-studies may also inform refinements to the methods. For example, further analyses may indicate recurring themes across case-studies, which may in turn contribute towards a priori coding criteria and more efficient approaches to analyses (e.g. framework analysis [ 44 ]). It will also be important to include assessment of how each trial is situated in the wider context of relevant evidence, across study types. For individual trials, combined qualitative and structured analyses may determine the extent to which that RCT is flawed and requires further evaluation in a more methodologically sound study. Alternatively, it may demonstrate that the problem in bridging the gap between evidence and practice resides in the competition between different bodies of evidence, comprised of different types of study, and appropriate understanding of their strengths and weaknesses, as well as their applicability to practice. Work should also be undertaken to investigate how contemporary practice may have changed alongside publication of such articles, to investigate the relationship between what is written about the trial, and clinical practice as delivered.

While this study has shown the potential of this new method, its strengths and limitations must be considered. Rigorous analysis using robust qualitative methods and double coding by experienced researchers was undertaken. The articles examined were written without knowledge that they would be analysed in this manner, limiting bias this could introduce. The use of multiple tools to assess the index RCT created a broad overview of its strengths and weaknesses. The most important study limitation was that we did not directly explore authors’ understandings and interpretations, so underlying understanding of the key issues was inferred, rather than directly scrutinised. Failure to articulate is not the same as a lack of understanding. Further, we did not ask authors their motivations to publish their articles, an activity with its own significance. In addition, this study attempted to provide insights into the authors understanding and interpretation of the trial, and it does not purport to be an assessment of practice itself, which would benefit from other approaches to investigation (e.g. qualitative observations, interviews, quantitative procedure rate analyses). This study applied our new method to a single, surgical RCT. The issues identified may be particular to that intervention, specialty, or trial design; further case studies are required to determine broader relevance.

This study has successfully applied a new method to better understand how clinicians and academics understand evidence from a surgical RCT - the TIME trial. It identified discussion of many issues with the trial, but the authors who cited the trial did not specifically articulate the implications of these issues in terms of its internal and external validity. The authors reached a wide range of conclusions, ranging from further evaluation of the intervention, to widespread adoption. Structured appraisal of TIME suggested that the trial was at high risk of bias with limited generalisability. Further application of this method to multiple trials will allow cross-case analyses to determine whether the issues identified are similar across other trials and yield information to better understand how this type of evidence is interpreted and related to practice. This approach may be complemented by other data, such as in-depth interviews. This may reveal genuine flaws in trial design that limit application, or that other issues such as poor understanding or competing non-clinical factors impede the translation of evidence into practice. We hope that this work may help existing efforts to close the research-practice gap, and help ensure that patients receive the best care, based upon the highest level of evidence.

Supplementary information

Additional file 1 Table S1. Identifying codes and bibliographic information on all citing articles included in analysis. Table S2. CONSORT-NPT checklist with notes on TIME trial.

Acknowledgements

We would like to thank Cath Borwick, Information Specialist at the University of Bristol for her help developing the literature search strategy, advising on the available tools and highlighting the full range of resources available for this study.

Abbreviations

Randomised controlled trial

Traditional Invasive versus Minimally invasive Esophagectomy

CONsolidated Standards Of Reporting Trials for Non-Pharmacological Treatments

PRagmatic Explanatory Continuum Indicator Scale

Context and Implementation of Complex Interventions

Risk Of Bias Tool

Minimally Invasive Oesophagectomy

Systematic Review

Meta-Analysis

Authors’ contributions

BEB and JMB conceived the study. BEB, LR, HL and JMB developed the protocol and refined the study design. BEB and LR conducted the qualitative analysis. BEB prepared a preliminary draft manuscript. JMB, LR and HL extensively revised the manuscript. All authors have approved the final manuscript.

B E Byrne is supported by the National Institute for Health Research. Jane Blazeby is a NIHR Senior Investigator. This work was undertaken with the support of the MRC ConDuCT-II (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures) Hub for Trials Methodology Research (MR/K025643/1) and the NIHR Bristol Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol (BRC-1215-20011) and support from the Royal College of Surgeons of England Bristol Surgical Trials Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The funders had no role in developing the protocol.

Availability of data and materials

The dataset upon which this work is based consists of articles already available within the published literature.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information accompanies this paper at 10.1186/s12874-020-01009-8.

• 1. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA, et al. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. Br Med J. 1998;317:465–468. doi: 10.1136/bmj.317.7156.465. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 2. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–1230. doi: 10.1016/S0140-6736(03)14546-1. [ DOI ] [ PubMed ] [ Google Scholar ]
• 3. Oxford Centre for Evidence-Based Medicine . Levels of evidence. 2009. [ Google Scholar ]
• 4. Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien P-A, Reeves BC, et al. Challenges in evaluating surgical innovation. Lancet. 2009;374:1097–1104. doi: 10.1016/S0140-6736(09)61086-2. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 5. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P. Methods and processes of the CONSORT group: example of an extension for trials assessing nonpharmacologic treatments. Ann Intern Med. 2008;148:295–309. doi: 10.7326/0003-4819-148-4-200802190-00008. [ DOI ] [ PubMed ] [ Google Scholar ]
• 6. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Br Med J. 2011;343:d5928. doi: 10.1136/bmj.d5928. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 7. Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;10:CD201601. [ Google Scholar ]
• 8. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. Br Med J. 2015;350:h2147. doi: 10.1136/bmj.h2147. [ DOI ] [ PubMed ] [ Google Scholar ]
• 9. McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7:1–8. doi: 10.1186/1745-6215-7-9. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 10. Donovan JL, Rooshenas L, Jepson M, Elliott D, Wade J, Avery K, et al. Optimising recruitment and informed consent in randomised controlled trials: the development and implementation of the quintet recruitment intervention (QRI) Trials. 2016;17:1–11. doi: 10.1186/s13063-016-1391-4. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 11. Antoniou SA, Andreou A, Antoniou GA, Koch OO, Köhler G, Luketina RR, et al. Volume and methodological quality of randomized controlled trials in laparoscopic surgery: assessment over a 10-year period. Am J Surg. 2015;210:922–929. doi: 10.1016/j.amjsurg.2015.04.022. [ DOI ] [ PubMed ] [ Google Scholar ]
• 12. Ali UA, van der Sluis PC, Issa Y, Habaga IA, Gooszen HG, Flum DR, et al. Trends in worldwide volume and methodological quality of surgical randomized controlled trials. Ann Surg. 2013;258:199–207. doi: 10.1097/SLA.0b013e31829c7795. [ DOI ] [ PubMed ] [ Google Scholar ]
• 13. Kristensen N, Nymann C, Konradsen H. Implementing research results in clinical practice - the experiences of healthcare professionals. BMC Health Serv Res. 2016;16:48. doi: 10.1186/s12913-016-1292-y. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 14. Blencowe NS, Boddy AP, Harris A, Hanna T, Whiting P, Cook JA, et al. Systematic review of intervention design and delivery in pragmatic and explanatory surgical randomized clinical trials. Br J Surg. 2015;102:1037–1047. doi: 10.1002/bjs.9808. [ DOI ] [ PubMed ] [ Google Scholar ]
• 15. Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365:82–93. doi: 10.1016/S0140-6736(04)17670-8. [ DOI ] [ PubMed ] [ Google Scholar ]
• 16. Orri M, Farges O, Clavien P-A, Barkun J, Revah-Lévy A. Being a surgeon - the myth and the reality. A meta-synthesis of surgeons’ perspectives about factors affecting their practice and well-being. Ann Surg. 2014;260:721–729. doi: 10.1097/SLA.0000000000000962. [ DOI ] [ PubMed ] [ Google Scholar ]
• 17. Garcia-Retamero R, Cokely ET, Wicki B, Joeris A. Improving risk literacy in surgeons. Patient Educ Couns. 2016;99:1156–1161. doi: 10.1016/j.pec.2016.01.013. [ DOI ] [ PubMed ] [ Google Scholar ]
• 18. Byrne BE, Rooshenas L, Lambert H, Blazeby JM. Evidence into practice: protocol for a new mixed-methods approach to explore the relationship between trials evidence and clinical practice through systematic identification and analysis of articles citing randomised controlled trials. BMJ Open. 2018;8:e023215. doi: 10.1136/bmjopen-2018-023215. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 19. Biere SSAY, van Berge Henegouwen MI, Maas KW, Bonavina L, Rosman C, Garcia JR, et al. Minimally invasive versus open oesophagectomy for patients with oesophageal cancer: a multicentre, open-label, randomised controlled trial. Lancet. 2012;379:1887–1892. doi: 10.1016/S0140-6736(12)60516-9. [ DOI ] [ PubMed ] [ Google Scholar ]
• 20. Yin RK. Case study research and applications: design and methods. Sixth: SAGE Publications; 2018. [ Google Scholar ]
• 21. Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11:100. doi: 10.1186/1471-2288-11-100. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 22. Stake RE. The art of case study research. Thousand Oaks: SAGE Publications; 1995.
• 23. Glaser BG, Strauss AL. The discovery of grounded theory: strategies for qualitative research. Observations. Aldine Transaction. 1967. [ Google Scholar ]
• 24. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101. doi: 10.1191/1478088706qp063oa. [ DOI ] [ Google Scholar ]
• 25. Pfadenhauer L, Rohwer A, Burns J, Booth A, Lysdahl KB, Hofmann B, et al. Guidance for the assessment of context and implementation in Health Technology Assessments (HTA) and systematic reviews of complex interventions: the Context and Implementation of Complex Interventions (CICI) framework. 2016. [ Google Scholar ]
• 26. Biere SSAY, Maas KW, Bonavina L, Garcia JR, van Berge Henegouwen MI, Rosman C, et al. Traditional invasive vs. minimally invasive esophagectomy: a multi-center, randomized trial (TIME-trial). BMC Surg. 2011;11:1–7. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=emed10&AN=71483010%5Cnhttp://nt2yt7px7u.search.serialssolutions.com/?sid=OVID:Embase&genre=article&id=pmid:&id=doi:10.1007%2Fs00464-012-2198-3&issn=0930-2794&volume=26&issue=1&spage=S40&. [ DOI ] [ PMC free article ] [ PubMed ]
• 27. Fitzgerald L, Ferlie E, Wood M, Hawkins C. Interlocking interactions, the diffusion of innovations in health care. Hum Relat. 2002;55:1429–1449. doi: 10.1177/001872602128782213. [ DOI ] [ Google Scholar ]
• 28. Fitzgerald L, Ferlie E, Wood M, Hawkins C. Evidence into practice? An exploratory analysis of the interpretation of evidence. In: Mark AL, Dopson S, editors. Organisational Behaviour in Health Care: The Research Agenda. Palgrave Macmillan; 1999.
• 29. Latour B. Reassembling the social: an introduction to actor-network theory. Oxford: Oxford University Press; 2005..
• 30. Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak. 2010;10:1–11. doi: 10.1186/1472-6947-10-67. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 31. Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375:454–463. doi: 10.1056/NEJMra1510059. [ DOI ] [ PubMed ] [ Google Scholar ]
• 32. Luketich JD, Pennathur A, Awais O, Levy RM, Keeley R, Shende M, et al. Outcomes after minimally invasive esophagectomy: review of over 1000 patients. Ann Surg. 2012;256:95–103. doi: 10.1097/SLA.0b013e3182590603. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 33. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369:1434–1442. doi: 10.1056/NEJMsa1300625. [ DOI ] [ PubMed ] [ Google Scholar ]
• 34. Ebrahim S, Ferrie JE, Smith GD. The future of epidemiology: methods or matter? Int J Epidemiol. 2016;45:1699–1716. doi: 10.1093/ije/dyx032. [ DOI ] [ PubMed ] [ Google Scholar ]
• 35. Jackson SE, Chester JD. Personalised cancer medicine. Int J Cancer. 2015;137:262–266. doi: 10.1002/ijc.28940. [ DOI ] [ PubMed ] [ Google Scholar ]
• 36. Backmann M. What’s in a gold standard? In defence of randomised controlled trials. Med Health Care Philos. 2017;20:513–523. doi: 10.1007/s11019-017-9773-2. [ DOI ] [ PubMed ] [ Google Scholar ]
• 37. Evidence-based Medicine Working Group Evidence-based medicine: a new approach to teaching the practice of medicine. J Am Med Assoc. 1992;268:2420–2425. doi: 10.1001/jama.1992.03490170092032. [ DOI ] [ PubMed ] [ Google Scholar ]
• 38. Kravitz RL, Duan N, Braslow J. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Q. 2004;82:661–687. doi: 10.1111/j.0887-378X.2004.00327.x. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 39. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 40. Urbach DR, Baxter NN. Reducing variation in surgical care. Br Med J. 2005;330:1401–1402. doi: 10.1136/bmj.330.7505.1401. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 41. Gramlich LM, Sheppard CE, Wasylak T, Gilmour LE, Ljungqvist O, Basualdo-Hammond C, et al. Implementation of Enhanced Recovery After Surgery: a strategy to transform surgical care across a health system. Implement Sci. 2017;12:67. doi: 10.1186/s13012-017-0597-5. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 42. Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Heal Care. 2005;14:26–33. doi: 10.1136/qshc.2004.011155. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 43. Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:1–17. doi: 10.1186/1748-5908-7-37. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 44. Ritchie J, Spencer L. In: Qualitative data analysis for applied policy research. Bryman A, Burgess RG, editors. Routledge: Anal Qual data; 1994. [ Google Scholar ]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Data availability statement.

• View on publisher site
• PDF (597.3 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections