Medical students’ and tutors’ experiences of directed and self-directed learning programmes in evidence-based medicine: a qualitative evaluation accompanying a randomised controlled trial
Authors
Peter Bradley (see below)
Christina Oterholt, Medical student, University of Oslo, Norway, E-mail: [email protected]
Lena Nordheim, Research librarian, Norwegian Centre for Health Services Research, Oslo, Norway,
E-mail: [email protected]
Arild Bjørndal, Medical director, Norwegian Centre for Health Services Research, Oslo, Norway,
E-mail: [email protected]
Corresponding author/main contact:
Peter Bradley, Acting head, Pharmaceutical reimbursement section Norwegian Medicines Agency, Sven Oftedals vei 8, 0950 Oslo Norway
Tel + (47) 22 16 84 33 Fax + (47) 22 89 77 99
E-mail: [email protected] (28th June to 31th July excluded)
May be contacted on from 28th June to 28th July on [email protected]. Please mark “for the urgent attention of Peter Bradley”.
Address for correspondence Brekkekroken 1, 1430 Aas, Norway
About the authors
Peter Bradley is currently acting head of the Pharmaceutical Reimbursement Section at the Norwegian Medicines Agency. Christina Oterholt is a medical student at the University of Oslo, Norway. Lena Nordheim is research librarian and Arild Bjørndal medical director at the National Centre for Health Services Research, Norway. All but CO have previously been actively involved in teaching evidence-based practice.
Acknowledgements
We would like to thank Julia Bradley-Norman, Claire Glenton and Louise Forsetlund at the Norwegian Centre for Health Services Research for editorial advice, and the course tutors and administrative staff working for the Medical Faculty at the University of Oslo for the teaching programme.
Abstract
This qualitative study aims to interpret the results of a randomised controlled trial comparing two educational programmes (directed learning and self-directed learning) in evidence-based medicine (EBM) for medical students at the University of Oslo, 2002/2003. There is currently very little comparative educational research in this field. In the trial, no statistically significant differences between the study groups were shown for any outcomes considered (EBM knowledge, skills and attitudes).
Further analysis suggests that main reason for the “negative” trial results was that the majority of students learned equally effectively, whichever programme they received, although implementation of the educational programmes was not complete, because of varying attendance. This study illustrates a step-wise evaluation model that might be useful in evaluating other socially complex interventions.
Abstract word count 122 words
Keywords evidence-based practice, medical education, evaluation of complex interventions, directed learning, self-directed learning
Word count 6526 (main text only), 9133 words (everything)
Background
Aims and objectives
This article describes a qualitative evaluation accompanying a randomised controlled trial comparing two educational interventions in EBM for 10th semester medical students at the University of Oslo, Norway in 2002/3 (Bradley et al. 2004b). One teaching programme was based on computer-assisted self-directed learning. The other was organised as directed learning, organised as workshops.
Evidence-based medicine (EBM) has been defined as ‘the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients’ (Sackett et al. 1996), and consists of the following 5 steps:
formulating clinically important questions
efficiently gathering of clinical evidence (research)
critical appraisal (assessment) of evidence
applying evidence to practice
evaluating own practice
The literature on how to investigate complex social interventions (including educational interventions) is extensive. Broadly speaking, a quantitative and a qualitative tradition can be identified. With regard to educational interventions within medicine, quantitative research has often followed a “behavioural tradition” focusing on observable outcomes such as improvement in cognitive, psychomotor and affective terms (“Bloom’s taxonomy”) measured as changes in knowledge, skills and attitudes. This approach has focused on
experimental designs such as randomised controlled trials and has been hypothesis-driven with the aim of producing results, which are generalisable to other situations. On the other hand, qualitative evaluation has followed the “humanistic tradition” and focused mainly on the process of learning, participant and tutor experience and course satisfaction (Bradley 2004a).
Recently, several authors have outlined the limitations of evaluations of social complex interventions, which only adopt of a single research approach. They have therefore emphasised the advantages of combining qualitative and quantitative methodologies (Speller 1997; Britton 1998; Buckley 1998; Hutchinson 1998; Hatala and Guyatt 1999, Stacey and Spencer 2000; Wolff 2001). As Wolff explains, “by collecting more information about the process it may be possible to disentangle the causal pathway that produces the observed effect and thereby enhancing its validity and reliability” (Wolff 2001)
In our evaluation, the combination of approaches was assumed to offer a complementary approach, where the trial could yield relatively objective data on the effect of educational interventions for the chosen outcomes (“how many, how often” questions) and the qualitative evaluation could help us to explain and understand the trial’s results by describing the educational process, and suggesting causes for variation within the study groups.
Before the evaluation began, we thought it likely that students would learn equally effectively, whichever programme they received, but that learning experiences would differ.
The accompanying RCT subsequently showed no statistically significant differences between the two groups for all outcomes (critical appraisal skills, knowledge about or attitudes to EBM)(Bradley 2004b).
The main research questions were:
1) Were the educational interventions delivered according to plan?
2) How did students interact with learning materials, fellow students and tutors in order to learn? Which other factors influenced learning?
3) What experiences did students and tutors have of the education programmes offered?
eg, satisfaction, performance, relevance and usefulness?
4) How did experiences of the two educational programmes differ for individual students and tutors?
5) What are participants’ anticipated and initial experiences of using evidence-based medicine in clinical practice?
6) How can the results be applied to future courses and learning?
Educational theory and findings from an extensive systematic literature review of quantitative and qualitative literature (Bradley 2004a) were used to develop the initial research questions and plan the interventions (see below). In addition, conceptual frameworks were created to show how factors impacting on learning might interact and differ in the two groups. These were gradually supplemented and refined as data was collected, analysed and new hypotheses developed. The conceptual frameworks are shown in figure 1 and a detailed list of final research questions is available from the authors.
Existing quantitative research
In the literature review, ten quantitative systematic reviews were identified which considered educational promoting critical appraisal for medical students or qualified doctors (Audet et al. 1993; Burls 1997; Norman and Shannon 1998; Green 1999; Hyde et al. 2000; Taylor et al.
2000, Ebbert et al 2001; Parkes et al. 2003; Coomarasamy et al. 2003). Educational interventions were mostly classroom-based didactic learning courses. The interventions had generally positive effects concerning knowledge gain and showed possible improvements in critical appraisal skills, attitudes to EBM, use of medical literature and reading habits.
Similar results were obtained for a more recent randomised controlled trial (Taylor et al.
1999), comparing CASP workshops (similar to the directed learning intervention) with no intervention. No trials considered patient outcomes or the process of care or compared different types of interventions. One additional review considered searching for evidence, but consisted in poor quality studies (Garg and Turtle 2003).
Existing qualitative research
Most of the included qualitative studies in the review accompanying this trial, considered educational interventions for evidence-based practice (EBP) for qualified health professionals. One of the main conclusions was that educational uptake seemed to be directly linked to the perceived relevance of courses to professional practice, but it was not clear which educational method participants preferred. It also seems likely that practical access to educational resources, self-perceived skills, and cultural and social norms influence as well as course content affect perceptions about course relevance (Bradley 2004a).
In addition, two systematic reviews were identified which considered studies on computer- based learning (CAL) for healthcare students (not EBP) (Greenhalgh 2001, Enock 2000).
Overall, these have only showed modest results in terms of student learning. One included qualitative study suggested that CAL was associated with time wasting, frustration and technical problems, whereas another qualitative study, where CAL was restricted to a
supervised classroom setting (similar to the self-directed intervention), found no such problems.
Participants and settings
The study was conducted in 2002/3 among 10th semester medical students and their tutors during a course in EBM at the Faculty of Medicine in Oslo, Norway’s largest medical school.
The first intervention took place in Spring 2002 and the last in Spring 2003 involving three cohorts. The new EBM course was one of many recent changes in medical undergraduate teaching at the University. Recent changes included the introduction of problem-based learning (PBL) in 1996 when traditional lectures and clinical placements were combined with teaching in small PBL groups. A new curriculum for the 10th semester was introduced in Spring 2001 (Lycke et al. 2002, figure 2). The new EBM course, evaluated in this study, was piloted in autumn 2001, a process which led to some small format changes, before formal evaluation began in spring 2002. Each semester, the EBM course was held in the first two weeks of the 10th semester and consisted in 5 half-day sessions. In the rest of these two weeks, students received didactic classroom-based teaching on other topics. The EBM course was not related to PBL activities. After the teaching, students continued with 12 weeks of clinical placements, where they met in groups of 7-8 for tutorials and had considerable patient contact.
The educational interventions
All students received one of the two educational interventions in EBM which covered the 5 EBM steps, as described above. The two interventions were chosen because they were familiar to the authors and educational quality could be guaranteed. Both educational interventions covered the same syllabus (figure 3).
The two educational interventions, referred to as directed learning group and self-directed learning group aimed to compare two distinct teaching methods explicitly based on adult learning theory (andragogy). Both approaches partially met the principles of andragogic and social cognitive theory, which aims to promote learning by increasing “self-efficacy”. The notion of “self-efficacy” attempts to explain how individuals cope with new social situations and choose a course of action. Self-efficacy considers that the effort invested in performing a task (e.g. a learning task), can be understood in terms of the individual’s motivation, various intervening factors (or barriers), the risk involved and the expected reward (efficacy).
Reflection on the balance of these factors is decisive for whether an individual acts.
The directed learning intervention promoted “self-efficacy” through structured or directed group-based learning. On the other hand, the self-directed intervention promoted “self- efficacy” through increased “self-directedness”. Self-directedness can be defined as increased personal responsibility for learning, control over and independence to pursue learning tasks. Table 1 explained how the interventions met the theoretical criteria for each educational approach.
After the course, students from both groups had access to medical bibliographic databases (eg, Cochrane Library, Medline) and other internet resources through the University’s web pages.
Directed learning group
Training was organised as tutor-led workshops, based on the methodology of the Critical Appraisal Skills Programme (CASP) based in Oxford, England (See
http://www.phru.nhs.uk/learning). A typical workshop lasted three hours and consisted of an interactive introductory talk, small group work and large group feedback. Teaching was conducted in Norwegian, using English-language articles for critical appraisal. Participants had received a pre-workshop pack containing a glossary of terms, an outline of the workshops and relevant articles. The interactive introductory presentation introduced basic terms and ideas. The small groups were used to consider clinical scenarios or critically appraise an article in a systematic way using a pre-determined checklist. The small group discussions were reported and discussed in the large group session. Participants received a post-workshop pack on the last day of the course, containing notes from all the sessions and some relevant articles and references.
The workshops aimed to establish a safe, supportive atmosphere. Curricula, methods for learning, learning resources and strategies were pre-determined. Learning objectives were stated in the introduction to each lecture. Students were encouraged to control discussion in the small groups and to ask questions during lectures and in plenary sessions. Clinically relevant scenarios were used during all sessions. Feedback on learning was obtained informally during group discussions from peers and tutors and learning was reinforced by material contained in the post-workshop packs.
Self-directed learning group
Training relied on computer-assisted learning, mainly using an English-language CD-rom (and accompanying workbook) written by the Critical Appraisal Skills Programme (see http://www.phru.nhs.uk/learning). The CD-rom consisted of 5 modules following the syllabus described above. In the CD-rom, students were presented with course notes and interactive exercises which posed questions and gave automatic feedback on answers. The
CD-rom also provided checklists to appraise articles, a glossary of terms, several sample articles to appraise and links to key internet sites. The accompanying workbook included all the necessary source material including several additional examples of scientific articles to appraise, further exercises, references and checklists to appraise them. Since the workbook was in English, it was supplemented with non-interactive internet pages in Norwegian. The internet resource contained a glossary of terms, checklists to appraise articles and further references. In addition, tutors (one clinical epidemiologist and one librarian) were available at 5 specified teaching sessions lasting three hours to answer any technical or academic questions. The students were encouraged to use the tutors actively. At the first session, the tutors’ role and choice of learning resources was explained and students were told they could keep the CD-rom and workbook.
The self-directed learning intervention aimed to establish a safe, supportive atmosphere. The curricula and learning methods were pre-determined, but students were encouraged to formulate their own learning needs and personal learning objectives (day 1 of the teaching week), using a copy of the curriculum as reference point. Students were encouraged to control the time, pace and site of their learning and to choose any of the available learning methods: interactive CD-rom, workbook, internet resources, background literature and tutor- support. Students could choose which sessions they attended. An automatic facility in the CD-rom gave immediate feedback on learning.
Roles of the authors
Three of the authors (PB, LN, AB) were main tutors for the EBM course and are otherwise involved in teaching EBM. AB and PB are public health doctors, LN a research librarian.
CO is a medical student, new to EBM and was not present during the teaching weeks. PB
and LN are actively involved in the CASP International Network (linked to the Critical Appraisal Skills Programme (CASP) which developed the and financially subsidised the evaluated educational programmes).
Methods
The methods for data collection and analysis were based on those described by Miles and Huberman (Miles and Huberman 1994). Computer software was not used to analyse data, as the methodology described below allowed systematic analysis of the relatively limited data set available.
Ethical considerations
The research project was approved by the local ethical committee in 2001. All students gave written consent to participation in the research project. Students received information on the course and its accompanying evaluation by e-mail, verbally during teaching sessions (in the semester before and on day one of the course), before the consent forms were signed.
Confidentiality of data was ensured by using coding procedures which prevented person identifiers being accessed at the same site as interview data. In addition, data was kept in a password protected network system.
Sample selection
192 students from the first two of three cohorts (spring and autumn 2002) evaluated in the randomised controlled trial, were invited to participate in the evaluation. Data was collected from students including the 40% of students (n=78) who did not participate in the randomised controlled trial, either because they refused consent or they did not meet the
inclusion criteria, as reported elsewhere(Bradley 2004a). By including all potential students it was judged that a more comprehensive evaluation would be achieved.
Data collection
All students who attended teaching from autumn 2001 to autumn 2002 were asked to complete a 2-page questionnaire of 6 questions about course satisfaction, appropriateness, time use and organisation according to which course they had attended. The questionnaire consisted of closed-questions answered using a Likert scale and open-questions answered with free text. The questionnaire was based on one developed to evaluate similar EBM courses (CASP workshops and CASP CD-rom) in 1997 (Burls 1997) and 1999 (Enock 2000). The questionnaire was piloted with a group of medical students in spring 2001 resulting in minor revisions. The final (translated) version of the questionnaire is available from the authors. The questionnaire was completed anonymously on the final course day or soon afterwards by all students who had attended the course. The questionnaire results were also used to identify and refine research questions posed during the interviews and focus groups.
Participant observation was conducted with the first two student cohorts (autumn 2001 and spring 2002) by PB and LN whilst they were teaching, during others’ teaching and during students’ discussions. Informal conversations and student and tutor behaviour were noted.
The observations were used to evaluate the educational process, develop an interview guide and select candidates for interview (see below).
Interviews: Students and tutors were selected using purposive sampling balanced with respect to gender and intervention attended, in order to cover a wide range of experiences. Students
and tutors were selected on a one-to-one basis by PB. About half the candidates were judged to be representative of the group and half had expressed particular praise or criticism for the courses. Just over 20% of the candidates refused to be interviewed, stating that they were too busy.
The interviews were held by CO in the institutes where teaching had taken place. She was not known to the students and it was anticipated that CO might encourage more openness from interviewees than a more senior person or course tutor. Interviews and focus groups were loosely based around an interview guide based on the research questions, conceptual frameworks and participant observation, which gave structure but allowed a “natural”
conversation to develop. The interview schedules are available from the authors. At the beginning of each interview, CO explained the interview’s purpose, her own background, and asked students to complete a further written consent form for taping the interview. A light snack was provided for the students.
The first student cohort was interviewed in the last 3 days of their first teaching block (semester week 2, n = 6), ie, before they started clinical placements. Two tutors were also interviewed at this time. Data was then analysed before moving on to the next stage of data collection.
To cross-check the results so far and investigate alternative perspectives, two focus groups were held with different students during the last three days of the teaching block in the second cohort (semester week 2). The focus groups used the findings from the first cohort to develop a new interview guide (available from the authors). Students were recruited to the focus groups using the same sampling strategy as for the interviews. One focus group was
held with 5 participants from the directed learning group and one with 4 participants from the self-directed learning group. The focus groups were held at the Medical Faculty in Oslo (one of the teaching sites) and lasted for 100 and 60 minutes respectively. The sessions were introduced as for the interviews. CO moderated the discussions, while PB observed, took notes and summarised the discussions. Towards the end of the focus groups, the main conclusions from the discussion were fed back to verify that messages had been clearly understood.
More focus groups were originally planned, but lack of student time made them impossible.
Instead we held further one-to-one interviews during lunch times nearer the students’ exams with different students from the cohort. Data was re-analysed and a further round of interviews was planned to investigate issues which had not been properly covered e.g.
students’ experiences of using EBM in clinical placements, as well as cross-checking results.
A new interview guide was again developed (available from the authors). The second student cohort was interviewed just before the final teaching block (semester week 15, n = 2 (3 students as one interview was with 2 students)), after clinical placements. Interviews were then stopped as saturation had been reached on a number of issues. The relative timing of the data collection methods is shown in figure 4
All interviews and focus group discussions were transcribed by CO and checked by PB. This only led to minor content changes. A plan to ask for student comments on the final data summary via e-mail was abandoned, as a poor response was anticipated and the stepwise data collection method had explicitly sought feedback from the students as hypotheses developed.
Method for analysis
The summarised questionnaires, notes from participation observation and transcribed texts from the interviews and focus groups were coded independently by PB and CO using a coding framework based on the conceptual frameworks and research questions. The initial coding system was not optimal as several quotations were difficult to classify despite the coding system’s relative complexity. Further discussion between PB and CO resulted in the production of “bridging codes” within and across developing topic categories or themes. The number and complexity of developing themes was further reduced after discussion. This process was repeated twice, until final themes were identified. All data was then re-classified accordingly. Quotations were selected from all existing text, either because they succinctly expressed a majority point of view, identified an important minority opinion or highlighted an area of particular interest warranting further investigation. Disagreement was resolved through discussion.
Analytical frameworks
Three main frameworks have been used to apply structures to data collected 1) The pre- formulated main research questions (as stated above), 2) educational theory, focusing on Knowles principles of adult learning theory (andragogy), self-directed learning and social learning theory and 3) the results of existing research. The analytical frameworks and the results of existing research are fully described in the accompanying review (Bradley, 2004a)
Results
In total 46 of 93 (49%) questionnaires were completed from the self-directed group and 72 of 144 (50%) from the directed learning group. From 10 interviews, 2 focus groups and participant observation, the eight themes were identified. . Participant observation was used almost exclusively to comment on the educational process, whereas the other data collection
methods identified the other themes. Most issues were consistently reported in the questionnaires, interviews and focus groups, except where otherwise stated.
Educational process
Direct observation of students suggested that the courses followed the project plan and that learning took place. In the self-directed learning group, students initially used about 80% of their time working alone with the educational materials, either critically appraising the articles in the workbook or using the CD-rom. Most students posed few questions to the tutors in the first two sessions (2 to 3 questions per student per session) and these were generally of a more technical nature (e.g. how to install the CD-rom). A minority of students (about 20%) worked together in groups of 2 to 3, particularly whilst critically appraising articles. The students who attended subsequent sessions seemed to gain confidence with this method of working and the subject matter. They posed more questions to the tutors (up to 6- 8 questions per student per session). Attendance rate varied between 20-80% per session, which was not surprising, as students had been specifically told to take responsibility for the pace and site of their learning.
Direct observation of the directed learning group showed that the introductory talk was generally received passively, with only a few questions to the tutor (about 5 to 6 questions per talk) and limited discussion between the students. However, in the small groups, students constantly discussed issues between themselves whilst critically appraising papers. At the beginning of the small group session, some students struggled to use the pre-prepared checklists for appraising studies, but seemed more confident with their content in the feedback session. In these feedback sessions, about 20-30% of the students spontaneously asked questions to the tutor and at times openly disagreed with fellow students or tutor.
Other students helped one another (2 to 3 times per session), by verbally explaining key terms (e.g. “intention-to-treat analysis”) in front of the whole group in an effort to “correct” a perceived misunderstanding. Fairly often, the tutors allowed student discussion to continue uninterrupted for up to 5 minutes. Students extensively used the pre-prepared educational materials ie, the articles and checklists for appraising studies. Attendance per workshop was between 50 to 85%. Students seemed to become more confident in discussions as the teaching week progressed.
However a minority of students in the directed learning group did not join in any discussions and chose to work on their own during the group work (about 5%). It was also noted that occasionally students seemed to lose confidence if they asked a question, when fellow students commented negatively eg, considered re-explaining a term as needless repetition.
The interaction between the observable factors promoting learning: tutors, students (as a group), students (as individual learners), tutors, ongoing learning and the learning resources seemed to differ between the two groups, but was largely consistent with initial conceptual frameworks, which required only minor modification (see figure 1).
General satisfaction
Most students evaluated the specific course for EBM positively. In the questionnaires the vast majority of students felt the course was useful or very useful for their studies, met their expectations and was appropriate in content. The results are shown in table 2.
This was consistent with the interviews and focus groups, where most 17 of 19 students were generally satisfied with their EBM course. However two students from the self-directed learning group expressed dissatisfaction (see below).
Most students from both groups specifically mentioned that it was useful for them to learn how to search in databases and critically appraise articles. A typical comment was:
‘Well, it is very useful in connection with our assignment and the like to appraise articles and learn about Medline. I am not good at these searching programmes and I think it has been great to learn a bit about Medline and Cochrane and such things.’
(student “211”, 1st cohort)
One of the focus groups highlighted the need for further EBM courses to promote students’
interest in EBM. This issue was not raised in the other interviews.
In contrast, the rest of the teaching in the first two weeks was reported to be of lower quality compared to the 5 EBM half days in nearly all interviews, both focus groups and many questionnaires, even though students were not directly asked to comment on the whole two weeks teaching.. A typical comment was:
‘(The rest of the teaching block was) messy, uninformative, it was difficult to see from the lecture titles what they would be about, (there was) a lot of introduction (to new topics) which has been very…has not been very well focused and a lot of talking without having learned anything.’ (Student “262”, 1st cohort)
Specific experiences of the programmes
Most self-directed learning students stated in their questionnaires and interviews that the topic was suitable for self-directed learning. Several students mentioned their initial fear or disappointment when allocated to the self-directed learning group, as they felt they would not have sufficient computer skills. However, most students were positive to self-directed learning at course completion. Nearly all the students from the self-directed learning group stated that it was an advantage to be able to work at their own speed. A typical comment was:
‘Well, we got the CD-rom and sat and worked with it even if we also asked those who were there and spoke to each other. It has actually been…it has actually been fine, erhm…because we have quite a few lectures at the moment, so I think it has been quite acceptable to work on your own and with the exercises in the CD-rom, which we could answer since we had the prepared answers, so I think it has been really great….a really good course.’ (229)
Nearly all the interviewed self-directed learning students specifically mentioned that they would have liked more discussion with their fellow students, eg, in critically appraising papers and during the statistics module. In connection with this, several students felt it was not easy to use the CD-rom alone as it gave ready-prepared feedback, which they did not always agree with and further discussion was not possible. Similar comments were made in the corresponding questionnaires.
Two interviewees from the self-directed learning group stated that they had got “little out of the course” as they had lacked the motivation necessary to go through the CD-rom and workbook conscientiously and intended to read a recommended text on EBM in preparation for the forthcoming examination. They also mentioned difficulties with English-language
materials. Another student wanted more time because of difficulties reading so much
“specialised” English. Similar comments were made by a minority of students in the questionnaires, but most students felt they had enough time to cover the topics and stated that there were no language problems. Direct observation had not highlighted language as a major issue, as few questions on English terms were posed.
Interviewees from the directed learning groups commented little on the details of the workshops. However, several students said they would have got more from the course if they had done more reading on the topic during the course weeks.
Individual directed learning group students mentioned: problems with the introductory lecture suggesting concepts were introduced too quickly without sufficient explanation (interviews from 1st student cohort) and that there was content overlap between lectures (interviews from 2nd student cohort). The directed learning focus group mentioned they would have liked more time critically appraising literature with the opportunity to compare good and poor quality scientific papers. Similar comments were made in the questionnaires.
Pariticipant observation and an interview with one of the tutors who worked with both groups suggested that the self-directed learning group had worked more independently and generally asked more “advanced” questions than the directed learning group. Interviewed students felt they did not have enough information to comment on this issue when asked.
The tutor said:
‘Interviewer: Hmm, so you noticed a difference between the two groups?
Tutor: Yes, I noticed the difference, definitely they….they (self-directed learning group) were more independent, as they ought to have been, that was the point of the CD-rom in a way, that they should have self-directed learning, they were not supposed to have someone tell them what to do..’
(tutor 2, 1st cohort)
Interviewed tutors felt both groups’ sessions had gone well, except for some organisational problems (see below). The tutors emphasised that medical students were generally challenging to teach because they were used to more traditional didactic teaching. Direct observation and interviews with tutors suggested that the workshops felt a little rushed at times, as students had more questions than could be answered during the time available.
However, tutors felt that the basic syllabus had been covered.
Time use and preferences
In the evaluation form, students expressed satisfaction with how much time was devoted to lectures, group work and PC. The groups differed most on the following points. 42% of the self-directed learning group (versus 0% in the directed learning group) would have liked more time with lectures. Whereas 47% of directed learning students (as opposed to 14% of the self-directed students), would have liked more time using the computer. 46% of the directed learning and 49% of the self-directed learning students would have liked more time for feedback.
Most students felt that the course that they had attended was most suitable for them. Two interviewed students and both tutors spontaneously suggested that a combination of methods might be best.
Both tutors and a majority of students felt that personality, learning style and previous learning experience ultimately determines preferences for learning method. Most students considered gender differences unimportant factors in determining preferred learning styles, but a minority in the focus group discussions, suggested that male students might prefer the CAL course.
Neither tutor stated a preference for which teaching method they felt was most effective. A tutor interview went as follows:
Interviewer: Which type of teaching do you think is most suitable for the students, CAL or teacher-based?
Tutor: I think that it is very dependent on personality, and which learning patterns you have and by which method you learn best. I think that for many students, in any case for those who are a little more independent and who like to go more deeply into things by themselves, a CD-rom like this or internet-based course where you acquire knowledge on your own, is great, better than being told things….’
….But there are also those who need to have people to ask and prefer the traditional teaching methods, it has something to do with what they are used to, of course. (tutor 1, 1st cohort)
Timing of the course
All tutors and students, in the focus groups and interviews, commented that the course came too late in their studies. This issue was not covered in the questionnaire. The majority of
students suggested that the course should be held before the 8th semester, when they start an assignment involving a literature search and evidence review on a particular topic.
In the two focus groups students were concerned about the timing of the course. A typical discussion from one of the focus groups was:
‘Student 1: Another point is that it is stupid having the course now, when you think what we are about to do.
Student 2: Yes.
Interviewer: Yes, you mean clinical work? What would you have preferred to do in these two weeks?
Student 3: General practice
(focus group for directed learning students, 2nd cohort)
Being a study participant
No students or tutors felt that the trial had adversely affected the learning experience. A number of students admitted initial scepticism to being a study participant. However, students told us, informally and formally, that they would have liked more information about the study before the course start.
None of the interviewed students admitted using the other group’s learning resources, but suspected that others had done so, without being able to think of specific examples. One student from the directed learning group said that s/he would try to get hold of the CD-rom before the examination.
Student attendance registers were kept and students were not allowed entry to sessions for the opposing intervention. Despite this, two individual students from the self-directed learning group attended one of five sessions intended for the directed-learning group, but were not discovered until the end of the session. These students were not allowed entry to further directed-learning sessions in this cohort.
Course organisation
In the questionnaire, self-directed learning group, 91% of the students reported the course had been organised well or adequately as opposed to 97% in the directed learning group.
There were a number of organisational problems mentioned by the students in the interviews and questionnaires. In the self-directed learning group, two main problems were; technical problems with CD-readers on computers and interruptions from non-course students entering the computer room presumably unaware of the course.
The tutors were more concerned about organisational problems in the directed learning group; the lack of space available for students during the computer-based workshops, and that that room set up forced students to sit in rows, discouraging active discussion.
Use of EBM in practice
Students about to go on clinical practice were divided about their ability to use research results in clinical practice. Some felt that their skills could be used directly, particularly as they would be more focused in their use of medical literature. Others were aware of potential barriers to using research in practice – particularly lack of time and internet access. A typical discussion was heard in one of the focus groups (our comments in brackets):
‘Interviewer: Do you think that you will use your knowledge (of EBM) from the teaching in practice?
Student 1: Yes
Student 2: In my assignment in any case
Student 1: The real question is whether we will get the chance? I worked for six weeks this summer in Ullevål (a local hospital) and you have a hundred questions a day, which is both good and bad. You ask the consultants and they give you clear answers, and they don’t use any information sources. You haven’t got internet access. You can of course run through Ullevål (to gain access to a computer room), but you haven’t got time, have you? So you have got to use the knowledge you have, there and then.
(focus group for self-directed learning students, 2nd cohort).
After clinical placements, two of three students reported having extensively sought research results and applied them to practice. The remaining student had not used them at all. Those using EBM, stated that they had been encouraged to do so by their clinical placement tutor(s). Those not using EBM, stated that lack of time had been their main barrier.
Discussion
Negative views from study participants might easily have been overshadowed by the authors of this evaluation who are enthusiasts for EBM. To increase the validity of the findings, robust research methodology was used incorporating theoretical, professional, methodological and investigatory triangulation. Data was collected from tutors and sources using more than one method, several theoretical perspectives were used to interpret the findings and the researchers had different professional backgrounds. Interview candidates
were selected deliberately to represent a broad range of viewpoints, not only those who appeared positive to the course. Data was collected and transcribed by the most impartial member of the group (CO), then systematically analysed by two members who resolved disagreement by discussion (PB and CO).
The questionnaire response rate was only 50%, but was complimented by information from and the interviews and focus groups so, the authors believe saturation was reached on a number of issues. It is likely that those who chose not to attend the educational sessions were underrepresented. Consequently, some issues such as reasons for course dissatisfaction have probably not been fully covered. This may be particularly the case in the self-directed group where course attendance was lower.
Potential explanations for the “negative” results for the accompanying randomised controlled trial were that: the programmes were insufficiently distinct from one another; were inadequately implemented, organised or of poor quality; students were demotivated by being denied educational choice, extensive “contamination” (e.g. sharing of educational materials) occurred or students learned effectively regardless of teaching method (Albanese 2000).
Participant observation and student interviews suggested the interaction between the observable factors promoting learning differed between the two groups, and was largely consistent with the conceptual frameworks for each group (table 1 and figure 1). This suggests the programmes were sufficiently distinct to allow a true comparison and were implemented according to the project plans based on distinct educational theories. However, the there was varying uptake of the programmes, because of non-attendance.
Overall, the programmes seem to have been of adequate and similar quality. Students’
unanimously stated that EBM should be taught earlier in their studies, so it could be applied to a compulsory research-based assignment. However, the majority of students were still satisfied with both programmes, found them useful and a good use of time. A slightly higher proportion of directed learning students evaluated their programme positively. Both educational programmes were also more highly evaluated than the didactic teaching the students otherwise received. This finding is consistent with social cognitive and adult learning theory, which suggests that interactive educational interventions motivate learners, if directly relevant for the fulfilment of their professional roles.
9% of self-directed and 3% of directed students felt the programmes had been poorly organised. However, no technical difficulties were reported unlike other CAL studies (Greenhalgh 2001). Suggestions for improved course organisation were focused on better teaching rooms and computer facilities. Some self-directed students commented the CD-rom did not fulfil their expectations for technical or educational flexibility, whereas directed learning students criticised topic overlap in lectures.
There were only a limited number of reports of “contamination” between the study groups.
Students suggested some directed learning students may have obtained the CD-rom, but no specific cases were identified. Students were not allowed to attend the sessions of the opposing study group, which with was enforced almost completely by using attendance registers.
Tutors and a majority of students suggested that personality, learning style and previous learning experience determined preference for learning method. Educational research also
suggests that learners can be broadly classified into two main groups, as self-directed, problem-centred learners who are internally motivated or teacher-dependent learners who are externally motivated, although it is also recognised that learning styles are context specific and may vary for individuals with time and place (Richardson and Birge 1995).
Although, the proportion students expressing dissatisfaction was similar and in both groups, more self-directed students felt directed learning would have been “best for them” (15% v 3%). Attendance rates were also lower in the former group, but this is not surprising as students were specifically encouraged to choose the site and timing of their learning.
However, it is still possible that this lower attendance expressed student dissatisfaction and
“non-compliance”.
The transition to self-directed learning may have been more demanding, as students had little experience with self-directed learning but were used to small group PBL working. Other studies have shown that learners can take time adapting to computer-assisted, self-directed learning (Bradley 2004a). Against this, most students stated a preference for the intervention they had actually received and felt that being part of a research project had not adversely affected their learning.
Students were divided about how likely they were to use EBM skills in clinical practice, but suggested that this partly depended on external factors. The following barriers and facilitators to implementing EBM had already been experienced during clinical placements;
students’ perception of their individual competence within EBM and its relative importance, the timing of the EBM course, EBM’s role in patient consultations, time barriers in clinical
practice, support from local tutors and access to the internet. These barriers are similar to those described in similar studies of qualified doctors (Bradley 2004a).
Conclusion
The two educational programmes seemed to be sufficiently distinct to allow a true comparison and were implemented according to the project plan. Most students and tutors evaluated the programmes positively, suggesting they were of adequate quality. Students unanimously stated they would have liked EBP teaching earlier in their studies but emphasised the programmes were still relevant and useful.
Only a minority of students suggested factors which may have weakened the implementation of the programmes and thus contributed to a “negative result”. The main issues were varying attendance rates, difficulty in adapting to self-directed learning and poor course organisation.
Contamination between the study groups also seems to have been limited. Students and tutors also suggested that personality, learning style and previous learning experience determined preferences for learning method. This suggests that enforced allocation of an educational programme may have demotivated some students. However, most students stated a preference for the intervention they had actually received. It seems unlikely that these factors offer a full explanation of the “negative” result. It seems more likely that the majority of students learned effectively, whichever programme they received.
There is currently very little research that compares educational programmes for EBP, or considers self-directed, computer-based learning. The results from this study and the accompanying randomised controlled trial suggest that both directed and self-directed learning were successfully introduced at the University of Oslo and may be relevant for other
medical schools. However, the students had prior PBL experience, which may have facilitated small-group working in the directed learning group. On the other hand, the students suggested that these isolated short courses in EBM would not be enough to change their professional practice, a finding consistent with other research. More research is needed on this issue.
This study is part of a step-wise evaluation model, developed to compare two educational programmes. Firstly, a systematic literature review was conducted to summarise existing research relevant to the evaluation. Secondly, relevant theory and research were used to plan the educational programmes. Thirdly, the programmes were compared in a randomised controlled trial. Finally, qualitative data was used to explain the trial results. The authors suggest that this model could be useful in evaluating other socially complex interventions.
References
Alabanese M. 2000. Problem-based learning: why curricula are likely to show little effect on knowledge and clinical skills. Medical Education 34:729-738
Audet N., Gagnon R. 1993. Ladouceur R., Marcil M., L’enseignement de l’analyse critique des publications scientifiques médicales est-il efficace? Révision des études et de leur qualité méthodologique? Canadian Medical Association Journal 148(6):945-52
Bradley P., De la Harpe D., Nordheim L., Innvær S., Thompson C. 2004. Educational interventions for evidence-based practice (EBP). A review of the qualitative literature.
Submitted Cochrane Qualitative Methods Review Group. York, England (Bradley 2004a) Bradley P., Oterholt C., Herrin J., Nordheim L., Bjørndal A. 2004. A comparison of directed and self-directed learning programmes in evidence-based medicine for medical students: a randomised controlled trial. Submitted British Medical Journal. London, England (Bradley 2004b)
Britton A., Thorogood M., Coombes Y., Lewando-Hundt G. 1998. Search for evidence of effective health promotion. Quantitative outcome evaluation with qualitative process evaluation is best. British Medical Journal 316:703-4
Buckley G. 1998. Partial truths – research papers in medical education. Medical Education, 32:1-2
Burls A.J. 1997. An evaluation of the impact of half-day workshops teaching critical appraisal skills. Unpublished report. Institute of Health Sciences, Oxford, England:.
Coomarasamy A., Taylor R., Khan K S. 2003. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Medical Teacher 25(1):77-81
Ebbert J.O., Montori V.M., Schultz H.J. 2001 The journal club in postgraduate medical education: a systematic review. Medical Teacher 323(5):455-61
Enock K. 2000. An Evaluation of Computer Aided Learning within Evidence-Based Health Care educational resources, MSc diss., University of Oxford, England.
Garg A., Turtle K.M. 2003. Effectiveness of training health professionals in literature search skills using electronic health databases--a critical appraisal. Health Information & Libraries Journal 20(1):33-41.
Green M.L. 1999. Graduate medical education training in clinical epidemiology, critical appraisal and evidence-based medicine: a critical review of curricula. Academic Medicine 74:
686-94
Greenhalgh T. 2001. Computer assisted learning in undergraduate medical education. British Medical Journal 322: 40-44
Hatala R., Guyatt G. 2000. Evaluating the teaching of evidence-based medicine. Journal of the American Medical Association 288(9):1110-11
Hutchinson L. 1999. Evaluating and researching the effectiveness of educational interventions. British Medical Journal 318:1267-9
Hyde C., Parkes J., Deeks J., Milne R. 2000. Systematic review of effectiveness of teaching critical appraisal. ICRF/NHS Centre for Statistics in Medicine, Oxford, England
Miles B.M., Huberman A.M. 1994. Qualitative data analysis: an expanded sourcebook (2nd edition). Thousand Oaks, CA: Sage
Lycke K.H., Stromsø H.I., Grøttum P. 2002. PBL goes ICT: problem-based learning in face- to-face and distributed groups in medical education at the University of Oslo. Faculty of education report no. 4, University of Oslo, Norway.
Norman G.R., Shannon S.I. 1998. Effectiveness of instruction in critical appraisal (evidence- based medicine): a critical appraisal. Journal of the Canadian Medical Association 158:177- 81
Parkes J, Hyde C, Deeks J, Milne R. 2003. Teaching critical appraisal skills in health care settings (Cochrane Review). In: The Cochrane Library Issue 3. Oxford, England: Update Software.
Richardson D., Birge B. 1995. Teaching physiology by combined passive (pedagogical) and active (andragogical methods). Advances in physiology education 13(1):66-74
Sackett D.L., Rosenberg W.M.C., Gray J.A.M., Haynes R.B., Richardson W.S. 1996.
Evidence based medicine: what it is and what it isn’t. British Medical Journal 312:71-2 Speller V., Learmonth A., Harrison D. 1997. The search for evidence of effective health promotion. British Medical Journal 315:361-363
Stacey R., Spencer J. 2000. Assessing the evidence in qualitative medical education research.
Medical Education 34(7):498-500
Taylor R., Reeves B., Ewings P., Binns S., Keast J., Mears R. 2000. A systematic review of the effectiveness of critical appraisal skills training for clinicians. Medical Education 34:120- 125
Taylor R., Reeves B., Ewings P., Mears R., Binns S., Keast J., Hartland J. 1999. A randomised controlled trial of the effectiveness of critical appraisal skills workshops for health service decision-makers in the South and Western Region. Unpublished report. Oxford, England.
Wolff N. 2001. Randomised trials of socially complex interventions: promise or peril?
Journal of Health Services Research and Policy 2:123-6
Figure captions
Figure 1a and 1b: Final conceptual frameworks Figure 2: Syllabus for the 10th semester
Figure 3: Syllabus for the EBM course Figure 4: The timing of data collection
Table 1a, 1b and 1c: Classification of interventions according to educational theory Table 2: Results from the satisfaction questionnaire
Figures and Tables
STUDENTS (as a group) - Ability to engage in group activity
- Peer pressure STUDENT (as individual)
- Learning style - Gender
- Language and PC skills - Knowledge level - Self-efficacy (to develop goals, interact, choose material)
- Satisfaction with learning
- Perceived usefulness
LEARNING RESOURCES - Flexibility: timing, cite of learning
- Language -Organisation
ONGOING LEARNING
- Interaction: Student- student, student-tutor, student-media - Superficial or deep learning
- Rate of learning TUTOR
- Ability to act as facilitator for learning - Motivation
Self-directed group*
*Participant observation has been used to inform these frameworks, which express the presumed interaction of groups of factors influencing students’ learning. The magnitude of interaction is expressed by the thickness of each arrow shaft.
Where factors are seen to affect one another mutually, a double-headed arrow is used. A relationship with is predominantly one-sided, is expressed by a single-headed arrow.
STUDENTS (as a group) - Ability to engage in group activity
- Peer pressure
STUDENT (as an individual) - Learning style - Gender
- Language and PC skills - Knowledge level
- Self-efficacy (to develop goals, interact, choose material)
- Satisfaction with learning - Perceived usefulness
LEARNING RESOURCES - Timing, cite of learning - Organisation
- Language
ONGOING LEARNING
- Interaction: Student- student, student-tutor, student-media - Superficial or deep learning
- Rate of learning TUTOR
- Teaching style (interactive v didactic) - Ability to act as facilitator for learning - Motivation
Directed learning group*
Week 1 and 2: Two weeks of introductory lectures (16 half-day sessions) at the university.
Week 3 to 14: Twelve weeks of clinical placements split equally between a local hospital and general practice.
Week 15 to 19: Five weeks of lectures at the university.
Week 20: One week of compulsory written examinations.
PBL assignments, where students meet on two occasions per assignment.
Although attendance at PBL groups is compulsory, the assignments are not formally evaluated
An introduction to evidence-based medicine and statistics (one workshop/one module)*
Searching for evidence (two workshops/one module)*
Randomised controlled trials (one workshop/two modules)*
Systematic reviews (one workshop/one module)*.
*the amount of time/number of CD-rom modules per topic is given in brackets:
First cohort (autumn 2001) Second cohort (spring 2002) Third cohort (autumn 2002)
Teaching (week 1 and 2) Teaching (week 1 and 2) Teaching (week 1 and 2) Participant observation
(all students attending teaching) PO1* and PO2* (weeks 1 and 2) Interviews
(selected students)
1** to I6** (weeks 2) 17 to I10** (weeks 2 and 15) Group discussion
(selected students)
GD1+ and GD2+ (week 2) Satisfaction survey
(all students in cohort)
Data collection from satisfaction survey
*
PO = participation observation, **I =interviews, + GD = group discussions
RCT
Androgogic theory (principles)
Directed learning group Self-directed learning group
Establish an effective learning climate, where learners feel safe and comfortable expressing themselves
Explicit aim to establish safe, supportive atmosphere in lectures, small group work and discussion in plenum. Discussion with tutors and between students encouraged throughout.
Explicit aim to establish safe, supportive atmosphere in with tutor as facilitator.
Discussion with tutors and between students encouraged throughout.
Involve learners in mutual planning of relevant methods and curricular content
Limited student involvement. Syllabus and methods for learning were pre-determined.
Limited student involvement. Syllabus and methods for learning were pre-determined, but a range of options were given.
Involve learners in diagnosing their own needs
Students not formally encouraged to diagnose learning needs.
Students formally encouraged to diagnose learning needs (day 1 of teaching week).
Students given a copy of the curriculum as reference point.
Encourage learners to formulate their own learning plans
Students not formally encouraged to formulate own learning objectives.
.Learning objectives stated in introduction to each lecture.
Students formally encouraged to formulate learning objectives (day 1 of teaching week) using the syllabus at reference point.
Encourage learners to identify resources and devise strategies for using the resources to achieve their objectives
Learning resources and strategies pre- determined.
Students were encouraged to control the time, pace and cite of their learning and to choose between a variety of learning methods: interactive CD-rom, workbook, internet resources, background literature and tutor-support. Available CAL modules were graded according to difficulty and possessed an automatic feedback facility.
Support learners in carrying out their learning plans
Sessions run according to pre-determined structure, designed to support learners generally.
Students encouraged to use tutor facilitators as main source of support in carrying out learning plans.
Involved learners in evaluating their own learning
Indirect evaluation of learning during small group and plenum discussions. Students not involved in setting examinations.
Direct evaluation using feedback facility in CD-rom. Students not involved in setting examinations.
Social learning theory Directed learning group Self-directed learning group
Learning through observation (modelling).
Tutor acts as role model during introductory talk and feedback session.
No specific role model to observe.
Examples in CD-rom with feedback only.
Clear objectives must be set Syllabus set and learning objectives stated at the beginning of lectures.
Learning objectives for each session were not set, but decided by students who were guided in their choices by a pre-determined syllabus.
Relevant task knowledge must be given
Clear instructions given for each part of the teaching session by tutor.
Specific task knowledge is not given, but decided by students for each session.
Guided practice of new skills with feedback
Pre-prepared checklists, practice papers and scenarios form basis for small group and feedback sessions. .
Pre-prepared checklists and scenarios form basis for learning. CD-rom has automatic feedback facility.
Opportunities to reflect Opportunities to reflect on learning in the small group and feedback sessions.
Students were given a post-workshop pack.
Opportunities to reflect on learning dependent on students’ motivation to repeat complete further exercises in workbook and talk to fellow students.
Self-directed learning Directed learning group Self-directed learning group
Achievement thro self- actualisation (personal responsibility)
No focus on learning through personal responsibility. Syllabus, course structure and tasks all pre-set.
Students encouraged to control time, pace, cite of learning and choice of resources including opportunities for validation, confirmation within set syllabus Organisation to ensure
learning tasks in students’
control
Syllabus, course structure and tasks all pre- set. Students were encouraged to be interactive and encourage small group learning process
Students control time, pace, cite of learning and choice of resources incl. opportunities for validation, confirmation within set syllabus
Assessment takes account of how much has been learned, not only what.
Final examinations assessed skills, knowledge and attitudes.
Final examinations assessed skills, knowledge and attitudes.
Student input to developing evaluation criteria
No student involvement No student involvement
Encourage learner to ask questions, critically appraise new information, identify own knowledge gaps, and critically reflecting on their own learning process and outcomes.
Limited opportunity. See criteria above (andragogy)
Criteria largely met. See criteria above (andragogy)
