Professor
Harry Tolley, External Evaluator, RAPID 2000 Project
1. The RAPID
(Recording Academic, Professional and Individual Development) Progress File (http://rapid.lboro.ac.uk/) is web-based,
and is intended for use in connection with personal and professional
development, mainly in the context of Higher Education (HE). As such, it
enables registered users to input and maintain information on a
password-protected database. It has been developed by Loughborough University
as part of the DfEE (Department for Education and Employment) funded project ‘Recording
Achievement in Construction’ (1998-2000), and has been widely implemented
through the HEFCE (Higher Education Funding Council for England) funded
project, ‘RAPID 2000’ (2000-2003). The RAPID Progress File has been designed to
enable students (within Construction and Civil Engineering disciplines) to
maintain a record of their achievements and to audit and develop their skills
in ways, which are compatible with the competence requirements of the relevant
Professional Institutions. Currently, eight customised versions of the RAPID
Progress File are in use in the UK; and further versions are being developed
under licence in Australia.
2. This report
sets out the principles on which the evaluation of the RAPID 2000 Project was
based, and provides an account of the methods used in the conduct of that
evaluation. The rest of the report is devoted to an examination of the
students’ experiences of using RAPID, and the insights in respect of the
implementation of personal development planning (PDP), which can be derived
through the adoption of this perspective. The aim of the exercise is
‘formative’ and ‘proactive’ rather than ‘summative’ and ‘retroactive’ (Nevo,
1989) – the purpose being to further the ongoing development of progress files
and their use in HE and beyond.
3. The principles underpinning the
evaluation of the RAPID 2000 Project were derived from Kemmis (1989) who
advanced the view that the role of evaluation should be seen as feeding
information into critical debates amongst the participants (or ‘stakeholders’)
in a curriculum development project – especially debates at which important
decisions are made. In practical terms this meant that evaluation was fully
integrated into all of the project’s developmental activities, and as such
featured prominently in the project’s Strategic and Operational Plans, was
discussed regularly at meetings of the Management Committee and Steering Group,
and was conducted systematically throughout the life of the project. The
guidance on project evaluation provided in 1998 by the Higher Education and
Employment Division (HEED) of the then Department for Education and Employment
(DfEE) was helpful in translating the underlying theoretical principles into
practice.
4. Within what was essentially an action
research paradigm, evaluation data were collected and analysed using both
quantitative and qualitative methods as appropriate – fitness for purpose being
the key selection criterion. The outcomes of these evaluation activities were
then used formatively by the project as an integral part of its decision-making
process, as well as for the purposes of accountability to its stakeholders,
including the funding body. The evaluation, therefore, was ‘responsive’ (Stake,
1975 and Guba and Lincoln, 1981) to the different concerns of multiple
audiences for relevant information about the progress of the project.
5. An External Evaluator monitored the
implementation of the evaluation strategy, advised on the methods and
procedures used in the collection and analysis of data, participated in the
ensuing discussions, and provided independent judgments on the outcomes of the
evaluation and their significance. Nevertheless, evaluation was
seen as the responsibility of all of the key participants in the project, who
acted throughout as a ‘self-critical community of practice’. In particular, the
Project Manager played a major role in this process – not just in terms of
planning, co-ordination and management, but also through his involvement, where
it was appropriate for him to do so, in the collection and analysis of
evaluation data.
6. Within this framework the ‘social distance’, which the External Evaluator negotiated between himself and the other active participants in the project, was of crucial importance. On the one hand, the External Evaluator needed to become close enough to them in order to understand the project’s evaluation needs, and to develop a rapport with those involved in its implementation. On the other hand, it was important to be sufficiently distanced from its day-to-day activities in order to be able to view them externally, and in so doing see them in a broader perspective. The term ‘critical friend’ has been widely used to describe this kind of relationship. It is one, which enabled the External Evaluator to ensure that the participants remained self-critical over the lifetime of the project, thus avoiding the pitfall of becoming complacent, or even self-congratulatory, in the face of their own success.
Methodology
7. A multiplicity of methods was used in
the conduct of the evaluation including, interviews with academic staff,
student questionnaire surveys, student focus groups, institutional case studies
(self-report) and an experts’ forum – the aim being to assemble valid and
reliable evidence using techniques, which were methodologically sound.
8. Interviews with university staff
involved in the implementation of RAPID were conducted at four of the
participating HE institutions. They were undertaken on a one-to-one basis using
a common, semi-structured interview schedule. A series of predetermined
‘prompts’ were deployed to stimulate discussion, and supplementary questions
(or ‘probes’) were used to encourage the respondents to clarify or develop
their answers (Drever, 1995). Notes were taken during the interviews, and these
were used to produce reports, the aim of which was to accurately represent what
the respondents had said, rather than to produce transcripts of the
interactions in their totality. To that end those interviewed were asked to
verify the accuracy of what they were reported to have said and to offer any
afterthoughts they might have had. A protocol, which addressed such issues as
confidentiality and anonymity, was discussed and agreed at the beginning of each
interview.
9. A sample of students at each of the HE
institutions, which had adopted RAPID, was surveyed by means of questionnaires
at three points in the course of its implementation. An initial (Benchmark)
survey was conducted prior to the students being introduced to RAPID. A further
(Monitoring) survey was undertaken in the middle of the implementation period,
and a (Final) survey was carried out at the end. The questionnaires were
devised by the Project Manager in consultation with the External Evaluator, and
forwarded to each HE Institution where they were administered by the staff
concerned. The completed questionnaires were then to the project where they
were analysed by the Project Manager in consultation with the External
Evaluator.
10. Focus
groups (Kreugar, 1994 and Morgan, 1988) were conducted with students at four of
the participating HE institutions - the same as those in which the staff were
interviewed. A total of sixty students attended the focus groups, their numbers
being evenly distributed across the four groups. On two occasions staff
representatives were present as observers – their participation in this
capacity having been negotiated with the students in advance. The aim of each
focus group was to gather evidence of the students’ experiences of using the
RAPID Progress File. In each case the External Evaluator acted as the
‘facilitator’ in order to ensure that: all of the agenda items (common across
all of the groups) were discussed systematically; everyone had an opportunity
to speak; and, the agreed ‘ground rules’ were adhered to by all those present,
including the staff observers. Each session lasted approximately 90 minutes. A
research assistant acting as a ‘scribe’ made notes on what was said using a
pre-prepared template, but took no other part in the proceedings. Ethical
issues relating to the use of the data were discussed, and agreed with each
group at the outset. After each focus group, the scribe produced a
word-processed record of the discussion. Finally, the External Evaluator, in
consultation with the scribe, produced a report of each of the four groups.
11. At the end of academic years 2001-2002,
and 2002-2003 each of the participating institutions was required to produce, a
case study report on the implementation of the RAPID Progress File. A template
was provided to guide each institution in the collection of the relevant data,
and on the structure and content of their reports.
12. A summary of the data arising from the
2001-2002 evaluation reports, and the data collected in 2002-2003 using the
methods described above was distributed to a group of people with relevant
expertise who had agreed to participate in an ‘experts’ forum’, and in so doing
to assist in the process of reviewing the data, and making sense of the
outcomes in terms of their significance for future policy and practice.
13. The methods
described above, which were used for the purposes of the evaluation, have
proved to be important sources of information about the use by staff and
students of the RAPID Progress File and the processes of teaching and learning
within which that use was embedded. However, the accounts of professionals and
students of their behaviour and actions, which the evaluation data represent,
should be viewed with a degree of caution. This is because research shows that
there is often a gap between what people say they do, and what they actually do
(Gilbert and Mulkay, 1984). Moreover, research also shows that people are often
unable to describe in detail many of the practices they use and rely upon in
accomplishing both individual and collaborative tasks (Greatbatch et al, 2001).
In normal circumstances these practices are tacit, taken-for-granted and
seen-but-unnoticed. Consequently, they are rarely discussed or, in some cases, even
thought about (Garfinkel, 1967). This raises the possibility that evaluation
studies of the processes of teaching and learning, which rely predominantly on
data derived from interviews, questionnaires and focus groups, as this one has
done, may overlook important aspects of the ways in which students
actually behave in learning situations, including how they interact with each
other, their tutors, learning resources and with electronic tools such as the
RAPID Progress File.
14. There is a need, therefore, to complement
the types of evaluation evidence collected by the project with ethnographic
accounts based upon participant and non-participant observation (Walken, 1989).
It has been argued that such studies will enhance our knowledge and
understanding of the students’ experiences of Higher Education (Murphy and
Scott, 2003). To that end, observational methods have already been incorporated
into the operational plan for the Loughborough University DART (Disabilities:
Academic Resource Tool) Project -(http://dart.lboro.ac.uk)
– a HEFCE funded project, which seeks to build upon the experiences of RAPID
2000.
15. Despite the words of caution voiced above, the data collected for the purposes of the evaluation proved to be a rich source of evidence of the students’ experiences of using the RAPID Progress File, and the processes of teaching and learning and PDP within which its use was rooted. It is argued that, by viewing the evaluation data through the lens of this perspective, lessons and insights can be derived which will be of value to a range of audiences with interests in the development of such tools and their use in furthering PDP in HE and beyond.
16. It is important to recognise, however, that the students whose experiences are being discussed here were by no means homogeneous groups of ‘traditional’ or ‘standard’ students (i.e. male, white, middle class and able bodied) - a construct, which Leathwood and O’Connell (2002) have argued still persists within political and policy based discourses on HE. In fact the students were highly heterogeneous – their diversity being evidenced by the numbers who were mature, had entered HE with non-standard qualifications (some by means of access routes), were working class and were from minority ethnic groups. Similarly, the students were diverse in the HE institutions they were attending, the courses they were pursuing, the qualifications they were seeking and whether they were studying full-time or part-time. However, persistence of a gender bias in the subjects they were studying did mean that overall more males than females were present at the focus groups – despite the best efforts of the institutions involved to counteract such disparities in the students they recruit.
17. The data collected for the purposes of this evaluation, are insufficiently fine-grained to facilitate a detailed enquiry into the variations, which exist in the experiences of different categories of students, and the institutional settings in which they used the RAPID Progress File. Nevertheless, the information, that is available, does provide a rich source of evidence about the experiences of students in a variety of contexts in HE (including a work placement year in the case of one group) from which some important generic lessons can be derived.
The students’ experiences
18. Few of the students questioned had used RAPID for a long period of time (e.g. over the duration of their undergraduate courses) making it difficult for them to view their experiences from ‘a distance’, and with the benefit of sustained reflection. Consequently, much of the feedback they provided was of a more immediate, short-term nature (e.g. the introduction of the Progress File and their induction into its use). Understandably, many remained to be convinced about the benefits they might be expected to gain through using RAPID over a longer timescale as an integral part of their course as a whole, and through a prolonged engagement in PDP and its related support structures and processes. However, there were notable exceptions as evidenced by one of the students who said ‘RAPID helps to develop a longer view of your learning, asking you to look ahead and not just focus on the immediate’.
19. Many of the participating students (i.e. the younger ones) had brought prior experiences of using the National Record of Achievement (ROA) with them into HE. For example, they reported that they had invested time and effort into producing portfolios, which no one but themselves appeared to value – especially university admissions tutors. As one student put it: ‘You make comparisons between RAPID and your ROA (Record of Achievement) and remember how nobody ever looked at it’. Understandably, such experiences had left the majority of the students questioned with negative attitudes towards the very idea of maintaining a Progress File, and with low expectations of the benefits they might derive from their involvement in PDP – thoughts and feelings which need to be recognised and counteracted if they, and students like them, are to capitalise on the learning opportunities using tools such as RAPID can provide.
21. A further observation offered by the students on their experiences of the induction process was that it left them with feelings of uncertainty as to the reasons why they were expected to make use of RAPID. As one student put it: ‘At present when you’re using it you’re not always aware as to why you are required to do so’. A second student advanced a similar view: ‘The advantages of using RAPID were not made clear to us – no one sold it to us’. In other words, the students were asking their tutors to make explicit (rather than leaving implicit) the intended learning outcomes, and the benefits (both short term and longer term) they might expect to accrue from achieving those goals. Armed with such knowledge, they would then be in a much better position to make strategic decisions about where to place maintaining a Progress File in their personal order of priorities. In the complex process of juggling the time and effort they devote to study, part-time work, their social lives, and in some cases family commitments, they need to be convinced that there is a chance that they will get a ‘return on their investment’ (see paragraph 24). In keeping with such a utilitarian approach to their learning (Smith and Spurling, 2001), the students advocated: inputs to the induction process from former students who have already had a positive experience of using tools such as RAPID; and, access to the literature on continuing professional development (CPD) from the relevant professional bodies – especially if the latter stress the importance of maintaining a Progress File or portfolio for the purposes of accreditation. One student summed up the last point as follows: ‘If using RAPID was linked to achieving chartered status, and those links were reinforced, then you would definitely be motivated to use it’.
22. The students’ accounts of using RAPID tell some very different stories about their experiences, which highlight the importance of attending to the precursors of delivery i.e. to advanced planning and preparation. For example, some students spoke positively about their experience of being able to: access their Progress File remotely using their user names and passwords (‘instead of having to travel 5 miles in to the university to work on it’); navigate between and within the different sections of the tool (PACE and SPEED); audit their skills using the drop down menus; print off parts of their files in order to discuss them with someone; and even plan future learning opportunities. In other words they were able to use the instrument in the way in which it was intended. However, there were others who reported that they had experienced a number of technical difficulties (most arising from their own institutional ‘intranet’ set-up or access rights to printers), which had prevented them from e.g. accessing their files, printing off hard copies, and archiving data they had added. In most cases the difficulties had eventually been resolved, but not before frustration and de-motivation had begun to set in amongst the students. One student summed it up, as follows ‘If technical problems persist you will only use RAPID if and when you have to’. Undoubtedly from the point of view of students such as this, ‘prevention’ would have been a far more effective strategy than ‘cure’.
23. The difficulties and frustrations experienced by those students who felt ill-prepared for using RAPID by their induction, and who did not find the in-built guidance within it easy to follow, were exacerbated when they were unable to find someone to help and advise them. Good as their student support networks were, there were times and places (e.g. late in the evening in the university library or out on a work experience placement) when this was not to hand and they were unable to resolve their difficulties by themselves. What these students seemed to expect when they ran into difficulties (apart from self help and from their peers) was access to various forms of assistance in the form of academic and ancillary staff who are not only familiar with the PDP tool the students are using, but are part of a wider support system. Those systems are in place to support the use of the library and ICT, the students argue, so ‘Why aren’t they available for RAPID and PDP?’ Clearly, if this aspiration is to be met when all students are expected to maintain a Progress File, then much capacity building will need to take place, including staff training and development and the provision of helpdesk facilities.
24. The students’
narratives about their experiences of using RAPID in the context of their HE
courses provide some interesting insights into their motivation and the coping
strategies they adopt to manage their learning (see paragraph 21). From the point of view of the former, it was evident
that their tutors’ decision to integrate the use of the Progress File into the
assessment system (e.g. as part of an assignment) motivated them in the
short-term. ‘I’ll go the extra mile just
to get the marks I need’ was how one student put the ‘stick and the carrot’ effect of structuring the use of RAPID into
an assessment task, which he had to complete in order to pass a particular
module. However, assessment was not the only means of ensuring that the
students engaged positively with their Progress Files. Students whose tutors or
workplace supervisors routinely included discussion about PDP in their sessions
said that this prompted them to take it every bit as seriously as if it was an
assessed piece of work. Conversely, those tutors and work placement line
managers who said that RAPID was important, but then proceeded to say or do
nothing to reinforce that, signalled to the students that they need not take
PDP as seriously as other aspects of their course. It would appear from what
was said that the implicit curriculum is every bit as important as the explicit
(if not more so), and that students being students think and act strategically
in ordering their priorities.
25. Paradoxically, whilst extrinsic motivation can be effective in the short term, it is unlikely to do much to further ‘deep learning’ as opposed to ‘shallow learning’ (Entwistle and Ramsden, 1983). Similarly, it does not address the question of their motivation in the longer term – the shift to intrinsic motivation, which is needed if students are to engage in PDP as a means of reflecting on their learning experiences, and identifying what they need to learn in the future. It is now widely accepted that, the long term aim of producing graduates who are capable of managing and improving their own learning, benefiting from opportunities for CPD and becoming effective lifelong learners depends upon students making such a shift. Happily, there were students at the focus groups who signalled that given the right incentives and support they were capable of making that transition. For example, one student expressed the view that ‘RAPID is a valuable tool for practising self-evaluation, which is important in personal development’, whilst another said of using the Progress File ‘It highlights your weaknesses and helps you realise what you have to do to improve’.
26. However, if students are to make such a
shift, the use of tools such as RAPID for the purposes of PDP will need to be
fully integrated into the curricula of their HE courses – something which a
number of students advocated e.g. 'RAPID should be integrated into the curriculum and
students should be made more aware of its importance' and ‘if RAPID is such a valuable tool, then why
is it just used within one module? Why not use it all the time?’ In
thinking through how to bring about such a transformation those responsible may
find it helpful in formulating their plans to draw upon the theory of
constructive alignment (Biggs, 1999). What is useful about this theory in the
current context is that it seeks to connect the idea of intended learning outcomes
to the things tutors and others actually do to help students to learn, and the
things, which students actually do and learn. The theory starts with the notion
that learners construct their own learning through relevant activities - what
students do being more important than what their tutors do. The tutor’s job,
therefore, is to create an environment, which supports the learning activities
appropriate to achieving the planned outcomes. The key is that all components
in the teaching-learning system – the intended learning outcomes, the course
design, the methods used to deliver the curriculum, the resources used to
support student learning, and the tasks and criteria for assessing that
learning – are aligned to each other
so as to facilitate the achievement of the intended outcomes.
27. The adoption of such an
approach to the design and delivery of courses which incorporate Progress Files
and PDP will require a substantial change in both thinking and practice
(Trowler, Saunders and Knight, 2003) – in fact nothing less than a culture
change affecting teaching and learning in HE. That it the challenge, which now
needs to be addressed.
Conclusions and recommendations
28. The principles underpinning the evaluation
of the RAPID 2000 Project have been described, and an account has been given of
the methods used in the collection and analysis of the evaluation data.
29. Students’ experiences in a diversity of
contexts have been used as a lens through which to consider the lessons, which
can be learned from the pilot studies of the implementation of RAPID. In so
doing, it has been argued that those with an interest in developing policy and
practice with regard to the use of Progress Files and PDP in HE can derive many
valuable insights from listening to the ‘authentic voice’ of students.
30. Future evaluation activities would benefit
from evidence derived from longitudinal studies, which track the students’
experiences over a longer timescale than the pilot studies considered in this
report. Such studies should seek to include ethnographic accounts based upon
participant and non-participant observation as well as evidence obtained by
means of questionnaires, interviews and focus groups.
31. As a result of their prior experiences,
many of the students questioned for the purposes of the evaluation, displayed negative attitudes towards the very idea of maintaining a
Progress File, and had low expectations of the advantages they might derive
from PDP. This way of thinking needs to be recognised and challenged if students
are to benefit from the learning opportunities offered by incorporating tools
such as RAPID into their HE courses.
33. In addition, the students asked that the induction process should make explicit the intended learning outcomes and emphasize the advantages (both short term and longer term), that the students might be expected to gain through using a Progress File such as RAPID for the purposes of PDP as an integral part of their courses.
34. Equipped
with such information, students are able to make strategic choices about where
to place the Progress File in their order of priorities i.e. it helps them to
manage conflicting demands on their time more effectively. In order to ‘commit’
themselves to it, they need to be convinced that there is a chance that they
will get a ‘return on their investment’.
35. The
students’ personal accounts of using an electronic Progress File such as RAPID
highlight the importance of university-wide advanced planning and preparation –
especially with regard to anticipating and preventing the technical
difficulties which can both frustrate and de-motivate students. From the
students’ perspective prevention of problems is a far more effective strategy
than addressing difficulties after they have occurred.
36. When
students encounter problems when using RAPID for the purposes of PDP their
expectation is that they will have access to more than self-help and peer
support i.e. to assistance from academic and ancillary staff who are familiar
with it and are part of a wider learning support system. If such systems are in
place to support their use of library and ICT services, students argue, that
they should be available for PDP. Clearly, if all students are to be required
to maintain a Progress File, such support systems will need to be developed,
and staff training needs will need to be identified and addressed.
37. The inclusion of work, which made use of
a Progress File, within the assessment system proved to be a powerful source of
extrinsic motivation. However, the routine
discussion of such work within tutorial sessions was said to prompt students to
take it every bit as seriously as if it were to be assessed.
38. Paradoxically,
whilst extrinsic motivation can be effective in the short term, it fails to
address the question of student motivation in the longer term. What is needed
is a shift to intrinsic motivation, which students need if they are to
routinely engage at depth in PDP as a means of reflecting on their learning,
and identifying what they need to learn in the future. The long-term goal of
producing graduates who can manage their own learning, benefit from CPD and
become effective lifelong learners requires students to make such a shift.
39. If
students are to make that transition, Progress Files and PDP will need to be
fully integrated into the curriculum. In thinking through how to bring about
such a change it may be helpful to draw upon the theory of constructive
alignment (Biggs, op cit). However, the widespread adoption of such an approach
would require a substantial change in both thinking and practice – a culture
change in HE no less.
Biggs, JB (1999) Teaching for Quality Learning at
University. Buckingham: Society for Research in Higher Education and Open
University Press
Drever, E (1995) Using Semi-Structured Interviews in
Small-Scale Research, Glasgow, The Scottish Council for Research in Education
Entwistle, NJ and Ramsden, P (1983) Understanding Student
Learning, London, Croom Helm
Garfinkel, H (1967) Studies in Ethnomethodology. Cambridge:
Polity Press
Gilbert, GN and Mulkay, M (1984). Opening Pandora’s Box: An
Analysis of Scientists’ Discourse. Cambridge: Cambridge University Press
Guba, EG and Lincoln, YS (1981) effective Evaluation, San
Francisco, Jossey-Bass
HEED (1998) Evaluating Development in Higher Education: A
Guide for Contractors and Project Staff, Higher Education Employment Division,
Department for Education and Employment, Moorfoot, Sheffield, DfEE, HEED
Kreugar, RA (1994) Focus Groups, Sage, London
Leathwood, C and
O’Connell, P (2002), It’s a struggle: The Construction of the ’New Student’ in
Higher Education. Paper Presented to the SRHE Annual Conference. 10-12
December, University of Glasgow, UK
Morgan, DL (1988) Focus Groups as Qualitative
Research, Sage, London
Murphy, RJL and
Scott, R (2003) The search for a fuller understanding of students’ experiences
of Higher Education, Paper delivered at SRHE/ESRC Symposium in Glasgow, March
2003.
Nevo, D (1989) The Conceptualization of Educational
Evaluation: An Analytical Review of the Literature, pp 15-29 in New
Directions in Educational Evaluation, Edited by House, ER. Lewes: The Falmer
Press
Smith, J and
Spurling, A (2001) Understanding Motivation for Lifelong Learning, Devon:
Campaign for Learning and The National Organisation for Adult Learning (NIACE)
Stake, RE (1975) Evaluating the Arts in Education: A
Responsive Approach, Teachers
College Record. 68. 523-40
Trowler, P, Saunders, M and Knight, P (2003) Change
Thinking, Change
Practices. York: The Learning and Teaching Support Network
Generic Centre
Walken, R (1989) Three Good reasons for Not Doing Case
Studies in Curriculum
Research, pp 103-116 in New Directions in Educational
Evaluation, Edited by House,
ER. Lewes: The
Falmer Press