There have been many lessons learned during the project – and we would like to share these openly in the hope that others can shortcut our learning.
We did not appreciate the gap in functionality that existed between what we had developed in the original STAR-Trak project and what would be required in order to satisfy staff and student users of the system. We had received enthusiastic feedback on the original application, and from this we concluded that only minor modifications would be required in order to provide a functional application that could be piloted. However once we started to run some workshops to explore that gap we realised that:
– our domain knowledge was insufficient,
– the metaphor we were using to identify students at risk was not best-suited to the task,
– the focus of the application was wrong
– we needed to put the student in control of the viewing of their data to maintain appropriate data privacy,
– we need to work on the ease of use and intuitiveness of the application
Internal domain knowledge: we assumed that the business (including IT) understood its data. However, while all parties did understand their data, they all had a more or less different understanding! As we are trying to develop an application that will be useful across the sector, we also had to make intelligent guesses about how other HEIs might in practice construct their ontologies. It was outside of the project scope to investigate this formally, however informal contacts were useful in this area.
Metaphor: one of the key objectives of the project is to “identify students at risk of dropping out”. We unquestioningly developed this notion into a traffic light metaphor, whereby the system calculated a red, amber or green result for each student based on their activity levels. While we were not unduly concerned about inbuilt cultural bias in this (traffic lights are common around the world and it is a common enough metaphor), we came to understand that it might be alarming and de-motivating for students, and, more importantly that it’s used crossed the divide between data analysis and judgement-making: whether the result indicates a higher risk or not is not for a computer, but for the student and their tutors, to decide on. We have changed the metaphor to “high, medium and low” engagement and identified the necessity for a flexible, data-driven metaphor as a future enhancement.
Focus of the application: Our early understanding of STAR-Trak was that it was primarily a tool for staff to identify students at risk of dropping out. As our ideas matured we came to realise that STAR-Trak was better seen as a place that facilitated a discussion between student and staff on a wide range of issues that impacted on their progress (still with a primary aim of improving retention). This idea developed still further to put the student at the heart of the application, aligned with and supporting our pedagogical aims of assisting students to become independent and reflective learners.
Putting student in control: We came to realise that, for the application to be successful, students had to be comfortable with the idea of sharing their data with staff. The easiest way to achieve this was to give them the ability to hide data from all or particular staff (or from the opposite perspective, to grant viewing rights to all or certain staff). In addition, students can choose to participate in STAR-Trak or choose not to. We hope that these decisions will help them reflect on what it is they are seeking to achieve and how they can do that through their various learning activities.
Ease of use and intuitiveness: students and staff already face a multitude of systems to learn and use in addition to the daily tasks of learning and supporting. If STAR-Trak is to have any chance of being successful it has to be intuitive to use and ideally have added value such as reducing access to other systems and synthesising information. Running through scenarios in workshops it became clear that we had more work to do than we had originally anticipated in this area. Almost certainly the pilot that we hope to run over this coming academic year (2011/12) will identify further work in this area.
STAR-Trak is an innovative application for Leeds Metropolitan. It is therefore not surprising that our understanding of how it can be best used will be emergent over the life of this project and during its early years of use, rather than fully understood up front. That the project has given us the time to undertake these reflections is absolutely crucial to the long-term sustainable success of STAR-Trak – it would have been pointless implementing an application that nobody used. The “culture” of the JISC call has been to take risks and experiment. By doing so we have learned a number of valuable lessons that will greatly increase the chances of success for STAR-Trak.