On Monday and Tuesday next week our 2nd years will have their practical exam. This consists of 2 x 3 hour sessions, one in a computer lab carrying out data analysis and the other is in the lab carrying out a literature style prep. All of the techniques and methods should be familiar to the students but the specific scenarios are not. This is worth 20% of a 15-credit module and we’ve run mock exams for each to give our cohorts an idea of what they have to do. I run the lab exam which is an eye-opening experience because do not supervise other 2nd year labs. It’s also a good perspective to have for this exam because I don’t have particular expectations of what the students do. The lab component is split 50/50 – in lab performance (including ‘did they make the product’) and lab diary writeup.
The general format for the lab exam is fairly obvious: the students are given a literature procedure and have to complete a COSHH form (generally a gapped form where they fill in certain bits), make the product, carry out some analysis and perhaps do some data interpretation. At the start of the year (September) we give out a practical exam check list that allows them to self-evaluate their skills on a list of techniques and they are asked to ensure that they get to a certain level for each skill over the year. Many labs are done individually in 2nd year but they are encouraged not to ‘hide’ behind their lab partners and ensure their skills are appropriate. The techniques required to complete the practical exam are generally late 1st year in level which I feel is appropriate for the added pressure of exam conditions.
One of the biggest challenges I have is how to develop a marking scheme that can be completed during the session by four different invigilators. Many aspects of lab performance (and safety) are fairly subjective. This year I’ve gone for a ‘subtractive rubric’ where every student walks into the lab with 50 points. Invigilators will note down interventions they make (put those safety glasses on!), questions asked and other elements of performance and take points off on four specific scales: safety, tidy, independent, and effective. We do not take marks away for safety questions because we always want to encourage a culture where students ask if they are uncertain about safety. We also recommend that students ask questions because the marks gained from getting the procedure correct generally outweigh the marks lost for asking sensible questions. We do still enforce the lab rules and if a student has to be reminded to follow a lab rule on three occasions, they are asked to leave. This would be things like eating, drinking, not wearing safety specs, flinging the chemicals around… I mark all of the lab diary sections and total (and moderate) the in-lab rubrics. This year I’m also double marking 6 – 8 students, 2 per invigilator to look at consistency between invigilators.
A serious question though is what constitutes a pass for a lab-based practical exam. Specifically, is it necessary to successfully make the intended product to pass. On one hand that seems like a silly question – how can you pass an exam but not make the product? On the other, plenty of very minor things can go wrong under the pressure of exam conditions that can lead to not being able to isolate the product. We also have to consider that the pass mark is 40% and that our ILOs are supposed to indicate achievement at that level. It adds considerably to the pressure if passing is dependent on submitting a vial of stuff at the end of the session so the mark scheme has to account for these minor errors. I’m thinking about things like a flask tipping over or adding too much solvent, simple accidents that happen to us all. Our mark scheme allows for this and still passing the exam. Things that have safety implications such as adding the wrong reagent because of misreading the label are more problematic and should be marked down more. And if the in-lab bit goes totally wrong, the lab diary entry can still get a decent mark (particularly when good explanations of what went wrong are included).
Practical exams like this take up a lot of staff time to prepare and invigilate but provide a direct measure of practical ability. It makes a real difference to how you think about students’ ability in the lab when you watch them take a prep and go about doing it. And I think it helps them focus a little throughout the year on the skills they must learn, and perhaps less likely to leave their lab partner to it. Lab reports and even lab books are good but only measure practical skills indirectly. As we’re fond of telling our project students, their project doesn’t have to ‘work’ for them to write it up in a manner that gets a good mark. Indeed you could be pretty terrible in the lab, rely extensively on the demonstrators for support (excluding those who require support for specific reasons of course) and still get a decent mark. And perhaps that’s commentary on how we publish science – the publication can be spun in many ways, often to tell a more satisfying tale of how the work developed, than the typically haphazard way projects evolved. And possibly spun to make weaker results stronger in support of hypotheses. There’s something to be said about reporting science ‘as it happened’ and sharing raw data! In any case, we’ll be watching the actual experiments tomorrow, determining if the students make the desired product, and whether the lab diary record would be reproducible or not.