Mark candidate responses on screen
The on screen marking (OSM) system facilitates on-screen marking for short and long text responses to complement the work of human examiners.
Accessed by web browser, the on-screen marking software automatically identifies the answer paper and extracts the relevant response areas from scanned images, presenting them to remote examiners for clerical marking in a secure, online environment. Invalid or unreadable responses will be shown to a marker for manual marking.
Markers are presented with an ‘Overview’ screen which shows their statistics and provides access to the ‘notice board’ and support messaging system.
Item level marking
Marking is carried out at item level rather than candidate level with markers able to view entire screens containing one question type (instead of one paper). This helps to identify incorrect responses quickly and expedites the marking process. It also reduces the effect of bias as each item on a candidate’s paper is marked by a different marker.
Once a marker has marked a pre-defined number of items, they are allocated a new question to prevent fatigue.
This approach to marking replaces whole script marking and reduces the effects of marker subjectivity.
Bulk item entry
If all responses displayed contain a correct answer, markers will be able to submit ‘correct marks for all’. Where there are incorrect responses, markers can flag these before submitting. The dashboard can also handle long item marking and multiple marks for relevant items.
The feedback from the team in terms of the on-screen marking solution was very positive and the consensus was that it was very user friendly and intuitive to use.
Operations Manager, International Assessment Organisation
Submitting marks
Once markers have completed a screen (i.e. indicated every response displayed to be correct or incorrect), they will be able to click a button to submit marks. If a marker attempts to submit without marking each candidate, an explanatory warning is displayed.
After ‘marks’ have been submitted for a screen, the process is repeated until all marking is finished. When a marker has finished their allocated marking and there are no more responses to mark, a message is displayed.
Team management
The system includes four different user types and each has different capabilities. The team structure, hierarchy and responsibilities are outlined below:
- admin user: can view statistics, manage users, team/products and system;
- team leader: can view team/marker
statistics, manage queues, release marks, manage
users and manage settings;
- monitor: can monitor errors/discrepancies by item and make judgments or issue re-scan requests if required; and
- marker: can mark items and view their marker statistics.
Quality control
Throughout, marking quality assurance rules are automatically enforced across a session, with doublemarking, error rates, marker probation periods, escalations and hierarchical marker monitoring built into the workflow. This enables you to identify training requirements and adhere to marking policies by only releasing accurate marks.
Remote marking
The on screen marking solution is accessed via a web browser with markers able to work remotely from home rather than travel to a marking centre.