Designing a fraud-proof certification program

It is important that a test is taken without any cheating or fraud happening. Otherwise, there is a negative impact on the validity, reliability, and credibility of a test. Firstly, it is uncertain if the ability of a student is tested or that (s)he knew all responses to the items that were asked. Secondly, the result of the test might depend more on items that were known to the test-takers rather than uncompromised items. Lastly, the credibility of a certificate can be questionable since it is uncertain whether the candidate has sufficient skills and knowledge. To make sure a test or program is valid, reliable, and credible it needs to be fraud-proof.

Cheating can happen on different levels. The test taker can copy the answers from another test taker, the answers might already be known to him/her, or a test taker can try to remember as many items of a test as possible and distribute or publish those items. On a larger scale, it is possible that an item bank might be hacked into so a lot of items can than be used in practice tests which decreases the reliability of the test.

When designing a fraud-proof certification program, the following aspects are considered: discouraging, preventing, detecting, responding to, and recovering from fraud. The importance and how they are taken care of is discussed for every aspect.

Discouraging

To discourage test takers to cheat, it is important to let them know what the consequences of their behaviour will be. For example, having to do a retake or being expelled from a course or study. The bigger the consequences, the more chance there is that the test taker will think again before cheating. Discouraging people to cheat or commit fraud because the punishment is not worth it, is the first step in making a test or certification program fraud-proof.

Preventing

For step two, making it difficult to cheat, steal items, or bribe supervisors or teachers, multiple measures that can be taken. In the first case, leaving enough space between two test takers, placing physical obstacles, distributing different versions of a test, handing in electronic devices and books, checking for cheat papers or notes, having supervision walking around are a few examples of means to prevent cheating during a test. On a different level, making sure that the test is stored somewhere safe before it is taken prevents the possibility of fraud being committed with the test. Lastly, a background check on supervisors might help if they have a history of providing answers to test takers or pretending not to see someone cheating.

Detecting

If discouraging and prevention were not enough to stop cheating and fraud, software can help to indicate if fraud was committed and by whom. There are multiple methods to detect fraud statistically, two examples are the Guttman error model and log-normal response time (Klerk, Noord, & Ommering, 2006). Additionally, the likelihood ratio test and score test can be used to detect preknowledge of items (Sinharay, 2017).

            The Guttman error model is defined by the Guttman scale, patterns, and errors. Test items are ordered from least to most difficult; it is expected that when a question is answered incorrectly, all items that are more difficult are also answered incorrectly. In practice, this is not always the case, so that is why the Guttman error is calculated. The Guttman score = (number of Guttman errors) / (items answered correct * items answered incorrect) (Klerk & Bijl, 2020).

            The log-normal response time models the response times on test items for each test taker (Klerk & Bijl, 2020). A long response time can indicate that the test taker is trying to remember the item so that (s)he can pass it on to others later, while a short response time can indicate that the test taker has seen the item(s) before. The response time of the test taker is compared with the expected response time and the differences are analysed to see if the test taker has cheated on the test (Van Der Linden, 2006).

            Lastly, the detection of item preknowledge is important too. If items on a test are available on the internet, for example, the result of the test does not represent the skills or knowledge of the test taker. Therefore, the validity, reliability, and credibility of the test diminish or disappear (Klerk & Bijl, 2020).  

Responding

When cheating or fraud is suspected, the specific test taker should be contacted. During the conversation, this person should be informed about what exactly (s)he is accused of and have a possibility to explain or defend him-/ herself in case an error was made. It is possible that the test taker admits to fraudulent behaviour, in which case should be communicated what the punishment will be. The student could be expelled from the course, program, or school or an extra test or assignment could be a substitution for the test that does not count anymore.

In case the test-taker does not admit to having committed fraud, it might be that the test still is not admissible, and the student must retake the test or should make an alternative assignment. It might depend on how reliable the software to decide how accurate the detection is when indicating that someone cheated.

Recovering

The last step in the process is the recovery. Depending on how the fraud or cheating happened, the recovery consists of a different action. If it was possible to cheat during the test, more obstacles or more strict measures should be implemented. If items are compromised, the security of items banks should be improved. If such items were used in a test there should be more checking the internet or test prepare organizations to be aware if a lot of items are known and being used for test purposes. In case a test consists of compromised items, the test should be thrown out and a new test with secure items should be made. Even a company, for example, eX:plain, could be hired to inform or improve security, or provide training to decrease the possibility that fraud is committed (“Data Forensics,” n.d.).

References

Data Forensics. (n.d.). Retrieved June 22, 2020, from https://www.explain.nl/onderzoek-en-innovatie/data-forensics

de Klerk, S., & Bijl, A. (2020, June 15). Cheating and the prevention and detection of test fraud [Slides]. Retrieved from https://canvas.utwente.nl/courses/5049/pages/data-forensics?module_item_id=148083

de Klerk, S., van Noord, S., & van Ommering, C. J. (2006). The Theory and Practice of Educational Data Forensics. In B. P. Veldkamp & C. Sluijter (Eds.), Methodology of Educational Measurement and Assessment (pp. 1–20). https://doi.org/https://doi.org/10.1007/978-3-030-18480-3_20

Sinharay, S. (2017). Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test. Journal of Educational and Behavioral Statistics, 42(1), 46–68. https://doi.org/10.3102/1076998616673872

Van Der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. https://doi.org/10.3102/10769986031002181