Validation of tests using an argument-based approach: a review based on the PRISMA model
DOI:
https://doi.org/10.51798/sijis.v6i4.1177Keywords:
validity, argumentation, assessment, test, inference, educational testingAbstract
In test assessment, the validation process is crucial for ensuring the proper interpretation and use of scores, as well as establishing reliability in the results. The Argument-Based Approach (ABA), introduced by Kane, provides a structured framework for validation by defining inferences, warrants, and assumptions. Although widely recognized, research indicates that the empirical application of ABA remains limited, with many studies neglecting evaluations of clarity, coherence, and plausibility. This study conducted a systematic review using the PRISMA framework, analyzing 28 empirical articles about ABA published between 2014 and 2023. Articles were sourced from databases such as ERIC and Web of Science, selected based on quality and relevance to educational contexts. Most of the reviewed studies originated from North America and focused on English language education and medicine. Common inferences included Generalization and Extrapolation; however, few studies addressed warrants and assumptions, reflecting a partial application of ABA and insufficient evaluations of validity. Despite its utility, the ABA's implementation is hindered by gaps in evaluations focusing on clarity, coherence, and plausibility. Standardized guidelines are recommended for assessing these aspects in future research, expanding the framework's applicability to strengthen its empirical foundation.
References
Abu Dabrh, A. M., Waller, T. A., Bonacci, R. P., Nawaz, A. J., Keith, J. J., Agarwal, A., ... Angstman, K. B. (2020). Professionalism and inter-communication skills (ICS): A multi-site validity study assessing proficiency in core competencies and milestones in medical learners. BMC Medical Education, 20(1), Article 2290. https://doi.org/10.1186/s12909-020-02290-3
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. American Educational Research Association.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Andersen, N. B., O’Neill, L., Gormsen, L. K., Hvidberg, L., & Morcke, A. M. (2014). A validation study of the psychometric properties of the Groningen Reflection Ability Scale. BMC Medical Education, 14(1), Article 214. https://doi.org/10.1186/1472-6920-14-214
Asociación Nacional de Universidades e Instituciones de Educación Superior (2023, 23 de abril). Anuarios estadísticos de educación superior. Ciclo escolar 2021-2022 y 2022-2023 de ANUIES. http://www.anuies.mx/informacion-y-servicios/informacion-estadistica-de-educacion-superior/anuario-estadistico-de-educacion-superior
Atchison, D., Garet, M. S., Smith, T. M., & Song, M. (2022). The validity of measures of instructional alignment with state standards based on Surveys of Enacted Curriculum. AERA Open, 8, Article 23328584221098761. https://doi.org/10.1177/23328584221098761
Aviad-Levitzky, T., Laufer, B., & Goldstein, Z. (2019). The new Computer Adaptive Test of Size and Strength (CATSS): Development and validation. Language Assessment Quarterly, 16(4), 418–437. https://doi.org/10.1080/15434303.2019.1649409
Bakewell, O., & Garbutt, A. (2005). The Use and Abuse of the Logical Framework Approach: A Review of International Development NGOs’ Experiences. Swedish International Development Cooperation Agency (Sida).
Backhouse, S., Chiavaroli, N.G., Schmid, K.L. et al. Assessing professional competence in optometry – a review of the development and validity of the written component of the competency in optometry examination (COE). BMC Med Edu, 21 (11). https://doi.org/10.1186/s12909-020-02417-6
Bejar, I. I., Deane, P. D., Flor, M., & Chen, J. (2017). Evidence of the Generalization and Construct Representation Inferences for the GRE ® revised General Test Sentence Equivalence Item Type. ETS Research Report Series, 2017(1), 1–25. doi:10.1002/ets2.12134
Chapelle, C. (2012). Validity argument for language assessment: The framework is simple…. Language Testing. 29. 19-27. 10.1177/0265532211417211.
Chapelle, C. (2021). Argument-Based Validation in Testing and Assessment. SAGE.
Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32(3), 385–405. doi:10.1177/0265532214565386
Chapelle, C. A., & Voss, E. (2014). Evaluation of language testing through validation research. In A. Kunnan (Ed.), Companion in language assessment (pp. 1–13). Wiley.
Chapelle, C., Enright, M., & Jamieson, J. (2010). Does an Argument-Based Approach to validity make a difference? Educational Measurement: Issues and Practice, 29(1), 3-13.
Chapelle, C., Enright, M., & Jamieson, J. (2008). Building a Validity Argument for the Test of English as a Foreign Language. Routledge.
Chapelle, C. A., & Douglas, D. (2006). Assessing language through computer technology. Cambridge University Press.
Cheng, L., & Sun, Y. (2015). Interpreting the impact of the Ontario Secondary School Literacy Test on second language students within an argument-based validation framework. Language Assessment Quarterly, 12(1), 50–66. https://doi.org/10.1080/15434303.2014.981334
Choi, Y. (2021). What interpretations can we make from scores on graphic-prompt writing (GPW) tasks? An argument-based approach to test validation. Assessing Writing, 48, Article 100523. https://doi.org/10.1016/j.asw.2021.100523
Choi, Y. (2022). Validity of score interpretations on an online English placement writing test. Language Testing in Asia, 12(42). https://doi.org/10.1186/s40468-022-00187-0
Cook, D., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: a practical guide to Kane’s framework. Medical Education, 49, 560-575. doi: 10.1111/medu.12678
Deygers, B., Van den Branden, K., & Van Gorp, K. (2017). University entrance language tests: A matter of justice. Language Testing, 1-28. doi:10.1177/0265532217706196
Duckor, B., Castellano, K. E., Téllez, K., Wihardini, D., & Wilson, M. (2014). Examining the internal structure evidence for the Performance Assessment for California Teachers. Journal of Teacher Education, 65(5), 402–420. https://doi.org/10.1177/0022487114542517
Durson, A., & Li, Z. (2020). A Systematic Review of Argument-Based Validation Studies in the Field of Language Testing (2000–2018). Validity Argument in Language Testing, 45–70. doi:10.1017/9781108669849.005
Esfandiari, M. R., Riasati, M. J., Vaezian, H., & Rahimi, F. (2018). A quantitative analysis of TOEFL iBT using an interpretive model of test validity. Language Testing in Asia, 8(7). https://doi.org/10.1186/s40468-018-0062-7
Fechter, T., Dai, T., Cromley, J. G., Nelson, F. E., Van Boekel, M., & Du, Y. (2021). Developing a validity argument for an inference-making and reasoning measure for use in higher education. Frontiers in Education, 6, Article 727539. https://doi.org/10.3389/feduc.2021.727539
Ferguson, K. J., Kreiter, C. D., Franklin, E., Haugen, T. H., & Dee, F. R. (2020). Investigating the validity of web-enabled mechanistic case diagramming scores to assess students’ integration of foundational and clinical sciences. Advances in Health Sciences Education, 24, 695–709. https://doi.org/10.1007/s10459-019-09944-y
Gotch, C. M., & French, B. F. (2020). A validation trajectory for the Washington Assessment of Risks and Needs of Students. Educational Assessment, 25(1), 65–82. https://doi.org/10.1080/10627197.2019.1702462
Hatala, R., Cook, D. A., Brydges, R., & Hawkins, R. (2015). Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence. Advances in health sciences education: theory and practice, 20(5), 1149–1175. https://doi.org/10.1007/s10459-015-9593-1
Hidri, S. (2021). Linking the International English Language Competency Assessment suite of examinations to the Common European Framework of Reference. Language Testing in Asia, 11, Article 9. https://doi.org/10.1186/s40468-021-00123-8
Huang, H.-T. D., Hung, S.-T. A., Chao, H.-Y., Chen, J.-H., Lin, T.-P., & Shih, C.-L. (2022). Developing and validating a computerized adaptive testing system for measuring the English proficiency of Taiwanese EFL university students. Language Assessment Quarterly, 19(2), 162–188. https://doi.org/10.1080/15434303.2021.1984490
Kane, M. (2006). Content-Related Validity Evidence in Test Development. Handbook of Test Development. doi:10.4324/9780203874776.ch7
Kane, M. (2011). Validating score interpretations and uses. Language Testing, 29(1), 3–17. doi:10.1177/0265532211417210
Kane, M. (2013). Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement, 50(1), 1–73. doi:10.1111/jedm.12000
Kane, M. (2016). Validation strategies: Delineating and validating proposed interpretations and uses of test scores. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.), Handbook of test development (2nd ed., pp. 64-80). Routledge.
Kane, M., & Bridgeman, B. (2017). Research on validity theory and practice at ETS. In R. E. Bennett & M. von Davier (Eds.), Advancing human assessment: The methodological, psychological and policy contributions of ETS (pp. 489–552). Springer Science + Business Media. https://doi.org/10.1007/978-3-319-58689-2_16
Kane, M., & Mislevy, R. (2017). Validating score interpretations based on response processes. In K. Ercikan & J. W. Pellegrino (Eds.), Validation of score meaning for the next generation of assessments (pp. 39–51). Routledge.
Kane, M., Crooks, T., & Cohen, A. (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18, 5–17. doi:10.1111/j.1745-3992.1999.tb00010.x.
Knoch, U., & Chapelle, C. A. (2017). Validation of rating processes within an argument-based framework. Language Testing, 1–23.
Koizumi, R., In’nami, Y., Asano, K., & Agawa, T. (2016). Validity evidence of Criterion® for assessing L2 writing proficiency in a Japanese university context. Language Testing in Asia, 6, Article 5. https://doi.org/10.1186/s40468-016-0027-7
Kumazawa, T., Shizuka, T., Mochizuki, M., & Mizumoto, A. (2016). Validity argument for the VELC Test® score interpretations and uses. Language Testing in Asia, 6(1). https://doi.org/10.1186/s40468-015-0023-3
LaFlair, G. T., & Staples, S. (2017). Using corpus linguistics to examine the extrapolation inference in the validity argument for a high-stakes speaking assessment. Language Testing, 34(4), 451–475. https://doi.org/10.1177/0265532217713951
Lavery, M., Bostic, J., Kruse, L., Krupa, E., & Carney, M. (2020). Argumentation Surrounding Argument‐Based Validation: A Systematic Review of Validation Methodology in Peer‐Reviewed Articles. Educational Measurement: Issues and Practice. doi:10.1111/emip.12378
Lee, E. (2020). Evaluating test consequences based on ESL students’ perceptions: An appraisal analysis. Studies in Applied Linguistics & TESOL, 20(1), 1–22. https://doi.org/10.7916/salt.v20i1.3394
Li, S. (2018). Developing a test of L2 Chinese pragmatic comprehension ability. Language Testing in Asia, 8, (3). https://doi.org/10.1186/s40468-018-0054-7
Marcinek, T., Jakobsen, A., & Partová, E. (2023). Using MKT measures for cross-national comparisons of teacher knowledge: Case of Slovakia and Norway. Journal of Mathematics Teacher Education, 26(3), 303–333. https://doi.org/10.1007/s10857-021-09530-3
McGill, D. A., van der Vleuten, C. P. M., & Clarke, M. J. (2015). Construct validation of judgement-based assessments of medical trainees’ competency in the workplace using a “Kanesian” approach to validation. BMC Medical Education, 15(1). doi:10.1186/s12909-015-0520-1
Mendoza, A., & Knoch, U. (2018). Examining the validity of an analytic rating scale for a Spanish test for academic purposes using the argument-based approach to validation. Assessing Writing, 35, 41–55. https://doi.org/10.1016/j.asw.2017.12.003
Messick S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, (18), 2, 5-11.
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2004). A brief introduction to evidence-centered design. CRESST/University of Maryland. National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
Moscoloni, N. (2004). El análisis multidimensional de datos como herramienta metodológica en ciencias sociales. VI Jornadas de Sociología. Facultad de Ciencias Sociales, Universidad de Buenos Aires.
Newton, P., & Shaw, S. (2014). Validity in educational & psychological assessment. SAGE.
Nomura, O., Wiseman, J., Sunohara, M., Akatsu, H., & Lajoie, S. P. (2021). Japanese medical learners’ achievement emotions: Accounting for culture in translating Western medical educational theories and instruments into an Asian context. Advances in Health Sciences Education, 26(4), 1255–1276. doi:10.1007/s10459-021-10048-9
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). Declaración PRISMA 2020: Una guía actualizada para la publicación de revisiones sistemáticas. Revista Española de Cardiología, 74(9), 790–799. https://doi.org/10.1016/j.recesp.2021.06.016
PubMed. (n.d.). Retrieved from https://pubmed.ncbi.nlm.nih.gov/advanced/
Rafatbakhsh, E., & Ahmadi, A. (2022). The Argument-Based Validation of a Large-Scale High-Stakes Vocabulary Test. Practical Assessment, Research, and Evaluation, 27. https://scholarworks.umass.edu/pare/vol27/iss1/28
Rodríguez, B. (2004). El modelo argumentativo de Toulmin en la escritura de artículos de investigación educativa. Revista Digital Universitaria, 5(1), 2-18. https://www.revista.unam.mx/vol.5/num1/art2/ene_art2.pdf
Scriven, M. (1967). The Methodology of Evaluation. In R. W. Tyler, R. M. Gagne, & M. Scriven (Eds.), Perspectives of Curriculum Evaluation Chicago (pp. 39-83). Rand McNally.
Seipel, B., Kennedy, P. C., Carlson, S. E., Clinton-Lisell, V., & Davison, M. L. (2023). Using Kane’s framework to build an assessment tool for undergraduate medical student’s clinical competency with point of care ultrasound. BMC Med Educ, 23 (43). https://doi.org/10.1186/s12909-023-04030-9
Sheppard, G., Williams, K. L., Metcalfe, B., Clark, M., Bromley, M., Pageau, P., Woo, M., Yi, Y., Devasahayam, A. J., & Dubrowski, A. (2023). Using Kane’s framework to build an assessment tool for undergraduate medical students’ clinical competency with point of care ultrasound. BMC Medical Education, 23, Article 43. https://doi.org/10.1186/s12909-023-04030-9
Tavares, W., Brydges, R., Myre, P., Prpic, J., Turner, L., Yelle, R., & Huiskamp, M. (2018). Applying Kane’s validity framework to a simulation-based assessment of clinical competence. Advances in Health Sciences Education, 23(2), 323–338. https://doi.org/10.1007/s10459-017-9800-3
Toulmin, S. (1958). The uses of argument. Cambridge University Press.
Yan, X., & Staples, S. (2019). Fitting MD analysis in an argument-based validity framework for writing assessment: Explanation and generalization inferences for the ECPE. Language Testing, 36(1), 1–26. https://doi.org/10.1177/0265532219876226
Yaneva, V., Clauser, B.E., Morales, A. & Paniagua, M. (2022). Assessing the validity of test scores using response process data from an eye-tracking study: a new approach. Adv in Health Sci Educ 27, 1401–1422. https://doi.org/10.1007/s10459-022-10107-9
Youn, S. J. (2015). Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32(2), 199–225. https://doi.org/10.1177/0265532214557113
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Karla Karina Ruiz Mendoza, Luis Horacio Pedroza Zúñiga, Alma Yadhira López García
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
