Research Insights #11: AI and Assessment

Exploring a few articles discussing assessment and generative AI

Jul 15, 2024

There’s still plenty to explore regarding generative AI and students’ experiences per my last exploration into the research. But I felt like taking a break from those to look at some on assessments. Maybe because it’s summer and I feel like folks are trying to prepare for the fall semester or it’s just the folder I happen to be looking at right now, but let’s explore!

I've chosen assessment because it is something that I know is a hot topic and one that I keep going back and forth about. I figured I'd take a dive into the research to see what's there currently

a diverse group of adult college students feeding their homework papers into a humanoid robot in a classroom. The robot's eyes light up with each paper it receives, all done in the style of a Sunday newspaper comic strip. — Source: ChatGPT 4-o

.Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21(1), 40.

Chatlog here

Generative AI summary

Part 1 - Study's Findings and Educational Implications
Main Findings:
The scoping review by Xia et al. (2024) explores how generative artificial intelligence (GenAI) transforms assessment in higher education. The study highlights three primary levels of impact: students, teachers, and institutions. The findings reveal that GenAI introduces both opportunities and challenges:
For Students:
Opportunities: GenAI provides immediate and diverse feedback, perceived unbiased feedback, and aids in self-assessment.
Challenges: It raises significant concerns about academic integrity and ethical behavior, particularly regarding the potential for cheating.
For Teachers:
Opportunities: GenAI enhances assessment literacy, fosters more diverse and innovative assessment methods, and drives teaching towards critical thinking and problem-solving.
Challenges: Teachers must balance human and AI assessment, develop new assessment literacies, and adapt to the changing landscape of education driven by AI technologies.
For Institutions:
Opportunities: Institutions are prompted to redesign assessment policies, develop new literacy and professional development programs, and shift educational focus towards holistic and interdisciplinary learning.
Challenges: Institutions face the need to disrupt traditional assessment methods and ensure academic integrity in the era of AI.
Educational Implications:
The findings suggest that higher education institutions must adapt to the evolving landscape of AI-integrated education. Key implications include:
Policy Redesign: Institutions need to revisit and reformulate assessment policies to address the changes brought about by GenAI, ensuring that assessments are meaningful and uphold academic integrity.
Professional Development: Teachers must engage in continuous professional development to stay updated on AI technologies and their applications in education, fostering a deeper understanding of AI literacy and assessment strategies.
Curriculum Development: Educators should incorporate AI and digital literacy into curricula, emphasizing critical thinking, creativity, and problem-solving skills.
Holistic Learning: The shift towards AI in education requires a focus on whole-person development, including traits like grit, perseverance, and ethical behavior.
Part 2 - Strategies, Methodologies, and Theoretical Perspectives
Strategies and Methodologies:
The review employs a scoping methodology, adhering to the PRISMA Extension for Scoping Reviews guidelines. This approach allows for a comprehensive exploration of existing literature, providing a broad overview of the impact of GenAI on assessment in higher education.
Data Collection: Articles were selected from three databases (ERIC, Web of Science, and Scopus) using specific search terms related to assessment, higher education, and generative AI.
Coding and Analysis: The selected articles were analyzed at three levels—students, teachers, and institutions—to extract relevant data on the impact of GenAI on assessment.
Theoretical Perspectives:
The study is grounded in the theoretical framework of digital and AI literacy, emphasizing the need for educators and students to develop competencies in using AI technologies. It also highlights the importance of self-regulated learning, ethical behavior, and interdisciplinary education.
Impact on Article's Ideas and Outcomes:
The methodologies and theoretical perspectives shape the study's comprehensive analysis and its forward-thinking approach. By focusing on digital and AI literacy, the study underscores the necessity for continuous learning and adaptation in the face of rapidly evolving AI technologies.
Applications for Higher Education:
Assessment Design: Educators can leverage AI to design innovative assessments that promote critical thinking and problem-solving.
Professional Development: Institutions should implement training programs to enhance teachers' AI literacy and assessment skills.
Policy Formulation: Policymakers must consider interdisciplinary approaches and ethical guidelines when developing AI-related educational policies.
Part 3 - Gaps, Challenges, and Limitations
Gaps and Challenges:
Lack of Specific Recommendations: The scoping review provides preliminary findings but lacks specific recommendations for implementing GenAI in educational assessments.
General Context of GenAI: Some studies reviewed discuss GenAI in a general context without detailing specific technologies, leading to potential gaps in understanding the precise impact of different AI tools.
Limitations:
Preliminary Nature: As a scoping review, the study offers an initial exploration rather than definitive conclusions. Systematic reviews and meta-analyses are needed for more detailed insights.
Ethical Concerns: Addressing academic integrity and ethical behavior in the use of GenAI remains a significant challenge for educators and institutions.
Considerations for Educational Developers:
Ethical Frameworks: Developers should emphasize the importance of ethical AI use and integrity in educational settings.
Detailed Research: Further research is needed to provide specific guidelines and best practices for integrating GenAI into assessments.
Contextual Understanding: Developers should consider the unique contexts of different educational settings and tailor AI integration strategies accordingly.
Part 4 - Application to Educational Development
Relevance to Educational Developers and Faculty:
The study's findings offer valuable insights for educational developers and faculty, emphasizing the need for continuous adaptation and innovation in assessment practices.
Recommendations for Application:
Professional Development Programs: Implement training programs to enhance teachers' AI literacy and assessment skills, ensuring they are equipped to navigate the challenges and opportunities presented by GenAI.
Innovative Assessment Methods: Encourage the use of diverse and innovative assessment methods that promote critical thinking, creativity, and problem-solving skills, reducing reliance on traditional exams and written assignments.
Ethical Use of AI: Develop guidelines and frameworks to ensure the ethical use of GenAI in education, fostering a culture of integrity and responsible behavior among students.
Interdisciplinary Learning: Promote interdisciplinary programs and assessments that integrate knowledge from various fields, preparing students for the complexities of the modern world.
Adaptation and Implementation:
Educational developers can adapt the study's recommendations to create tailored professional development programs, redesign assessment policies, and integrate AI literacy into curricula. By fostering a holistic and interdisciplinary approach, developers can enhance the quality of education and prepare students for a future driven by AI technologies.

My Commentary

This piece is more proscriptive than actually providing evidence of what's been done. I know there's some disagreement on whether to use AI in assessment spaces but if you're in the "maybe" or "yes" category, this piece does a nice summary of the possibilities and challenges. Unsurprisingly, like much of the literature I'm coming across, it talks a lot about what should be done but not how to get there or most importantly, what should be taken off the plate. This feels additive without any sense of how to do this in the context of institutions already with 5 bajillion other things to navigate. To me, this is one of the challenges that there is a tailoring that needs to take place to each and every institution. It's not just a tailoring but then an entire campaign to get buy in, followed by the action to make it happen--all the while trying to figure out what is not happening. That's not new, of course; it's happening everywhere within higher education as it each institution has to take on more than it can handle in order to stay relevant.

Madland, C., Irvine, V., DeLuca, C., & Bulut, O. (2024). Developing the Technology-Integrated Assessment Framework. The Open/Technology in Education, Society, and Scholarship Association Journal, 4(1), 1-19.

Chatlog here.

Generative AI summary

Part 1 - Study's Findings and Educational Implications
Main Findings: The study by Madland et al. (2024) developed a new framework for understanding technology-integrated assessment in higher education. Key findings include:
Identification of Themes: Seven major themes were identified in the literature on technology-integrated assessment: focus on tools and tasks, instructor workload/efficiency, purposes of assessment, academic integrity and remote proctoring, assessment design, ethics and equity, and systemic transformations of practice.
Incongruities with Existing Frameworks: The study highlighted gaps and incongruities between the Bearman et al. (2022) framework and the reviewed literature, suggesting the need for an updated framework.
Development of New Framework: The Technology-Integrated Assessment Framework (TIAF) was proposed, which includes four constructs: assessment purposes, duty of care, technology acceptance, and assessment design, each with specific sub-constructs.
Educational Implications: The findings have significant implications for higher education:
Enhanced Assessment Practices: By adopting the TIAF, instructors can improve assessment practices, making them more efficient, inclusive, and aligned with contemporary digital environments.
Focus on Human-Centered Approaches: The emphasis on ethical and equitable assessment practices ensures that technology integration does not compromise the well-being and fairness towards learners.
Systemic Transformation: The framework provides a structured approach to incorporating technological advancements, such as AI, into assessment practices, ensuring they are pedagogically sound and ethically implemented.
Part 2 - Strategies, Methodologies, and Theoretical Perspectives
Strategies and Methodologies: The article discusses various strategies and methodologies:
Literature Review: A comprehensive review of literature from 2016 to 2023 to identify prevalent themes and gaps in technology-integrated assessment.
Framework Development: The authors developed the TIAF by integrating findings from the literature review with existing frameworks and their professional expertise.
Theoretical Perspectives:
Bearman et al. (2022) Framework: Served as the initial lens for analyzing the literature, highlighting areas where it was congruent and incongruent with contemporary practices.
Five Rs of Indigenous Education: This perspective informed the ethical and community-oriented aspects of the TIAF, emphasizing respect, relevance, reciprocity, responsibility, and relationships.
Impact on Ideas and Outcomes: These methodologies and theoretical perspectives ensure that the TIAF is comprehensive and applicable across diverse educational contexts. The integration of Indigenous perspectives particularly enriches the framework by embedding values of respect and community.
Application in Higher Education:
Improved Assessment Design: The TIAF guides educators in designing assessments that are not only technologically integrated but also ethically and pedagogically sound.
Inclusive Practices: The framework's focus on duty of care and inclusion ensures that assessments are accessible and fair to all learners.
Part 3 - Gaps, Challenges, and Limitations
Identified Gaps and Challenges:
Instructor Workload: The need for technologies that reduce instructor workload was identified but is not comprehensively addressed in existing frameworks.
Academic Integrity: Concerns about maintaining academic integrity in technology-mediated assessments, particularly with remote proctoring and AI advancements.
Equity and Ethics: The presumption that technology inherently increases equity, despite evidence of potential inequities.
Educational Developer Considerations:
Workload Management: Developers should advocate for and help implement technologies that genuinely reduce workload without compromising assessment quality.
Integrity Measures: Implement robust measures to maintain academic integrity, leveraging secure technologies and designing assessments that minimize opportunities for dishonesty.
Equitable Access: Ensure that technology integration does not exacerbate existing inequities and actively works to include all learners.
Part 4 - Application to Educational Development
Relevance to Educational Developers and Faculty: The TIAF provides a structured approach to integrating technology into assessment practices in a way that prioritizes ethical considerations and inclusivity.
Adaptation and Application:
Workshops and Training: Educational developers can use the TIAF as a basis for workshops and training sessions aimed at helping faculty design better assessments.
Policy Development: The framework can inform institutional policies on technology use in assessments, ensuring alignment with best practices and ethical standards.
Ongoing Support: Provide continuous support and resources for faculty to implement and refine technology-integrated assessments, addressing emerging challenges and incorporating feedback.
Conclusion: The study by Madland et al. (2024) offers a robust framework that significantly contributes to the field of higher education by guiding the ethical and effective integration of technology in assessment practices. Educational developers and faculty can leverage this framework to enhance teaching and learning outcomes, ensuring assessments are fair, inclusive, and aligned with contemporary educational needs.
References
Madland, C., Irvine, V., DeLuca, C., & Bulut, O. (2024). Developing the Technology-Integrated Assessment Framework. Open/Technology in Education, Society, and Scholarship Association Journal, 4(1), 1-19.
Bearman, M., Nieminen, J., & Ajjawi, R. (2022). Designing assessment in a digital world: An organising framework. Assessment & Evaluation in Higher Education, 48(3). https://doi.org/10.1080/02602938.2022.2069674
Tessaro, D., et al. (2018). The five Rs of Indigenous education. Journal of Indigenous Education, 21(1), 1-15.

My Commentary

I included this one even though it's not strictly focused on generative AI, it does provide some guidance. (Guidance that does seem to show up a bit in the article above). Like the piece above, there is this idea that when dealing with technology, it works best in teaching, learning, and assessment when we know we've also made the classroom a human-centered space. It's part of why I straddle between instructional design (a field often focused on technology) and faculty development (a field often uninterested in technology) so much--they feel intractable--and in all learning in the higher education space, technology is involved. I wasn't that excited about the generative AI summation of this but I did go back to the chat-log and ask it to create a table that would summarize the framework and provide examples which I thought was interesting.

Refer a friend

El Khoury, E. (2024). Mapping the Response to AI and Its Impact on Assessment Redesign Through Document Analysis. Assessment, 5(1).

Chatlog here.

Generative AI summary

Part 1 - Study's Findings and Educational Implications
Main Findings of the Study
The study "Mapping the Response to AI and Its Impact on Assessment Redesign Through Document Analysis" identifies seven principal themes from the document analysis of higher education institutions' guidelines on using generative AI (GenAI) in assessment:
Terminology: Diverse terminology used to describe AI's role in assessment.
Provisional Principles: Emerging provisional principles guiding GenAI's use.
Advice to Instructors: Specific advice provided to instructors for implementing GenAI.
Opportunity to Refocus on Assessment Purpose: Encouraging a refocus on the purpose of assessment.
Examples, Uses, and Roles of AI: Real-world and hypothetical examples of AI use in assessments.
Theories, Pedagogies, Concepts, and Analysis: Theoretical and pedagogical frameworks for AI integration.
First Wave: Recognition that these documents represent an initial wave of guidance, subject to change and evolution.
Educational Implications
These findings have significant implications for higher education:
Terminology: The varied terminology highlights the need for a shared understanding and language around AI in assessments, which can foster better communication and implementation strategies among educators.
Provisional Principles: Institutions are beginning to acknowledge both the opportunities and risks of GenAI. For example, the University of Guelph emphasizes the multifaceted impact of GenAI, influenced by different disciplinary cultures .
Advice to Instructors: The guidance documents provide practical advice for instructors, including how to incorporate GenAI into assessments while maintaining academic integrity. The British Columbia Institute of Technology's query about the value of assignments that can be automated by GenAI is particularly thought-provoking .
Refocusing on Assessment Purpose: This theme encourages educators to rethink assessment design to ensure it aligns with learning outcomes and skills needed for employability. For instance, King’s College London suggests a long-term approach to redesigning assessments entirely .
Examples, Uses, and Roles of AI: By providing real and hypothetical examples, institutions like the University of Sydney help educators visualize how AI can be integrated into their teaching practices .
Theoretical and Pedagogical Frameworks: The integration of relational pedagogies, such as the ethics of care, into AI assessment design suggests a more humane approach to technology use in education .
First Wave: Recognizing this as the initial wave of guidance underscores the need for continuous adaptation and reevaluation as understanding of GenAI evolves .
Part 2 - Strategies, Methodologies, and Theoretical Perspectives
Strategies and Methodologies
The study employs document analysis, reviewing 135 English-speaking websites from higher education institutions and educational organizations. This method involves examining and evaluating documents to extract useful information, identify patterns, and develop understanding through thematic analysis .
Theoretical Perspectives
Theoretical perspectives include:
Relational Pedagogies: Incorporating relational pedagogies, such as the ethics of care and trauma-informed pedagogy, emphasizes the importance of creating meaningful and humane assessments .
Constructive Alignment: The University of Maryland employs constructive alignment, where learning outcomes are clarified before teaching, ensuring assessments align with these outcomes .
Impact on Ideas and Outcomes
These strategies and theoretical perspectives emphasize a holistic and adaptive approach to integrating AI in education. For instance, relational pedagogies ensure that technology use does not undermine the human element of education, while constructive alignment ensures assessments remain relevant and purposeful despite technological advancements.
Part 3 - Gaps, Challenges, and Limitations
Identified Gaps and Challenges
Terminological Confusion: The varied terminology used to describe AI in assessment can create confusion and hinder effective communication among educators and institutions .
Evolving Nature of Guidelines: The recognition that these documents represent an initial wave implies that guidelines are still evolving and may not yet be fully comprehensive or universally applicable .
Implementation Challenges: Practical challenges include detecting unauthorized GenAI use, which some institutions find unfeasible, leading to varied enforcement and monitoring practices .
Considerations for Educational Developers
Educational developers should focus on creating a common language around AI in assessments to mitigate confusion. Additionally, they should be prepared for continuous updates and adapt to evolving guidelines. Addressing practical implementation challenges requires robust support systems and clear policies to ensure consistent and fair application of AI technologies.
Part 4 - Application to Educational Development
Relevance to Educational Developers and Faculty
The study's findings are directly relevant to educational developers and faculty as they navigate the integration of GenAI in assessment design. The practical advice and examples provided in the guidance documents can help educators understand how to effectively incorporate AI into their teaching practices.
Adapting Recommendations
Developing AI Literacy: Institutions like the University of Technology Sydney emphasize the importance of developing AI literacy among students and staff to prepare them for a technologically advanced workforce .
Creating Authentic Assessments: By focusing on the purpose of assessments and designing authentic, meaningful tasks, educators can ensure that assessments remain relevant and valuable even in the age of AI .
Balancing Technology and Educational Values: Educational developers should strive to balance technological advancements with core educational values, ensuring that assessments are equitable, relevant, and beneficial to all students .
Conclusion
The study "Mapping the Response to AI and Its Impact on Assessment Redesign Through Document Analysis" provides a comprehensive overview of the initial responses of higher education institutions to the integration of GenAI in assessments. By highlighting the diverse approaches and emerging trends, the study offers valuable insights and practical guidance for educational developers and faculty, emphasizing the need for continuous adaptation and thoughtful integration of AI technologies in education.

My Commentary

I appreciate this article's attempt to coral all the different conversations going on in the assessment discussion by drawing first on what are we talking about when we say generative AI and assessment. Even in my own dive into it, when talking about GenAI and assessment, it can be confusing. Are we talking about AI's role in assessing students; students using AI for assessment, using AI for communicating the assessment that's been done by a human? There's a lot to explore so defining and clarifying is a good first go. Additionally, I appreciate that the article draws upon specific examples from different universities and colleges. To me, this document at least feels like it would be a good start for many institutions to explore to help them figure out their own strategies and guidance about how to go forward with generative AI in assessment.

Foung, D., Lin, L., & Chen, J. (2024). Reinventing Assessments with ChatGPT and Other Online Tools: Opportunities for GenAI-empowered Assessment Practices. Computers and Education: Artificial Intelligence, 100250.

Chatlog here.

Generative AI summary

Part 1 - Study's Findings and Educational Implications
Main Findings:
The study "Reinventing Assessments with ChatGPT and Other Online Tools: Opportunities for GenAI-empowered Assessment Practices" highlights several key findings:
Student Acumen with AI Tools: Students displayed a strong ability to choose and use appropriate AI tools for specific tasks.
Flexibility and Equity in AI Tool Usage: The study emphasizes the importance of flexibility in assessment practices to allow the use of various AI tools, advocating for equitable access to these tools.
GenAI Tool Benefits and Concerns: While students found GenAI tools helpful for brainstorming, generating outlines, and grammar checks, they also noted limitations like false information and the discrepancy between free and premium versions.
Educational Implications:
Enhanced Student Learning: The ability of students to choose suitable AI tools suggests that integrating GenAI tools into the curriculum can enhance learning by fostering critical evaluation and decision-making skills.
Assessment Redesign: The study calls for assessments that accommodate the use of different AI tools, promoting a balanced approach that leverages both traditional and GenAI tools. This can prevent overreliance on a single tool and ensure a more comprehensive learning experience.
Equity in Education: Addressing the differences between free and premium versions of AI tools is crucial. Institutions should consider providing access to premium versions to ensure all students can benefit equally, thus promoting fairness in educational opportunities.
Part 2 - Strategies, Methodologies, and Theoretical Perspectives
Strategies and Methodologies:
Qualitative Research Approach: The study adopted a qualitative design, analyzing students' written reflections and conducting focus group interviews to gather in-depth insights into their experiences with AI tools.
Thematic Analysis: A thematic analysis was employed to identify key themes from the data, such as the perception and use of tools, limitations, and critical evaluation.
Theoretical Perspectives:
Postmodernist Paradigm: The study is rooted in a postmodernist paradigm, focusing on the subjective experiences and perceptions of learners rather than seeking a single "truth."
Balanced AI Adoption: The study aligns with Chan's (2023) recommendation for a balanced approach to AI adoption, promoting the use of various online tools rather than over-relying on GenAI tools.
Impact on Ideas and Outcomes:
Comprehensive Assessment Practices: The use of qualitative methods and a balanced theoretical approach has led to the development of more inclusive and flexible assessment practices that cater to diverse student needs.
Critical Evaluation Skills: By encouraging students to use and critically evaluate various AI tools, the study fosters essential skills that are beneficial for their academic and future professional lives.
Part 3 - Gaps, Challenges, and Limitations
Gaps and Challenges:
Limited Generalizability: The study's findings are based on a specific group of students (SYA students in a communication course), which may not be generalizable to students with higher language proficiency or in different contexts.
Potential Bias in Data Collection: The involvement of one of the authors in teaching the participants could introduce bias in data collection and interpretation.
Equity Issues: The study highlights equity issues related to access to premium AI tools, which need to be addressed to ensure fair opportunities for all students.
Educational Developers' Considerations:
Diverse Student Needs: Educational developers should consider the diverse needs and backgrounds of students when integrating AI tools into the curriculum.
Bias Mitigation: Ensuring unbiased data collection and interpretation is crucial for the validity of research findings.
Policy Development: Institutions should develop policies to provide equitable access to AI tools, addressing the digital divide among students.
Part 4 - Application to Educational Development
Relevance to Educational Developers and Faculty:
Assessment Redesign: The findings can guide educational developers in redesigning assessments to incorporate AI tools effectively, promoting a balanced and inclusive approach.
Professional Development: Faculty can benefit from understanding the critical evaluation of AI tools, enhancing their ability to support students in using these tools constructively.
Equity in Access: Recommendations for providing access to premium AI tools can be adapted to ensure all students have equal opportunities to leverage technology in their learning.
Adaptation and Application:
Workshops and Training: Implementing workshops and training sessions on the effective use of AI tools can help both students and faculty maximize the benefits of these technologies.
Policy Implementation: Developing and implementing clear policies on AI tool usage and providing access to premium versions can address equity issues and promote fair educational practices.
Continuous Feedback Loop: Establishing a feedback loop where students and faculty share their experiences with AI tools can inform ongoing improvements in assessment practices and AI tool integration.
References
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20(1), Article: 38.
Chen, J. (2023). Four directions for assessment redesign in the age of generative AI. The Times Higher Education.
Mizumoto, A. (2023). Data-driven learning meets generative AI: Introducing the framework of metacognitive resource use. Applied Corpus Linguistics, 3(3), Article: 100074.

My Commentary

This piece actually looked at what students experienced in using generative AI in their activities. So a bit different from the previous pieces above that were discussing its role in assessment differently. The recommendations seem to be much the same as what we saw before but I'm intrigued to see that the students had some of their own concerns about access issues and the limitations of use for these tools. Again, it continues to reinforce my own belief that students are not as naive as we often portray them to be.

AI + Education = Simplified

Discussion about this post