Research Insights #5: Student-Focused Studies Part 4

The exploration of student-focused research continues

It’s been a little bit since I last jumped into the research and I still have a bunch of students focused on just students.

Image Source: DALL-E 3 from ChatGPT 4

Liu, Chen-Chung, et al. "Teachable Q&A Agent: The Effect of Chatbot Training by Students on Reading Interest and Engagement." Journal of Educational Computing Research (2024): 07356331241236467.

Generative AI summary

Based on the comprehensive analysis of the provided document on the "Teachable Q&A Agent: The Effect of Chatbot Training by Students on Reading Interest and Engagement," here is the structured response as per your request:

Part 1 - Study's Findings and Educational Implications:

The study explored the impact of a teachable Q&A agent on enhancing students' reading interest and engagement. By incorporating a learning-by-teaching approach where students trained chatbots using AI techniques, the research aimed to boost students' involvement in reading activities. The findings revealed significant improvements in students' reading interest and engagement due to the chatbot training activity. Notably, the diversity in question types students asked was linked to their interest and engagement levels, indicating the importance of fostering a variety of questioning skills in learners.

Educational Implications:

Enhancing Student Engagement: The integration of teachable Q&A agents in reading activities can significantly boost students' engagement and interest. Educators can leverage this approach to make reading sessions more interactive and enjoyable, potentially leading to improved reading habits and skills.

Questioning Skills: The study highlighted the importance of diverse questioning skills in maintaining student interest and engagement. Educational strategies should include developing students' abilities to ask a variety of questions, enhancing their critical thinking and comprehension skills.

Part 2 - Strategies, Methodologies, and Theoretical Perspectives:

The study employed a quasi-experimental design with a focus on learning-by-teaching methodologies using AI-powered chatbots. Students interacted with these chatbots, training them with questions and answers, thereby actively participating in their learning process. This approach aligns with the theoretical perspective that teaching others can enhance one's own learning and engagement.

Application in Higher Education:

Interactive Learning Environments: Higher education institutions can adopt similar AI-driven tools to create more interactive and student-centered learning environments. This can facilitate deeper engagement with the material and promote active learning among students.

Skill Development: The methodology emphasizes the importance of questioning skills, which are crucial for higher education students. Educators can design activities that encourage students to develop and practice these skills, potentially enhancing their analytical and critical thinking abilities.

Part 3 - Gaps, Challenges, and Limitations:

Generalizability of Findings: The study was conducted with a specific demographic (fifth graders) in a particular educational setting. This raises questions about the generalizability of the findings to other age groups, educational levels, and contexts. Educational developers should consider how the age and background of learners might influence the effectiveness of teachable Q&A agents in enhancing reading interest and engagement.

Teacher Involvement: The document primarily focuses on student interaction with the chatbots, with little mention of the role of teachers in facilitating this process. Understanding the teacher's role in integrating and supervising chatbot interactions is crucial for educational developers, as teacher guidance can significantly influence the outcomes of such technological interventions.

Depth of Engagement: While the study indicates an increase in reading interest and engagement, it does not deeply explore the quality and depth of this engagement. Are students engaging with the material at a surface level, or are they developing deeper comprehension and critical thinking skills? Educational developers should delve into the nuances of engagement to ensure that such technologies foster meaningful learning experiences.

Technical Limitations and Ethical Concerns: The study briefly touches on the accuracy of chatbots but does not fully address the potential technical limitations and ethical concerns related to AI in education, such as data privacy and the potential reinforcement of biases. Educational developers need to be aware of these challenges and work towards developing responsible and ethical AI tools for educational purposes.

Part 4 - Application to Educational Development:

Professional Development for Educators: Educational developers can use insights from this study to design professional development programs that train teachers on integrating AI tools effectively in their teaching. This includes not only technical training but also pedagogical strategies to maximize the benefits of such tools for student learning.

Curriculum Design: The findings highlight the potential of chatbot interactions to enhance reading interest and engagement. Educational developers can leverage this by incorporating similar AI-driven activities into curriculum design, ensuring that these activities align with learning outcomes and complement traditional teaching methods.

Assessment Strategies: The study's insights into the types of questions students ask and how these relate to their engagement levels can inform assessment strategies. Educational developers can guide faculty in creating assessments that encourage diverse and critical questioning, mirroring the interactive nature of chatbot training.

Scalability and Customization: The study demonstrates a model where AI can be used to provide personalized learning experiences at scale. Educational developers can explore how such models can be adapted and scaled for different subjects, educational levels, and learning environments, ensuring that AI tools meet the diverse needs of learners.

Research and Evaluation: Encouraging ongoing research and evaluation is crucial. Educational developers should advocate for continuous assessment of AI tools' effectiveness in various educational settings, encouraging faculty to engage in research and share best practices for implementing AI in teaching and learning.

Incorporating these insights and recommendations into educational development can pave the way for more informed and effective use of AI in education, enhancing teaching practices and student learning experiences.

My Commentary

Right off the bat, this ignites a couple interests of mind. That reading becomes more interactive resonates with me as much of my life, reading has happened alongside or in tandem with access to the internet so that my queries could be answered. I could follow up inquiries that come up in the reading to help me situate something. The increased questioning skills also resonate strongly as I've been interested in question development ever since starting to read Warren Berger in the mid-2010s and, of course, since figuring out my own research question for the dissertation. Questioning is something that I think is now doubly important as we start to engage with generative AI tools taht are both blank-slates of context and incredibly skilled at drawing in lots of information.

The method description makes me want to dig more to better understand what interactions looked like. That it was with 5th grades might feel like it is beyond the use of folks in higher education, but I wonder about how this might be used to better infuse and re-invest folks in strategic reading. By the time students are in college, they often have a firm view of whether reading is for them or not. (Though some may return to it years after college by choice rather than by the artificial construct of mandated reading of lots of texts, few of which are inviting or resonate).

The outputs are all interesting ideas though, as is often the case with generative AI, they are as potentially interesting as they are practically useless. Though, ideally, this would entail continuing to inquire and learn more to figure out how it would fit in a given context.

---

Varshney, A. K., Garg, A., Pandita, S., Gaur, M. P., Singhal, R. K., & Sharma, H. (2024). Exploring the Impact and Factors to Consider in Higher Education Practice: A Study in Reference to Generative Artificial Intelligence. European Economic Letters (EEL), 14(1), 1970-1978.

Generative AI summary

Part 1 - Study's Findings and Educational Implications

Findings:

The document explores the impact of generative artificial intelligence (GAI) on higher education, particularly focusing on international students. It discusses the challenges in detecting AI-generated content, the biases within AI models, and the necessity for AI literacy and ethical usage. The study acknowledges the potential and problems of AI in education, advocating for a balanced approach that includes ethical considerations and competency frameworks.

Educational Implications:

For higher education, the findings suggest a need for institutions to develop strategies that recognize the benefits and challenges of AI. For instance, the difficulty in detecting AI-generated content calls for a reevaluation of assessment methods to ensure fairness and integrity. The paper's emphasis on AI biases highlights the importance of developing AI literacy among educators and students to critically assess AI-generated content. Additionally, the focus on international students underscores the need for inclusive practices that consider the varied backgrounds and language proficiencies of the student body.

Part 2 - Strategies, Methodologies, and Theoretical Perspectives

Strategies and Methodologies:

The study employs a non-systematic literature review, focusing on works published between 2019 and 2023 on AI in higher education and its impact on international students. This method allows for a broad exploration of the topic, though it may not be as exhaustive as a systematic review. The use of the AI competency framework as a strategy to address AI-related challenges in education provides a structured approach to developing AI literacy and ethical guidelines for AI usage in academic settings.

Theoretical Perspectives:

The paper applies theoretical perspectives related to AI's role in education, including concerns about academic integrity, bias in AI, and the digital divide. It also touches on the theoretical underpinnings of AI literacy, suggesting that understanding and interacting with AI should be a fundamental component of modern education. This perspective aligns with broader educational theories emphasizing critical thinking, ethical reasoning, and technological proficiency as essential skills in the 21st century.

Application and Enhancement of Document's Value:

By integrating these strategies, methodologies, and theoretical perspectives, the document enhances its value by providing a comprehensive view of the challenges and opportunities presented by AI in higher education. It not only identifies potential problems but also offers practical solutions, such as the promotion of AI literacy, that educators can implement to leverage AI's benefits while mitigating its risks. This approach helps in shaping a more informed, ethical, and inclusive educational environment that prepares students for a future where AI plays a significant role.

Part 3 - Gaps, Challenges, and Limitations:

Identified Gaps, Challenges, and Limitations:

Non-Systematic Review: The study employs a non-systematic review of literature, which may miss critical studies, lack comprehensive coverage, and could introduce selection bias. A systematic review could provide a more exhaustive and unbiased overview of the existing research.

Generalization of AI Impact: The document primarily focuses on international students without deeply exploring the diverse impacts of AI across different demographics, disciplines, or educational contexts. This limitation could affect the applicability of findings to broader educational settings.

Detection Tool Efficacy: The discussion on the efficacy of AI-generated content detection tools acknowledges their limitations but does not delve into potential future improvements or alternatives in detecting such content.

Bias and Fairness: While the document mentions biases within AI, it lacks a deep exploration of how these biases manifest in educational tools and the potential long-term effects on various student populations.

Lack of Empirical Data: The paper discusses theoretical implications and frameworks but lacks empirical evidence to support its claims, particularly concerning the impact of AI on academic integrity and student learning outcomes.

Considerations for Educational Developers:

Educational developers should be aware of these limitations and strive to fill these gaps through their own research or practice. They should consider the nuanced impacts of AI across different educational settings and student groups. Developers should also stay informed about advancements in AI detection tools and strategies to mitigate AI biases in educational contexts. Empirical studies could be valuable to validate the theoretical frameworks and recommendations presented in the document.

Part 4 - Application to Educational Development:

Relevance to Educational Developers and Faculty:

AI Literacy and Ethical Use: The emphasis on AI literacy is crucial for educational developers to integrate into curricula and professional development. Understanding AI's capabilities, limitations, and ethical implications is vital for faculty and students to navigate the evolving educational landscape.

Bias and Inclusivity: The discussion on AI biases is particularly relevant for developing inclusive teaching practices that address the potential disadvantages AI may impose on certain student groups, especially international and non-native English speakers.

Assessment and Academic Integrity: The challenges associated with AI-generated content and academic integrity underscore the need for educational developers to innovate in assessment design, ensuring it evaluates student understanding and critical thinking beyond what AI can generate.

Application in Educational Development:

Developing AI Literacy Programs: Educational developers can design and implement AI literacy programs that empower faculty and students to critically engage with AI technologies, understand their implications, and utilize them responsibly.

Inclusive Pedagogy: Developers should consider the document's insights on biases and adapt their pedagogical strategies to ensure that AI tools do not exacerbate inequalities or undermine the learning experience of diverse student populations.

Redefining Assessment: Given the limitations of AI detection tools, educational developers should lead in crafting innovative assessment methods that are less susceptible to AI manipulation, focusing on skills like critical thinking, creativity, and problem-solving.

In summary, the document provides a foundational understanding of the challenges and opportunities presented by AI in higher education. Educational developers should take these insights to inform their practices, ensuring they address the gaps and limitations identified while leveraging the potential of AI to enhance teaching and learning.

My Commentary

This article review starts out promising with the mention of impact of AI tools on international students, but from the review, it feels like I walk away with very little specific thinking about international students in the space of generative AI. It seems to lean into a space of being skeptical of AI usage both by educators and students, which at least feels more balanced. Towards the middle, there's a line about "technological proficiency" as one of the "essential skills in the 21st century." That always makes me wary because it's not like these skills are ubiquitous nor is access and nor is technology static enough for that to happen. In the last 25 years, we see amply leaps and shifts in technologies (across different arenas, so how do we all keep up?).

I do find some of the gaps and limitations to be redundant--reminders again about the limitations and gaps as AI as an output. For instance, the "generalization of AI Impact" section notes that focusing on international students limits the generalizability. For sure--and also, that's the intentional population being explored--probably, in part, because other studies are too general. Nevermind "who" makes up the generalized population. It's things like that which demonstrate some of the bias in these models.

The insights or applications for educational development--and this is something that I want to track over the span of these posts seems to raise some of the greatest hits regarding issues around these tools: AI literacy, ethical considerations, bias & inclusion, assessment improvement. The one that stands out from the the normal results is inclusive pedagogy, though it makes sense in why it's present and makes me think about what the curriculum of AI and inclusive pedagogy might look like because while this seems to hint at turning away from it, I wonder how much it might also help some pockets of diverse student populations.

---

Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study. Smart Learning Environments, 11(1), 9.

Generative AI summary

Part 1 - Study's Findings and Educational Implications

Main Findings

The study explored the impact of ChatGPT as a feedback tool on undergraduate ESL students' academic writing skills. The findings revealed a significant positive impact on the students' writing abilities, as evidenced by their improved performance in post-intervention tests compared to a control group. This enhancement was sustained over time, as indicated by the delayed post-test results. Additionally, students' perceptions of ChatGPT's feedback were overwhelmingly positive, with many noting improvements in idea generation, content organization, and grammatical accuracy.

Educational Implications

These findings have substantial implications for higher education, particularly in ESL writing instruction. ChatGPT, as a formative feedback tool, can significantly enhance students' writing skills, offering personalized, immediate feedback. This can be especially beneficial in large classrooms where individualized feedback from instructors may not be feasible. Furthermore, the study supports the integration of AI-driven tools like ChatGPT in writing curricula to foster autonomous learning and peer collaboration, which are critical skills in higher education settings.

Part 2 - Strategies, Methodologies, and Theoretical Perspectives

Strategies and Methodologies

The study employed a mixed methods intervention design, incorporating quantitative and qualitative analyses to assess the impact of ChatGPT on writing skills. The methodology included pre-tests, post-tests, delayed post-tests, and focus group discussions. This comprehensive approach allowed for a robust evaluation of ChatGPT's effectiveness as a feedback tool and its acceptance among students. The intervention's design, emphasizing direct application in the classroom and student training on using ChatGPT, demonstrates a practical strategy for integrating AI tools in educational settings.

Theoretical Perspectives

The study is grounded in theories of feedback as a dialogic tool and the pedagogical value of AI in education. By using ChatGPT, the study leverages the dialogic nature of AI-driven feedback, aligning with theories that emphasize interaction and dialogue in the learning process. The positive results reinforce the idea that AI tools like ChatGPT can serve as effective feedback mechanisms, promoting student engagement, and facilitating deeper learning in academic writing.

Application and Value

This study's methodologies and theoretical underpinnings offer valuable insights for educational developers and instructors. By illustrating a successful integration of ChatGPT in an ESL writing course, the study provides a model for leveraging AI tools to enhance educational outcomes. It underscores the importance of blending traditional instructional strategies with innovative AI technologies to address the diverse needs of students in higher education.

Part 3 - Gaps, Challenges, and Limitations:

Identified Gaps and Challenges:

Limited Genre Focus: The study focused on only three writing genres, which may not encompass the full range of ESL academic writing. This limitation could impact the generalizability of the findings to other writing contexts or genres, such as argumentative or narrative writing.

Short Intervention Duration: The six-hour intervention period may not be sufficient to capture the full extent of ChatGPT's impact on writing skills. Longer-term studies could provide deeper insights into the sustained effects of AI feedback on student writing.

Lack of Artifacts: The study did not include student writing artifacts in the analysis, which could have provided richer qualitative data on the specific improvements or changes in student writing attributable to ChatGPT feedback.

Dependence on Technology: The study hints at a potential over-reliance on AI tools, which could inhibit students' independent thinking and problem-solving skills in writing.

Considerations for Educational Developers:

Educational developers should be aware of these limitations when integrating AI tools like ChatGPT into writing instruction. They should consider expanding the range of writing genres and extending the duration of interventions to better assess the tools' effectiveness. Including writing artifacts in the analysis can provide a more nuanced understanding of AI's impact on student writing. Finally, developers should balance the use of AI tools with activities that promote independent critical thinking and writing skills to avoid over-dependence on technology.

Part 4 - Application to Educational Development:

Relevance to Educational Developers and Faculty:

Enhanced Feedback Mechanisms: The study's findings highlight ChatGPT's potential as a valuable feedback tool in writing instruction, especially in large classrooms. Educational developers can leverage these insights to design and implement AI-enhanced feedback systems that provide timely, personalized, and detailed feedback to students.

Facilitation of Peer Learning: The positive impact of ChatGPT on promoting peer collaboration offers a model for using AI to facilitate peer learning and assessment in writing. Developers can integrate AI tools to encourage peer interactions, enhancing the learning experience and fostering a collaborative learning environment.

Professional Development: The successful integration of ChatGPT in the study underscores the need for training educators in utilizing AI tools effectively. Educational developers can create professional development programs that equip faculty with the skills to integrate AI in teaching and provide guidance on balancing AI use with traditional instructional methods.

Application and Adaptation:

Curriculum Integration: Educational developers can work with faculty to integrate ChatGPT and similar AI tools into writing curricula, using the study as a reference point for designing interventions that improve writing skills.

Research and Evaluation: Encouraging faculty to conduct their research on AI's impact on writing and other skills can contribute to a deeper understanding of these tools' effectiveness and limitations.

Ethical Considerations: Developers should emphasize ethical considerations in using AI, guiding faculty and students on the responsible use of AI tools in academic work, addressing concerns about originality, and ensuring that students develop their writing skills independently.

By addressing the study's gaps and challenges and applying its insights thoughtfully, educational developers can enhance writing instruction, leverage AI technology effectively, and contribute to the field's ongoing evolution in the digital age.

My Commentary

This study feels like it is in an opposite space of the one above wherein it focuses on how might these tools aid ESL students in their writing. As a formative feedback tool, it seems to have some positive impact on student writing. I think this is where there's a lot of potnetial and so research like this feels important (and reinforcing my own bias around the ways that technology may be beneficial to learning and growth).

I find it strange that in the application section, we find out there was an element of peer collaboration in this study but not before. It makes sense and makes me wonder about how generative AI could be used in teams with two or more students, but I would want to know more about how it is integrated among peers.

---

Johnston, H., Wells, R. F., Shanks, E. M., Boey, T., & Parsons, B. N. (2024). Student perspectives on the use of generative artificial intelligence technologies in higher education. International Journal for Educational Integrity, 20(1), 2.

Generative AI summary

Part 1 - Study's Findings and Educational Implications

Main Findings:

The study aimed to understand student perspectives on generative artificial intelligence (GAI) technologies, like ChatGPT, to inform changes to the University of Liverpool's Academic Integrity code.

A significant majority of students were aware of GAI technologies, with over half having used or considered them for academic purposes.

There was a general support for using GAI tools for tasks like grammar checking, but a strong opposition towards using them to write entire essays.

Students with higher confidence in academic writing were less likely to use GAI for academic purposes and less supportive of others using it.

A notable portion of students believed there should be clear university policies on GAI use, advocating for neither a complete ban nor unrestricted usage.

Educational Implications:

These findings suggest a need for universities to develop clear policies on GAI technology usage that balance innovation with academic integrity.

There's a potential for GAI tools to assist in learning, particularly for students who may benefit from supplementary support, like international students or those with disabilities.

The data indicates a shift towards a more nuanced understanding of academic dishonesty, differentiating between tools aiding in learning and those facilitating dishonesty.

Educators and policymakers should consider incorporating student perspectives in creating or updating academic integrity policies, ensuring they reflect current technological landscapes and student experiences.

Part 2 - Strategies, Methodologies, and Theoretical Perspectives

Strategies and Methodologies:

The study utilized surveys and focus groups to collect data, ensuring a broad and inclusive understanding of student perspectives.

Data analysis included statistical methods to discern patterns and associations between students' confidence levels, knowledge of GAI technologies, and their views on academic integrity.

Theoretical Perspectives:

The research aligns with constructivist perspectives, emphasizing the importance of student voices in shaping educational policies and practices.

It also resonates with theories of digital ethics, highlighting the evolving nature of academic integrity in the age of digital technology and AI.

Impact on Content:

The methodologies used provided a comprehensive view of the student body's stance, allowing for informed decision-making regarding academic integrity policies.

Theoretical perspectives underscore the importance of adapting educational frameworks to accommodate the rapid advancement of technology, ensuring they remain relevant and effective.

In summary, this document offers valuable insights into student perceptions of GAI technologies in higher education, underscoring the necessity for clear, nuanced, and inclusive policies that support academic integrity while embracing technological advancements.

Part 3 - Gaps, Challenges, and Limitations:

Identified Gaps and Challenges:

Scope of Technologies Discussed: The study focuses significantly on ChatGPT and similar AI tools, potentially overlooking a broader spectrum of GAI technologies that could influence academic integrity. This narrow focus might limit the applicability of the findings to a wider range of emerging technologies.

Diversity of Student Perspectives: While the study captures a broad range of student opinions, it may not fully account for the diversity of student backgrounds, particularly in terms of academic disciplines, cultural backgrounds, and accessibility needs.

Rapid Evolution of GAI Technologies: The fast-paced advancement of GAI technologies presents a challenge for educational integrity policies to stay current, a dynamic not deeply explored in the document.

Methodological Limitations: The reliance on self-reported data and surveys could introduce biases or inaccuracies in understanding students' true perceptions and behaviors concerning GAI technologies.

Implications for Educational Developers:

Inclusivity in Policy Development: Educational developers should consider a broader array of GAI technologies and their potential implications for academic integrity, ensuring policies are inclusive and adaptable to future technological advancements.

Contextual Sensitivity: There's a need for educational policies that are sensitive to the diverse student body, accommodating various academic and cultural backgrounds.

Proactive Policy Adaptation: Educational developers should adopt a proactive stance, regularly updating policies to align with the rapid technological advancements in GAI.

Comprehensive Data Collection: Future research should employ a mix of quantitative and qualitative methods, ensuring a deeper and more nuanced understanding of student perspectives on GAI technologies.

Part 4 - Application to Educational Development:

Relevance to Educational Developers and Faculty:

Policy Development: The findings underscore the need for clear, communicated policies on GAI use, highlighting the balance between leveraging technology for educational benefits and upholding academic integrity.

Educational Support: The document's insights into student confidence and the use of GAI tools suggest that educational developers can use such technologies to support students, particularly those who might struggle with academic writing or language barriers.

Faculty Awareness and Training: The study highlights the importance of faculty awareness and training on the capabilities and limitations of GAI technologies to guide students effectively.

Adapting Recommendations for Educational Development:

Guidelines and Training: Develop comprehensive guidelines and training programs for faculty and students on the ethical use of GAI technologies, informed by the study's findings on student perceptions and usage patterns.

Innovative Assessment Design: Leverage insights from the study to design assessments that encourage critical thinking and originality, reducing the temptation or utility of using GAI for dishonest purposes.

Continuous Dialogue: Foster a culture of continuous dialogue and feedback among students, faculty, and educational developers regarding GAI technologies, ensuring that policies and practices evolve in tandem with technological advancements.

Research and Collaboration: Encourage further research into the impact of GAI technologies on different aspects of education and promote collaboration between educational developers, technologists, and ethicists to address emerging challenges.

By integrating these insights and recommendations into their work, educational developers can play a pivotal role in shaping how higher education institutions navigate the complexities introduced by GAI technologies, ensuring they enhance learning while maintaining academic integrity.

My Commentary

This study explores students' views on the Academic Integrity code of the University of Liverpool and has some interesting findings. More than half were using generative AI technologies (not surprising but interesting to have it documented). It's not surprising that students felt strongly against using them in essay writing and that there was a correlation between usage level and writing confidence. That students wanted guidance that was balanced is not surprising. I certainly saw that with my students as we developed our institutional policy. It's this kind of research that resonates with why I wanted to get into these discussions because I saw too quickly back in January 2023 what the conversation among administrators and faculty was and what they assumed about their students.

I do find the application section here a little bit disappointing--though not surprising coming from a generative AI tool. That only one of the recommendations (Continuous Dialogue) includes students actively feels like a missed opportunity. It would seem that nearly every recommendation provided could include students as contributors and participants to improve the recommendation.

---

Meta-commentary

In this collection of articles, I feel like there's a way of connecting them that could be interesting.

If learning to engage with generative AI can improve questioning skills (article 1) as well as writing skills (article 3), then those skills can be used to enhance our understanding of the challenges generative AI represents for international students (article 2) and be integrated into policy development (article 4). See that--now all it needs is a nice little bow.

But seriously, we have two articles that lean into the tool with students and two that just note that students are aware and using it and how do we account for that. I'm glad to see more research focused on students but would love to see more research with students or taking a community-engaged approach with students in this work.

---

AI+Edu=Simplified by Lance Eaton is licensed under Attribution-ShareAlike 4.0 International