Research Insights #3: Student-Focused Studies Part 1
Exploring the recent research on generative AI and students.
If you’re just tuning in, you can see the first and second pieces in this series. I’ve been enjoying using generative AI to explore some of the research out there on generative AI because:
There’s so much of it.
I’ve never going to get to it on my own.
This gives me a particular approach to wade through what I encounter.
It provides relevant content to this space.
What I’m starting today is a bit of a short series within the larger series. In scanning through my to-be-read folder of PDFs on generative AI and education, I found about 15 or so of them have focused on students and decided that would make for a good collection of articles to explore together.
But since there are 15 of them, it made sense for me to break that up into a series. I imagine by the time I’m done, there will be even more to add. So in each post, I’ll cover 3-4 as I have before—doing the generative AI summary, my commentary, and then a total commentary about all of it. However, when I’m done with this sub-series, I’ll piece together all the article findings on students (both myself and with AI).
Sun, R., & Deng, X. (2024). Using ChatGPT to Enhance Experiential Learning of College Students.
Generative AI summary
Main Points and Suggestions: The study explores using ChatGPT in experiential learning for college students, incorporating Kolb's Experiential Learning Model and the Revised Bloom's Taxonomy. It reveals how ChatGPT prompts designed by instructors influence the types of questions students later pose to ChatGPT, emphasizing the significant role of instructors in guiding AI-based learning. The findings suggest that ChatGPT can be an effective tool for enhancing various cognitive processes and knowledge types in learning.
Supplementary Details: The research highlights a unique dependence of student-created prompts on those provided by instructors, suggesting a directional influence in learning. It also notes the effectiveness of ChatGPT in covering various cognitive levels from understanding to creating. This points to a new dynamic in student engagement with AI in education, where the quality of instructor prompts significantly shapes student interactions with AI tools.
Practical Considerations and Implementation Recommendations:
Design of Learning Activities: Utilize AI tools like ChatGPT in structured activities that align with experiential learning models. Instructors should create initial prompts that are diverse and challenging to encourage students to explore a range of cognitive processes.
Encourage Critical Thinking: Teach students to critically evaluate AI-generated responses. This can be incorporated as a part of learning activities where students not only use ChatGPT for information but also assess its accuracy and relevance.
Continuous Feedback and Adaptation: Regularly review and adapt the use of AI tools based on student feedback and learning outcomes. This iterative process ensures that the integration of AI in teaching remains effective and relevant.
My Commentary
This looks promising and I went back to the thread to engage with the article a little bit more to get a stronger sense of what it meant with how prompts could be beneficial (full thread here). I got a clearer idea of what the prompts would look like but even more importantly, I asked it to create one such prompt. When it did, it was pretty bland but I edited it to ask about what a specific prompt for early American literature might look like and it came out much better. Furthermore, I asked it to explain why the prompt it created exemplified the two ideas in use and it broke it down quite clearly.
I think an article like this could be quite useful for faculty looking to integrate in it in meaningful ways. Additionally, it feels like this opens up a way to engage also with metacognition with students and discuss using the theories as a grounding for their learning. That is, one might not just use the prompts structured by Kolb's Experiential Learning Model and the Revised Bloom's Taxonomy, but explain those ideas to students to help them in their overall usage and interaction with generative AI.
Pellas, N. (2023). The Effects of Generative AI Platforms on Undergraduates’ Narrative Intelligence and Writing Self-Efficacy. Education Sciences, 13(11), 1155.
Generative AI summary
Main Points, Ideas, Arguments, and Suggestions: The document investigates the impact of generative AI platforms on undergraduates’ narrative intelligence and writing self-efficacy. It contrasts generative AI platforms with conventional storytelling tools in an educational setting. The study utilizes a pretest-posttest design and finds that generative AI significantly enhances students' narrative intelligence and writing self-efficacy. It emphasizes the potential of AI in education, suggesting its integration into curricula to foster narrative skills and confidence in writing.
Summary of the Thread: This thread has explored the article's methodology, findings, and implications for educational development. We've discussed the study's empirical approach, its focus on narrative intelligence and writing self-efficacy, and its evidence base. The thread highlighted the surprising finding that generative AI can significantly enhance narrative skills, a traditionally human-led domain. It also noted limitations like the small sample size and the lack of focus on creative identity.
Practices and Recommendations for Educational Developers:
Integration of AI Tools: Introduce AI platforms in curriculum design to enhance narrative intelligence and writing skills. For implementation, start with pilot projects to assess the effectiveness of these tools in specific courses.
Professional Development: Organize workshops for educators on the effective use of AI in teaching, focusing on narrative skill enhancement and fostering writing self-efficacy.
Research and Evaluation: Conduct further studies to explore the broader applicability of AI in education, focusing on diverse educational settings and larger sample sizes.
Curriculum Design: Incorporate findings from this study into curriculum design, emphasizing the role of AI in developing narrative and writing skills. Collaborate with AI experts to tailor these tools to specific educational needs.
Ethical Considerations: Educate faculty and students on ethical considerations related to AI use, including data privacy and the responsible use of AI-generated content.
My Commentary
These responses were exciting to consider but felt a bit hollow when exploring the article itself. That summary of the thread seemed to add little, so once again, I'll need to edit it and refine it more.
The gist of the article as a study of how generative AI was used as a support or enhance narrative intelligence and self-efficacy. I had hoped they include some examples, but I guess that's for use to figure out. I continued to ask questions of it and draw out more details but they were ok at best.
Chen, B., & Zhu, X. (2023). Integrating generative AI in knowledge building. Computers and Education: Artificial Intelligence, 5, 100184.
Generative AI summary
Comprehensive Explanation of the Document's Main Points, Ideas, Arguments, and Suggestions: The document "Integrating Generative AI in Knowledge Building" primarily explores how high school students in a World Religions class interact with and utilize ChatGPT, a generative AI tool. The study highlights how students engage in 'prompt engineering' to refine AI responses to aid their learning and knowledge building. It also delves into the varying levels of AI literacy among students, with some holding rudimentary understandings of AI as a mere 'database of the Internet'. The study emphasizes the critical and ethical reflection of students on AI use, recognizing risks like misinformation and dependency. Additionally, the role of the teacher is underscored as pivotal in guiding effective AI use and enhancing AI literacy among students. These insights provide a nuanced understanding of how AI tools like ChatGPT can be integrated into educational practices to support learning and encourage critical engagement with technology.
Supplementary Summary with Further Details and Information: The study presents unconventional approaches in educational settings, such as the active shaping of AI interactions by students through iterative query refinement, which is a shift from passive consumption to active engagement with AI. This approach is significant for educational developers as it highlights the evolving role of AI in education. The students' critical perspectives on the ethical use and potential misinformation risks of AI are also noteworthy. They show a level of awareness and critical thinking about AI's societal implications, which is not typically expected in high school students. Additionally, the study stresses the importance of the teacher's role in guiding students to use AI tools effectively, which is crucial for educational developers to consider in curriculum design.
Specific Practices, Actions, and Considerations for Educational Developers:
Practice: Integrating AI Tools in Curriculum - Implement AI tools like ChatGPT as part of the curriculum, ensuring they complement and enhance the learning objectives. Educators should design activities that encourage students to use AI not just for information retrieval but for critical thinking and knowledge building.
Action: Facilitating Prompt Engineering Skills - Train students in the art of prompt engineering to shape AI responses effectively. This could be done through workshops or integrated classroom activities where students learn to iteratively refine their queries.
Consideration: Ethical Use and Critical Engagement with AI - Develop modules or discussion sessions on the ethical use of AI, focusing on the potential risks, misinformation, and dependency. This encourages students to use AI tools responsibly and critically.
Recommendation for Implementation: To integrate these practices effectively, educational developers should collaborate with AI experts to design curriculum modules that blend AI literacy with subject matter. Regular training sessions for teachers on the latest AI tools and their educational applications are also essential. These steps ensure that both educators and students are equipped to navigate the evolving landscape of AI in education.
My Commentary
An interesting aspect of this article was that it focused on high school students and World Religions courses. I was certainly surprised that this was focused on high school students and world religions, no less, given that there are definitely bias issues around religions. The emphasis on AI literacy and the importance of the instructor is key; we know this but the grounding it in the study might be really important as we go forward and there is likely a push to reduce the role of educators (we know it's coming). The most interesting recommendation and one that I think we'll see more of in the next year is blending AI experts with specific disciplines. I did a few of these talks this year but they were still somewhat general. My guess is that stronger pairing and deeper exploration in given disciplines is more of what will be needed to better understand the best possibilities (and problems) of generative AI in teaching and learning.
Adams, D., Chuah, K. M., Devadason, E., & Azzis, M. S. A. (2023). From novice to navigator: Students’ academic help-seeking behaviour, readiness, and perceived usefulness of ChatGPT in learning. Education and Information Technologies, 1-18.
Generative AI summary
Comprehensive Explanation of the Document: The article investigates university students' readiness and perceived usefulness of ChatGPT for academic purposes, employing a sequential explanatory mixed-method design. It reveals students' positive perceptions of ChatGPT for simplifying complex concepts, providing prompt and accurate responses, and aiding in task completion. However, it also highlights concerns about overreliance on AI for critical thinking and problem-solving skills. The study emphasizes the need for balanced integration of AI tools in education.
Comprehensive Summary of the Thread: Previous discussions focused on the article's writing intention, readability, evidence, argument coherence, clarity of ideas, diversity of perspectives, and application to practice. It was noted for its empirical approach, evidence-based findings, and relevance in integrating AI into education. The study's potential impact on traditional student-teacher interactions and ethical considerations of AI use in education were highlighted, offering a nuanced view of AI's role in learning.
Practices for Educational Developers:
Implementation of AI Tools: Encourage a balanced use of AI tools like ChatGPT in the curriculum to augment learning while fostering critical thinking skills. Develop workshops to familiarize faculty with these tools.
Ethical and Responsible Use: Educate students and faculty about the ethical use of AI, emphasizing academic integrity and the importance of original thought.
Continuous Evaluation: Regularly assess the impact of AI tools on students' learning experiences, adapting strategies to ensure they complement, not replace, traditional teaching methods.
Customized Training Programs: Develop training modules specific to subject areas, showing how AI tools can be effectively integrated into different disciplines.
Collaboration and Feedback: Foster a collaborative environment where students and teachers can share feedback on the use of AI tools, promoting continuous improvement.
My Commentary
Ok--next time around, question 2 needs a full overhaul. I had tweaked it but it's still not quite getting at what I *think* I want here.
Still, I found that I could learn a bit more by asking for some examples of prompts to consider. I used 2 prompts; the first to elicit ideas ("Based upon the findings of the article and its recommendations, what specific ways might leverage generative AI for an early American Literature college course?") and the second to get prompt suggestions ("Provide 2 detailed prompts to use with generative AI for each out those practices. Include them in a table with the description of the category, the prompt, and why that prompt would be helpful."). These results were not amazing but could give a good start to engage with students to further explore a given topic.
Meta-commentary
Things I'm learning and thinking about based upon these:
All of these emphasize the importance of using it with students and helping to build some critical thinking skills in generative AI's use. Straightforward and understandable--reminds me of why I can't stand abstinence-only education; not talking about it does not make it go away and only makes folks more poorly understanding exactly what they are doing. That is, if we don't talk about or meaningfully engage with generative AI, we are not going to help our students.
In this set of articles, I found that I definitely got more out of it when I went back to ask more questions after doing the summaries. That's an obvious thing to do but one that I had hesitated to do with what direction to go. This time, it made sense to start asking about specific applications, prompts, or things to do next based on the article's findings.
I found the first article in this bunch to be the most intriguing in terms of its usage and grounding. It felt like I could get a clearer path forward on how one might apply and use the study and generative AI to enhance learning. The article on narrative intelligence and self-efficacy also had hints of this or at least some clarity on why it might be useful to students. The third article highlights misconceptions that need addressing and I want to look further to see what (if anything) it says about how it approaches World Religions with generative AI. The final article I think might provide a good set of practices to encourage usage in teaching and learning for faculty to consider in their courses.
Appendix
I’ve once again updated the prompts that I’m using. Some things of note include that including the word “must” improves the specific outcomes (e.g. you must write each paragraph rather than write a few paragraphs). I’ve also moved away from using the term “summary” at times because ChatGPT almost always reads that as a single paragraph.
Another interesting quirk that I’m finding is that if you leave a thread and come back later (e.g. the next day), ChatGPT doesn’t have the capacity to re-read or analyze that article. It seems like it doesn’t store the article, which means you would have to reload it to get more out of the article.
Prompt 1: Scoring the Article
You're an expert in educational development and reviewing academic research to elicit the most relevant and useful information and ideas to apply in education.
Assess the quality of the article. On a 10 scale (10 being the strongest), rate and explain the following aspects of the article. Think deeply about and consider that this is primarily for the benefit of educational developers.
You must draw upon and share specific details of the article for each explanation or insight you provide:
Writing Intention: Indicate what kind of writing piece this is such as a theoretical argument, an empirical study, a practical explanation, case study, etc. It can be more than one type of writing. Specify what the main objective or thesis. Additionally, determine this piece's intended audience(s).
Ease of Reading: Review the degree to which the authors rely on complex terminology and complicated language and assume what the audience knows. Consider to what degree the overall structure and organization aid understanding.
Evidence: Clarify the types of resources and evidence that are used. How consistent are the authors in drawing upon additional scholarship research to validate claims and how well do they demonstrate the evidence for their findings. Evaluate the reliance on qualitative, quantitative, mixed methods, primary sources, and other types of evidence.
Coherent, clear and strong argument: What the article is trying to communicate is structured in a way that guides the reader thoughtfully through all of its points. Identify if they engage with counterarguments or alternative perspectives.
Clarity of ideas: How straightforward the key points of the article can be deduced including the logical flow of the article, the explicitness of key points, or how well the article summarizes its main ideas.
Reliance on diverse perspectives and research: Consider interdisciplinary approaches, global versus local perspectives, intersectional considerations, historical and contemporary lenses or other types of approaches. The degree to which the research they draw upon represents a variety of standpoints in terms of time of publication, diversity of authorship, publication type, and other relevant factors.
Ability to directly apply ideas into practice: The ease of taking what the article discusses and translating it into teaching, learning, and supporting faculty in developing teaching and learning.
Prompt 2: Reviewing the article
Review the attached document. As an expert educational developer, you must provide a detailed response of the attached article, focusing on its key points, findings, and recommendations. In particular, highlight how the article's insights and conclusions are applicable to the work of educational developers and faculty.
Each paragraph of your discussion must include specific details and examples from the document.
The first paragraph must discuss any specific strategies, methodologies, or theoretical perspectives presented in the document that can inform or improve teaching and learning practices.
The second paragraph must identify any gaps, challenges, or limitations mentioned in the article that would help educational developers in their work.
Your response must offer significant depth to help educational developers determine the article's relevance and value to their work.
Prompt #3: Interesting Findings
Review the attached document and identify all surprising or unconventional findings or approaches for educational developers or educators?
You must include specific examples and detailed explanations about why they are surprising, unconventional, and/or previously not discussed in the literature on this topic.
Prompt #4: Effective summary
Review the attached document. As an expert educational developer, you must produce the following responses.
The first paragraph must be a comprehensive explanation of the document's main points, ideas, arguments and suggestions.
The second paragraph must be a comprehensive summary of this thread but must not repeat anything that has already been said in the previous paragraph. It should supplement it with further details and information that might be relevant and useful for an educational developer.
The third paragraph must include several specific practices, actions, and considerations for an educational developer to integrate into their work based upon the attached document and your summary. For each practice, action, or consideration, you must also include a recommendation for how to implement it.
Every paragraph you generate must include specific and distinct details that are not repeated from previous paragraphs and draw upon examples from the document or extend and apply the ideas to other possibilities.
Prompt #5: Relevant keywords
Provide a bulleted list of 5-8 keywords (no explanations). The keywords should draw from the article, be relevant and beneficial to help educational developers categorize this article.
AI+Edu=Simplified by Lance Eaton is licensed under Attribution-ShareAlike 4.0 International
