Research, Blogs & Sources

Introduction

Artificial Intelligence is rapidly reshaping how education is delivered, offering new opportunities to enhance learning and streamline administrative tasks. Yet, the growing reliance on AI raises concerns about the balance between automation and the essential human touch. Education is more than data and algorithms; it involves empathy, context, and adaptability, all of which are inherently human qualities.

The concept of Human-in-the-Loop (HITL) provides a way to thoughtfully integrate AI in education. HITL systems involve human input, oversight, and decision-making alongside AI, supporting educators rather than replacing them. This paper explores the history of HITL across various domains, highlights key research findings, and proposes a framework to effectively apply HITL in education, balancing innovation with human insight.

Background

Human-in-the-Loop (HITL) originated from cybernetics and systems theory, emphasizing the role of humans in overseeing automated systems in critical fields. Pioneers like Norbert Wiener identified the value of feedback loops in maintaining human decision-making within technological processes. In HITL systems, the control loop is an iterative feedback mechanism that integrates human decision-making with system responses in real time.

This loop comprises several components: system input, where sensors and human inputs provide data; processing, where the system analyzes inputs and generates responses subject to human evaluation; human interaction, involving active human intervention to adjust system outputs; system output, where actions are executed affecting the environment; and a feedback loop, ensuring continuous system monitoring and real-time adjustments by humans.

Over time, HITL principles were adopted in diverse fields, including healthcare, robotics, and military applications, where human judgment was essential for ethical and effective operations. In healthcare, AI supports diagnostics, but physicians are vital for interpreting results and making decisions. Similarly, in military operations, HITL places human judgment at the center of AI-driven processes to address ethical concerns.

Autonomous vehicles depend on human intervention as a fallback mechanism for safety in unpredictable scenarios. Social media platforms combine AI detection of harmful content with human moderators who handle context-sensitive decisions. These examples highlight the value of combining AI with human expertise. Education can apply this approach by using AI for tasks like data analysis, allowing educators to focus on the relational and nuanced aspects of teaching.

The level of human involvement in HITL systems is determined by factors such as system complexity, level of automation, task criticality, and the purpose of the interaction (e.g., supervision, decision-making, or training).

Research in HITL AI has revealed its potential and limitations across various applications. For instance, in robotics, HITL approaches have shown significant advantages in blending human cognitive skills with machine autonomy. A study by Leeper et al. (2012) demonstrated that incorporating human guidance into robotic grasping systems improved task success rates and reduced operational errors. Similarly, in human-AI symbiosis, Becks and Weis (2022) explored how strategic nudging can optimize interaction by prioritizing human cognitive skills without over-dependence on AI. In education, these lessons are vital as AI systems must complement, not override, educators' expertise.

The role of HITL in ethical decision-making has also been emphasized. Fenwick and Molnár (2022) argued that humanizing AI through behavioral insights creates systems that align with human values. Meanwhile, in military contexts, Zweibelson (2023) raised concerns about the diminishing role of human operators in decision-making as AI systems grow more autonomous. These examples underscore the need for careful role definition and human oversight in HITL systems.

Proposed Framework

To effectively integrate HITL in education, a structured approach is needed. The six-step framework proposed here is designed to address educational needs while leveraging the strengths of both humans and AI. This framework is practical and adaptable, focusing on simplicity, necessity, and iterative improvement.

Conclusion

The HITL framework offers a balanced way to integrate AI in education. By focusing on needs analysis, simplicity, and clearly defined roles, this approach helps AI systems complement rather than replace educators. The proposed six-step framework draws on lessons from other domains and current research, providing a practical roadmap for applying HITL in education. This approach not only improves the effectiveness of AI systems but also respects the irreplaceable role of human educators in shaping learning experiences.

Call to Action

Educational institutions, policymakers, and technology developers are encouraged to adopt this HITL framework when considering AI solutions. Start by identifying the specific needs within your institution, explore simple alternatives, and carefully assess whether AI is necessary. If AI is introduced, prioritize human oversight and define clear roles to maintain a balance between technology and human expertise. Pilot programs should be rigorously evaluated, with adjustments made based on real-world feedback. By following this framework, stakeholders can harness the potential of AI while preserving the essential human element in education.

References

I've been searching for a more concrete and practical way to explain what human-centered use of AI looks like, and lately I've been learning about how to be a "human in the loop."

I discovered the term in this 2019 article by Ge Wang, a music and computer science professor at Stanford. He advocates for a human-centered, values-driven approach to AI systems design that recognizes where and how we derive meaning from different tasks.

"We don't just value the product of our work; we often value the process. For example, while we enjoy eating ready-made dishes, we also enjoy the act of cooking for its intrinsic experience — taking raw ingredients and shaping them into food... It's clear there is something worth preserving in many of the things we do in life, which is why automation can't be reduced to a simple binary between 'manual' and 'automatic.' Instead, it's about searching for the right balance between aspects that we would find useful to automate, versus tasks in which it might remain meaningful for us to participate."

This pre-ChatGPT insight reflects the most common concern I hear at schools in the post-ChatGPT era: that AI is threatening the value of learning processes (like writing) because it is so efficient at generating competent versions of learning products (like essays).

Wang suggests we should be designing AI in a way that requires the integration of human voice and insight. AI needs a human in the loop.

Being a Human in the Loop

Being a human in the loop means embracing the idea that generative AI is a tool, and because it is a tool, we have agency over it. Schools and individual educators may not have a huge amount of influence over how AI systems are designed, but we do have influence over how we teach students about human-centered use of AI and how to carry that approach with them into an AI world. I think the "human in the loop" concept applies not just to AI's design, but also to its use in schools.

Learning how to be a good human in the loop aligns well with learning how to nurture and develop agency, a longstanding goal of school. Consider Jennifer Davis Poon's excellent description of student agency and the competencies associated with it:

Agency + AI = Human in the Loop

So, what could this look like in our uses of AI?

1. Turn personal goals into good prompts

I recently met a teacher who wanted to help his 8th grade students use AI to help them be better project managers. His students were about to embark on an independent capstone research project, and the teacher knew that project management (allocating time, setting benchmarks and deadlines, reporting out on progress) would be a new skill for many of them. So, he created a template for a structured interaction with a chatbot. It included an initial prompt that students could copy and paste into the bot and suggestions for how to nudge it to get responses tailored to the student's individual project.

I think this is a good example of how to support students in effective use of AI: these students may not know enough about project management to know how to ask AI to help them with project management, but we as teachers do, so we can provide them with language and guidance on how to engage in a productive, personalized exchange.

2. Engage AI output actively, not passively

A student told me about how she approached AI in this way. If something she heard in a class was confusing to her, she would go home, take out the notes she had taken in class, and then ask ChatGPT to explain the concept or topic to her as if she was in fifth grade. She would then see if the bot's explanation clarified her understanding and allowed her to better understand the notes she had taken.

In many ways, this is interacting with AI as one would with a skilled tutor; its ability to instantly explain ideas at a variety of levels can support students in processing information. In addition, the student is not relying on AI as a single source of truth. She has her notes, she has what she remembers from class, and she has her ability to look across those sources to better understand what she is being asked to learn.

3. Teach and coach AI to be better

When I watch educators and students use AI, I am struck by how often the exchange ends after one prompt and one response. For example, "make a rubric about X for Y class" is not enough information for AI to produce a really good rubric, but I have seen teachers look at the first response and either use it immediately or reject it as low-quality and close the chatbot. Yet one of the most powerful features of AI is that it learns from and adjusts its behavior to our feedback.

This requires some patience and resilience. The work of determining exactly what feedback is needed and how to craft it is challenging: sometimes you need to make multiple attempts, and sometimes you have to start fresh with a whole new conversation to "reset" the bot. But, what you learn from that process is transferable to your future interactions with AI: just as we become better teachers and coaches with practice, so we become better trainers of AI.

4. Make AI inputs and outputs your own

Leon Furze has been doing a very good series on using AI to teach writing, and his ideas and suggested prompts reflect Wang's emphasis on process over product. Consider this suggested AI prompt from his post on using AI for exploration of texts:

"We are using mentor or model texts as a way of learning the techniques and style of quality writing. Generate an annotation checklist of things we could look for in our mentor texts related to style, structure, voice, tone, language use, word choice, etc."

Students can adjust this prompt according to the kind of mentor text they're working with (a news article, a blog post, an academic paper, etc.), and generate a checklist that includes elements relevant to their personal goals. Similar to the student who uses ChatGPT to explain complex concepts to her, Furze's prompts put the learner in the position of having to do something with the output: evaluate it, compare it to their own thinking, apply it to their chosen text, and consider how it can launch them in new directions.

5. Reflect on the experience

If we use AI with students, we should ensure there is time to reflect on the experience. In my own conversations with students, I find their reflections to be varied and interesting: some talk about "lightbulb moments" that AI enabled for them, others dismiss the output as mediocre and unhelpful, others are trying it in many different ways and landing on a few reliable uses for it, and others don't like using it at all for ethical reasons or out of fear that they'll "get caught" using AI and be accused of cheating.

I have written before about Tom Sherrington's quote that "understanding is the capacity to explain," and metacognitive work is a simple and powerful way to see if and how students are learning (with or without AI). I've also written about how and why we should talk openly with students about AI. Here's a simple reflection protocol I learned a few years ago from students and teachers at Urban Assembly Maker Academy that we could use as a reflective "check" as students use AI:

Conclusion

Being a good human in the loop requires knowledge. We have to access prior knowledge to create precise prompts, to evaluate AI outputs, to give AI good feedback, to customize the output to our needs, and to reflect on whether and how AI has been helpful. One of the most frequent concerns I hear expressed by teachers is that using AI will diminish students' interest in and ability to acquire knowledge. The human in the loop framework helps illustrate to students how knowledge supports agency, and that knowing things helps us use tools like AI to make our goals and visions for ourselves a reality.

Abstract

Background: Human-in-the-loop research predominantly examines the interaction types and effects. A more structural and pragmatic exploration of humans and Artificial Intelligence or AI is lacking in the AI in education literature.

Purpose: In this systematic review, we follow the Entity-Relationship (ER) framework to identify trends in the entities, relationships, and attributes of human-in-the-loop AI in education.

Methods: An overview of N = 28 reviewed studies followed by their ER characteristics are summarized and analyzed.

Results: The dominant number of two or three-entity studies, one-sided relationships, few attributes, and many to many cardinalities may signal a lack of deliberation on beings that come to interact and influence human-in-the-loop and AI in education.

Conclusion: The contribution of this work is identifying the implications of human-in-the-loop and AI from a more formal ER perspective and acknowledging the many possibilities for placement of humans in the loop with the AI, system, and environment of interest.

1. Introduction

The goal of this review is to understand the characteristics of humans in the loop and Artificial Intelligence (AI) in education. Work to date has examined the role of AI and human-in-the-loop for Natural Language Processing or NLP, text classification, question and Answering, computer vision or recommender systems. Such specialized fields of research may deter educators from understanding the role and impact humans and AI may play in educational settings.

A high-level view of AI and machine learning is that they are entities with attributes and impact through relationships they build with humans and the environment. We thus wish to find a common ground and evaluation framework for human-in-the-loop and AI in education. To achieve this, we follow the entity-relationship (ER) framework to code and chart entities, relationships, entity attributes, and cardinality between humans and AI.

1.1. Overview

Broadly, human-in-the-loop can be considered a model that requires human interaction. More specifically, human-in-the-loop may be used in multiple contexts such as human-computer interaction, teleoperation systems, and human-in-the-loop machine learning. Different operations may qualify as human-in-the-loop models such as one or a combination of prediction, and optimization.

In this work, we regard:

• Human in the loop: the contact of human beings with AI to complete a goal, operation, or task (in this work, the focus is educational).
• Artificial Intelligence or AI: a combination of technologies and algorithms that may include smart features that are employed as part of human-in-the-loop interaction.
• Education: teaching and learning contexts in institutions and industry settings.

1.2. Gap

The field of machine learning examines human-in-the-loop from more of a data-lifecycle perspective. Humans may be expected to intervene during data extraction, preprocessing, integration, cleaning, annotation, labeling, training, and inference. Despite human interventions, challenges remain. Examples include messy and incomplete data, heterogeneity and isolation, insufficient labels in training data, feature construction with limited data, and trust and transparency.

2. Methods

2.1. Research questions

We seek to examine:

• What is the overview of the reviewed studies?
• What is the Entity-relationship summary and findings of the reviewed studies?

2.2. Theoretical framework

In this work, we employ the basic structure of the Entity-Relationship or ER framework, which originated in database and software design. It may also help study interactions of humans with AI in a more structured manner. The ER framework comprises of:

3. Results

3.1. Overview of studies

3.1.1. Educational purposes

Sixteen out of twenty-eight studies use humans in the loop for educational purposes.

• Five studies facilitate assessment. These studies present AI-supported assessment to replace and advance manual assessment, with expert teachers reviewing AI's assessment of multiple-choice and open-ended questions.
• Eight studies focus on personalized learning. These studies examine the shift from curriculum to student-centered learning using AI technology for personalized learning.
• Three studies focus on improving teachers' competencies. These examine the role of AI that uses deep-learning and natural language processing to educate novice teachers.

3.2. Entity-relationship (ER) analysis

3.2.1. Overall characteristics

The human entities, as expected, often take on a learner entity in the reviewed studies. There are also more general entities such as the user. Each study most often tends to have two entities, namely a human and an AI. The minimum number of entities is two and the maximum is three in the reviewed studies.

Besides one study, the relationship between humans and AI in every study is just one and is one-directional. That is the relationship and data collected either flowing from the human to the AI entity or vice versa.

4. Discussion

4.1. Summary of findings

Of the 28 studies, 16 use human-in-the-loop for education and 12 for industry purposes. Personalized learning, and assessment, followed by teachers' competencies are the most common themes of educational studies conducted.

4.2. Human-in-the-loop and AI challenges and future considerations

Key challenges include:

• Having AI understand and account for unclear speech in noisy environments
• Considering involuntary and indirect feedback mechanisms
• Managing self-motivated learning setups for students
• Addressing student acceptance of AI assessment
• Handling bias in AI systems and explainability for non-technical audiences
• Balancing customization with equitable educational experiences
• Managing complexity in programming and scaling solutions

4.3. Implications

4.3.1. Is AI entity better off to be weak or strong?

The use of the ER framework helped us understand that an AI entity is often dependent on the human for one or a combination of source data/analysis or validation and is considered a weak entity. We find that the term "weak" may not be necessarily a bad thing concerning AI.

4.3.2. Cardinality effect on AI's precision and generalizability

Cardinality may take on different forms. Each form may in turn pose implications for the development of AI as an entity:

• One-to-one: AI designed for a specific learner or group of learners
• One-to-many: One type of AI entity programmed to handle many learners
• Many-to-one: Many AI entities that relate to a learner
• Many-to-many: Both learners and AI who have multiples of each other

5. Concluding remarks

The goal of this systematic review was to shed light on the configurations of human-in-the-loop and AI in education. A review of 28 studies is conducted and an overview of studies, Entity Relationship (ER) configurations, and analysis of ER is presented. The dominant number of two or three-entity studies, one-sided relationships, few attributes, and many to many cardinalities may signal a lack of deliberation on beings that come to interact and influence human-in-the-loop and AI in education.

Abstract

The present discussion examines the transformative impact of Artificial Intelligence (AI) in educational settings, focusing on the necessity for AI literacy, prompt engineering proficiency, and enhanced critical thinking skills. The introduction of AI into education marks a significant departure from conventional teaching methods, offering personalized learning and support for diverse educational requirements, including students with special needs.

However, this integration presents challenges, including the need for comprehensive educator training and curriculum adaptation to align with societal structures. AI literacy is identified as crucial, encompassing an understanding of AI technologies and their broader societal impacts. Prompt engineering is highlighted as a key skill for eliciting specific responses from AI systems, thereby enriching educational experiences and promoting critical thinking.

There is detailed analysis of strategies for embedding these skills within educational curricula and pedagogical practices. This is discussed through a case-study based on a Swiss university and a narrative literature review, followed by practical suggestions of how to implement AI in the classroom.

Introduction

In the evolving landscape of education, the integration of Artificial Intelligence (AI) represents a transformative shift, stipulating a new era in learning and teaching methodologies. This article delves into the multifaceted role of AI in the classroom, focusing particularly on the primacy of prompt engineering, AI literacy, and the cultivation of critical thinking skills.

The advent of AI in educational settings transcends mere technological advancement, reshaping the educational experience at its core. AI's role extends beyond traditional teaching methods, offering personalized learning experiences and supporting a diverse range of educational needs. It enhances educational processes, developing essential skills such as computational and critical thinking, intricately linked to machine learning and educational robotics.

Key Challenges and Downsides of AI in Education

The many precious possibilities in positively transforming the education systems through AI also comes with some downsides:

• Teachers feeling overwhelmed because they do not have much knowledge of the technology and how it could best be used.
• Both teachers and students not being aware of the limitations and dangers of the technology (i.e. generating false responses through AI hallucinations).
• Students uncritically using the technology and handing over the necessary cognitive work to the machine.
• Students not seeking to learn new materials for themselves but instead wanting to minimize their efforts.
• Inherent technical problems that exacerbate malignant conditions, such as GPT-3, GPT-3.5 and GPT-4 mirroring math anxiety in students.

Three Essential Skills for AI-Enabled Education

To remedy these problems, there are three necessary skills that can address these challenges:

1. AI Literacy

AI literacy consists of several sub-skills:

• Architecture: Understanding the basic architectural ideas underlying Artificial Neural Networks (only on a basic need-to-know basis). This should primarily entail the knowledge that such systems are nothing more than purely statistical models.
• Limitations: Understanding what these models are good for and where they fail. Most poignantly, students and teachers should understand that such statistical models are not truth-generators but effective data processors.
• Problem Landscape: Understanding where all the main problems of AI systems lie, due to the fact that they are only statistical machines and not truth-generators.
• Applicability and Best Practices: Understanding not only the risks but also the many ways AI can be beneficially used and implemented in daily life and the context of learning.
• AI Ethics: Understanding the major AI basics, its limitations and risks, as well as potential problems and how it can be used should lead to a nuanced understanding of its ethics.

2. Prompt Engineering

Prompt engineering involves the strategic crafting of inputs to elicit desired responses or behaviors from AI systems. In educational settings, this translates to designing prompts that not only engage students but also challenge them to think critically and creatively.

Key Prompting Methods:

• Input-Output Prompting (IOP): The classic form of prompting: simple input, simple output
• Chain-of-Thought Prompting (CoT): The AI should slowly elaborate on how a given response is generated
• Role-Play or Expert-Prompting (EP): The AI should assume the role of a person or an expert before providing an answer
• Self-Consistency Prompting (SC): The AI should generate several responses and discern itself, which would be the best answer
• Generated Knowledge Prompting (GKn): Before prompting the AI with our actual task, we first let the model generate knowledge about the topic
• Tree-of-Thought Prompting (ToT): The AI is provided with a complex setting where it is prompted to use its arguments like a chess game

3. Critical Thinking with AI

Critical thinking, in the context of AI education, involves the ability to analyze information, evaluate different perspectives, and create reasoned arguments, all within the framework of AI-driven environments.

Teaching Scaffolding Methods:

• Prompt scaffolding: The teacher provides helpful context or hints and also asks specific questions to lead students on the path to better understand and transpire a topic
• Explicit reflection: The teacher helps students to think through certain scenarios and where the potential pitfalls lie
• Praise and feedback: The teacher provides acknowledgments where good work has been done and gives a qualitative review on how the student is doing
• Modifying activity: The teacher suggests alternative strategies how students can beneficially work with AI, thereby fostering responsible use
• Direct instruction: Through providing clear tasks and instructions, students learn how to navigate the digital world and how AI can be used
• Modeling: The teacher highlights examples of where students make mistakes in their proper use of digital tools and helps them where they have difficulties to interact

Case Study: Swiss Educational Institution

The paper presents a detailed case study from the Kalaidos University of Applied Sciences (KFH) in Zurich, Switzerland, which developed "AI Guidelines" for student use. The institution found a middle ground between banning AI completely and allowing unrestricted use.

Key Requirements for AI Use:

• Declaring which model was implemented
• Explaining how and why it was used
• Explaining how the responses of the AI were critically evaluated
• Highlighting which places in the manuscript the AI was used for

The guidelines emphasized using AI as a "sparring partner" rather than a tutor, teacher, or ghostwriter.

Practical Suggestions

Enhancing AI Literacy

• Create AI literacy courses covering essential AI concepts, ethical considerations, and practical applications
• Adopt an interdisciplinary approach, integrating AI literacy across various subjects
• Use specific AI tools like Teachino for curriculum development, Perplexity for knowledge retrieval, and HelloHistory for interactive teaching

Advancing Prompt Engineering

• Educate teachers and students about prompt methodologies
• Conduct collaborative sessions where students and teachers experiment with different prompts
• Create exercises for each educational module that incorporate prompt engineering
• Align exercises with learning objectives

Critical Thinking with AI

• Conduct workshops focusing on developing critical thinking skills in the context of AI use
• Use case studies to examine real-world situations where AI decisions have significant consequences
• Establish institutional channels where students and teachers can share AI-related problems and experiences
• Create a culture of AI adoption built on principles of ethical AI use, continuous learning, and critical engagement

Future Research Directions

• Curriculum Integration: Explore effective methods for integrating AI literacy across various educational levels and disciplines
• Ethical AI development: Investigate how to develop and implement AI tools that are transparent, unbiased, and respect student privacy
• AI in Policy Making: Understand how AI can assist in educational policy-making and administration
• Cultural Shifts in Education: Research how educational institutions can foster a culture of critical and ethical AI use
• Longitudinal Studies: Assess the long-term impact of AI integration on learning outcomes, teacher effectiveness, and student well-being

Conclusion

The integration of AI in education marks a transformative era that is redefining teaching and learning methodologies fundamentally. The paper argues that this can be effectively done through implementing AI literacy, prompt engineering expertise, and critical thinking skills. The future of education, augmented by AI, holds vast potential, and navigating its complexities with a focus on responsible and ethical practices will be key to realizing its full promise.