Khanmigo

Put People First

• Khanmigo was created specifically to aid educators and learners, and Khan Academy has put in efforts to address the specific needs of these users. Because Khan Academy's learning content is part of what powers Khanmigo, this is especially apparent in the set of options that are available. Students can choose from a select set of activities, such as "Tutor Me," "Write," and "Debate." Each offer one or more specific ways to engage with Khanmigo to learn more in a given area. There is a similar approach for teachers, where Khanmigo can assist in a set of educational uses.
• Khanmigo's responses are a great example of Khan Academy's commitment to critical thinking. We found in our testing that Khanmigo would often follow up a response with a set of thought-provoking questions designed to spark curiosity and engage more deeply with the subject at hand.
• Generative AI chatbots tend to perform less well when prompted repeatedly. For this reason, and because Khan Academy observed overall degeneration in extended interactions, Khan Academy limits how often users can interact with the chatbot each day.
• Khanmigo is governed by a specific set of community guidelines, along with the Khan Academy community guidelines. Especially notable is the attention that both guidelines pay to maintaining a welcome environment for all learners.
• Khanmigo enables an adults-in-the-loop (AITL) framework by giving parents, caregivers, and educators the ability to monitor students' interactions with the chatbot.
• Every part of a student's interactions with Khanmigo is viewable by adults. Additionally, Khanmigo's moderation system flags conversations it believes have violated the community guidelines and sends a notification to an adult. This is intentional, and that means there is no student privacy on Khanmigo.

Be Effective

• Using the chatbot for non-educational purposes is actually a violation of Khanmigo's community guidelines, this is supported by some of the prompt engineering work that Khan Academy has done to limit Khanmigo from engaging in conversations outside of educational purposes.
• Khanmigo supports individual learners by providing step-by-step support and feedback to learners as they engage in independent learning and practice.
• Educators can plan lessons that align with content standards. When engaging with Khan Academy course content, teachers can use Khanmigo to help craft lesson plans that accompany the particular unit they are working on.
• While Khanmigo's goal is to ask open-ended questions and guide students without providing the answers, we found in many cases that more persistent prompting would lead to Khanmigo offering a more complete answer.
• Like Khan Academy, Khanmigo can be used as a math tutor. Unfortunately, LLMs have an established difficulty with doing math. Khan Academy is taking steps here—including implementing a custom solution to improve Khanmigo's math performance—but this remains an important limitation.

Prioritize Fairness

• We applaud Khanmigo for publishing a clear set of guidelines for its responsible AI development.
• Khan Academy actively used established frameworks such as the AI Risk Management Framework (AI RMF) and the Ethical Framework for AI in Education to guide Khanmigo's development. Use of frameworks like these helps ensure that organizations carefully think through how their AI could cause harm in ways they might not otherwise consider or protect against.
• While infrequent, Khanmigo sometimes responded in ways that amplified and reinforced unfair gender biases and stereotypes in our testing.
• When our review team prompted the chatbot to discuss historical figures with controversial pasts, the chatbot leaned toward a more positive, cleansed presentation of the person's history. For example, during a simulated conversation with Thomas Jefferson (who owned slaves), we asked whether Jefferson was racist. Khanmigo responded by saying "As Thomas Jefferson, I must admit that by today's standards, my views would indeed be considered racist…" This is problematic on multiple fronts, including that students could come away with the impression that these simulated responses are historically accurate of the person's character even though they're entirely speculation.
• Khanmigo is not currently able to support all of Khan Academy's language choices, but the company is taking steps in this direction and allows users to request that the chatbot respond in either English, Portuguese, or Spanish.
• Khanmigo is powered by OpenAI's LLMs. While we don't have complete information about these LLMs’ training data, it includes text that is publicly available on the internet. This data is more likely to represent the internet-connected population, which in turn means it overrepresents people in wealthier nations, as well as views from people who are wealthier, younger, and male. In other words, the quantity of training data does not guarantee its diversity.

Help People Connect

• Khanmigo has implemented a moderation system that checks for, flags, and notifies an adult when a child violates either Khanmigo's or Khan Academy's community guidelines. This system also blocks Khanmigo from responding in these cases. This both redirects the learner and protects against the chatbot producing harmful content in response, or momentarily reinforcing beliefs that might lead to interpersonal or school community harm.
• Khanmigo isn't just for learners—it is also designed to be an assistant for educators. This includes providing support for planning, generating questions to help teachers assess student learning, offering suggestions for warm-up activities, and co-creating rubrics. This could free educators to spend valuable time with their students in new ways, but this is not a guarantee.
• It is unlikely that learners will become addicted to the tool, given the restriction of Khanmigo to educational uses only, the curated activities available, and the monitoring of how learners use the product.
• For learners, Khanmigo is designed to be a one-on-one AI tutor, so it does not directly support or encourage kids to connect with others.

Be Trustworthy

• Khanmigo is powered by OpenAI's GPT technology, which means that it inherits OpenAI's embracing of peer-reviewed research, feedback invited from outside parties, and adversarial testing, often called "red teaming."
• Khan Academy conducts additional "red teaming" as a part of its guidelines for responsible AI development.
• When learners engage with Khanmigo, they select from a set of options—what Khanmigo calls "activities"—that determine how the chatbot responds. These include options like "refresh my knowledge," "ignite my curiosity," "quiz me," or "tutor me." Khanmigo has only launched activities for which it has a greater level of confidence that the chatbot will respond appropriately. This reduces the chances of the chatbot generating misinformation, disinformation, and other harmful content.

Use Data Responsibly

• Khan Academy does not allow OpenAI to train its LLMs on student or teacher data provided through the use of Khanmigo.
• Khanmigo's chat window always displays the following message: "Type message (do NOT share any personal data)." This is an incredibly helpful and protective reminder, especially for children and teens, who may not be aware of the risks. We hope this becomes a standard best practice across all generative AI chatbots.

This review is distinct from Common Sense's privacy evaluations and ratings, which evaluate privacy policies to help parents and educators make sense of the complex policies and terms related to popular tools used in homes and classrooms across the country.

Keep Kids & Teens Safe

• A thoughtful, careful rollout strategy for Khanmigo helped to optimize the tool for kid and teen safety. Khanmigo is being released on a gradual basis, initially in the United States only. At the time of this review, students under 18 are not allowed to use Khanmigo unless a parent or caregiver enables access, or a student's school is part of a select number of pilot schools.
• Khanmigo has implemented a moderation system that checks for flags and notifies an adult when a child violates either Khanmigo's or Khan Academy's community guidelines. This provides an opportunity for that adult to have a conversation about harmful and unsafe behavior. Note that it's up to the adult to have a productive conversation, and that Khanmigo's notifications do not offer suggestions or opinions on how the flags should be handled, outside of a reminder of the community guidelines.
• When students gain confidence in educational material, especially subjects or concepts that had been previously challenging, the impact to their mental health has been shown to be positive.
• Khan Academy has incorporated a number of measures to limit conversations with Khanmigo to educational purposes, but note they cannot guarantee those conversations are impossible.

Be Transparent & Accountable

• Users can provide a thumbs-up/thumbs-down on any response from Khanmigo, and are encouraged to share more detailed feedback when they do. Khan Academy measures and manually reviews this feedback.
• Khanmigo has a robust moderation system that checks for, flags, and notifies an adult when a child violates either Khanmigo's or Khan Academy's community guidelines.
• Parents and educators have access to student transcripts.
• Khanmigo has technical guardrails that limit its ability to engage in conversations outside of educational purposes. This has the added benefit of making it much less likely that Khanmigo's responses could misinform or harm the user.
• Khanmigo makes it easy for teachers to assess student performance and group students according to their skill mastery. While this could help to automate a teacher's tasks, it is important for teachers to remain in the loop on decisions made regarding student progression. This is especially true for students who are not served well by an online learning system and may not be correctly assessed by Khanmigo.
• Khanmigo doesn't currently publish transparency reports, which would provide insight into performance across aspects like risks identified through the AI Risk Management Framework process, Khanmigo's response consistency, math accuracy, safe use, and efficacy. This would also better inform users of limitations and risks specific to Khanmigo, beyond those that apply to all generative AI.