When can language models be used?

What happens if the output is incorrect?

Before using AI, consider the potential consequences of an incorrect output. If you are using AI to generate an Email to your colleague to request a meeting and the AI sounds too formal or informal given your relationship, that is a minor inconvenience but unlikely to have any meaningful consequences. However, AI is not a safe choice for high-stakes decision-making such as evaluating (and possibly failing) students. Language models can produce plausible-sounding but incorrect or misleading outputs, often referred to as hallucinations (see Limitations).

Use AI as part of an augmented decision-making process to amplify your creativity and productivity, but not as the sole source of information.

In practice, augmented decision-making means the model can help you generate options, structure your thinking, or summarize information—but a human remains responsible for the final decision and can justify it without “because the AI said so”.

Examples of safer augmentation in education:

• Brainstorming alternative lesson ideas or explanations
• Drafting rubrics, then revising them to match your learning goals
• Suggesting feedback phrasing, while you verify it against the student’s work

Examples that are typically not appropriate (or require very strict safeguards and institutional guidance):

• Automatically grading, ranking, or passing/failing students
• Using a model’s output as evidence of misconduct or plagiarism
• Making disciplinary, admission, or support decisions based on AI-generated “risk” predictions

If a tool is used anywhere in a decision pipeline, ensure there is clear accountability (who decides), transparency (what was used and how), and a way to contest the outcome.

How will you check the accuracy of the output?

AI is your assistant, not a replacement. Only use AI for tasks you could confidently solve independently, given enough time. For example, AI can quickly generate complex legal documents that seem polished and thorough. But unless you’re trained in law, you might not be able to spot errors or gaps in those documents. Thus, if you’re unfamiliar with a topic, the AI’s errors may go unnoticed, potentially misleading your students. That’s why it’s important to use AI within the limits of your own knowledge.

We have written more extensively about how to fact-check AI-generated outputs in the article on Limitations.

What data are you sharing with the AI?

Generative AI models are hosted by operators, meaning any data you provide is sent to their servers and may be processed outside your control. Avoid inputting sensitive data such as personal information, student grades, or proprietary materials (unless you have obtained informed consent).

Treat prompts as semi-public. As a rule of thumb, we recommend that you put nothing into a prompt that you would not post on your personal social media. While some platforms offer “incognito” modes or data privacy settings, you are still potentially sending confidential information to be processed by a third party.

If privacy is critical, explore running AI models locally on your computer. Open-source models such as LLaMA can offer privacy, but they require basic programming knowledge and adequate hardware.

Do you require ownership of the output?

Generative AI outputs can usually be claimed as your own,¹ provided you comply with the model’s terms of use and applicable laws. However, there are nuances to consider, particularly if the output is for commercial purposes or resembles existing copyrighted material.

• When the creative outcome is driven by predominant human intellectual activity—where a person makes free and creative choices during conception, execution, or revision—the work is generally eligible for copyright protection.² The Court of Justice of the European Union supports this position, emphasizing that existing IP laws apply when AI is used as an assistive tool.³
• On the other hand, if the AI operates autonomously with little to no human contribution to the creative output, IP protection may not apply. Minor modifications to AI-generated outputs, such as light editing, are usually insufficient to establish legal ownership.⁴
• Language models must not be used for legal infringements (e.g., copyright infringements). Asking AI to create something “in the style of [Author/Artist X]” could lead to outputs that resemble copyrighted work. In this case, it could be legally unclear who owns the result - the user, the artist or the model developer.^{5 6}
• Importantly, when an output is provided directly to users (i.e., without a human in the loop), it must be clear to users that the answers do not originate from a human being.