Limitations of language models

While generative AI tools like language models can be powerful and versatile, they have significant limitations that users must understand to use them effectively. Below are key constraints, with detailed explanations and examples.

Hallucinations

Language models are trained on datasets containing information up to a specific cut-off date. This means they lack awareness of events, discoveries, or developments that occurred after that date.¹

• Good Fit: Asking about the historical significance of the Renaissance or the basics of quantum mechanics. These topics are well within the training data.
• Limitation: Asking for insights on the latest research paper published this month or for predictions about ongoing political events. The model won’t have this information unless you include it in the prompt.

For tasks requiring up-to-date information, you must provide the latest context within your query (e.g., current events, recent technology updates, or new legal policies). The model cannot spontaneously update itself with new knowledge. Otherwise, a large language model may generate outdated or irrelevant information, so-called hallucinations.

Language models may confidently present false information, especially for queries beyond their knowledge boundary. This phenomenon occurs when the model generates plausible-sounding but incorrect or misleading answers.^{2 3}

• Example: If you ask a language model about the health benefits of a fictional fruit, it may provide a detailed response based on its general knowledge of fruits, even though the fruit doesn’t exist.
• Mitigation: When using language models, be cautious with answers that seem too specific or detailed, especially in niche or fictional contexts. Cross-check the information with reliable sources to verify its accuracy.

You should always critically evaluate the outputs, especially when the AI is used for research, teaching, or publishing. If the answer seems implausible or too specific, it’s wise to verify it independently.

Reducing the likelihood of hallucinations

To mitigate hallucinations, you can provide additional context or constraints in your prompt to guide the model towards more accurate responses. Furthermore, you can use techniques like chain-of-thought prompting to encourage the AI to explain its reasoning step-by-step.⁴

However, no technique can guarantee that the model will always produce accurate information. Therefore, it’s crucial to verify the AI’s output against reliable sources, especially when dealing with factual claims or sensitive topics.

Fact-checking AI-generated outputs

Fact-checking the final output is necessary. If the case study includes any factual claims (data points, historical events, names of laws, etc.), verify each one from a reliable source. For example, if the AI narrative says “Carbon emissions in 2020 were X metric tons” or “Study Y found Z”, double-check those facts.

If the AI did not provide sources, you can do a quick web search yourself or use your subject matter expertise to validate the information. A good practice is to prompt the AI to provide sources for factual claims and then check those sources. If the AI can’t provide a source, treat the information with skepticism. However, don’t take AI-generated references at face value – always confirm.⁵ In one notorious incident, lawyers using ChatGPT got in trouble because the AI fabricated legal case citations that looked real.⁶ Examine the provided sources to ensure they are legitimate and that the AI correctly summarized them. If you’re not sure about a fact and can’t verify it, it’s safest to remove it or replace it with a more general statement.

Ensuring accuracy is crucial not only to avoid misleading students but also to maintain your credibility as an instructor.

Biased training data

Language models are trained on vast datasets sourced from the internet, books, and other materials. These datasets inevitably reflect the subtle (or not so subtle) biases present in their sources. As a result, the model’s outputs can unintentionally reinforce stereotypes, cultural biases, or inaccuracies.⁷

Examples

• Gender bias: Default assumptions about roles, e.g., associating “nurse” with women or “engineer” with men.
• Cultural bias: Overrepresentation of certain cultural perspectives while neglecting others.
• Political bias: Responses that reflect the political leanings of the data sources.

AI can inadvertently perpetuate harm when biased outputs are used uncritically. It’s essential for users to review and contextualize the responses, especially in sensitive or high-stakes scenarios.

Check for biases in AI-generated outputs

Read the output with an eye for any unintended bias or stereotype. Ensure that the output portrays individuals and groups respectfully. For example, if the AI made all scientists in the story male by default, you might choose to change some names or descriptions to be more inclusive. Check for any language that might be insensitive or triggering for your students. It’s wise to diversify names, contexts, and examples in your cases deliberately. If your output touches on a cultural or social issue, be extra cautious and perhaps have a colleague review it as well to ensure it’s presented in a balanced way or with appropriate sensitivity.

References & Footnotes