Use delimiters and special tokens

Think of delimiters and tags as the drawers in a filing cabinet, each holding specific types of information. When you label each drawer clearly—like “Instructions,” “Example,” or “Output”—it’s easier to find what you need, and there’s no confusion about where things go. In prompt engineering, delimiters act the same way, “filing” different parts of the prompt so the AI can “pull out” each section without mixing it up.

Delimiters and special tokens (like XML tags) are powerful tools in prompt engineering for structuring input and making it easier for AI to understand and respond accurately to complex instructions. By enclosing information within clear boundaries, delimiters and tags allow you to specify instructions, set sections of content apart, or define formats for structured responses.

What are Delimiters and Special Tokens?

Delimiters are symbols or characters—such as quotation marks, brackets, or XML-like tags—that are used to separate different parts of text. In prompt engineering, they are often employed to highlight specific instructions, distinguish between multiple sections, or mark particular content types (like input/output or questions/answers).

For instance, enclosing a sentence in quotation marks (“…”) signals to the AI that this is a distinct part of the prompt, while XML tags like <instruction>...</instruction> can mark off instructions, making the purpose of each section clear and reducing ambiguity.

Benefits of Using Delimiters

• Improved Clarity: Delimiters make instructions stand out, helping the model differentiate between different parts of the prompt.
• Precise Structure: By defining clear sections, you can specify the format of the output, especially useful for tasks that require a structured response (e.g., lists, tables).
• Reduced Misinterpretation: When each part of the prompt is marked clearly, the AI is less likely to mix up or ignore details, improving accuracy.

How to Use Delimiters and Special Tokens

• Separate Instructions from Content: Use delimiters to keep instructions distinct from the main content, ensuring the model understands which sections to treat as directives versus response material.
• Define Input and Output Sections: For prompts that involve multiple inputs or outputs, use tags to label each section. This helps the AI process each part as intended, especially when dealing with multiple questions or data points.