Context Engineering

Arvin Liu @ RayAegis

• Context Engineering vs Prompt Engineering 
• AI Agent Basics
• Prompt Engineering - How to adjust your prompt?
• Reduce Hallucination
• Strategy

Schedule

Link

Prompt v.s Context Engineering

What is the target?

What is Prompt Engineering?

Use Magic Spell

• Emotional Prompt

• Structure Preserving

What is Context Engineering?

• Sometimes an LLM fails not because of the prompt, but due to its limitations.

  • Requires tool calls (e.g., calculations, Google Workspace

  • Requires up-to-date knowledge (RAG)

Conclusion

Prompt / Context Engineering ≠ Changing model parameters

Goal: Make the model perform better through better prompts.

AI Agent Basics

AI Agent Basics - LLM Encoding

Encoders used in modern LLMs

Modern LLMs use self-attention.

• Tokens reference the other tokens in the context.
• Strong imitation and pattern-following ability.

AI Agent Basics - LLM Encoding

Decoding process:

• Generate token → Append to context → Repeat
• That's why Role prompts can guide generation.

AI Agent Basics - LLM Decoding

• Modern LLMs often use MoE (Mixture of Experts).
• A router model selects the appropriate expert.
• Clear and simple prompts may help improve routing decisions.

Mixture of Experts

AI Agent Basics - MoE

AI Agent Basics - Multi-round

• That’s why when there are too many rounds, an LLM may start to forget things.

• What you do in multi-round conversations, reasoning models also perform internally.
• Some models support CoT reasoning and self-correction.

Thinking Model

AI Agent Basics - Reasoning Model

Thinking Model

AI Agent Basics - Reasoning Model

Reasoning Model in ChatGPT

Prompt Engineering

How Anthropic, OpenAI, Google Use Prompt Engineering

Prompt Engineering Guide

1. Instruction Clarity

2. Context and Task Definition

3. Output Control

4. Guide the thinking

Prompt Engineering Guide

疊甲聲明

Instruction Clarity

Treat LLM as your colleague

1. Instruction Clarity

You

Co-worker

1. Be Explicit About the Task

2. Define the Task in Detail

3. Define Input Type

1. Instruction Clarity

• Be Explicit About the Task

  • Clearly state the required action

  • Avoid vague terms (e.g., “some”, “fairly short”)

  • Specify the output format (Markdown / XML / JSON)

1. Instruction Clarity

✅

❌

• Define the Task in Detail

  • Specific instructions – step-by-step actions

  • Explicit constraints – limits on what to do or avoid

  • Success criteria – what counts as correct

1. Instruction Clarity

✅

❌

e.g. painter example

1. Instruction Clarity

• Define Input Type

  • ​ Labeling the input (question, task, text, entity) helps the model choose the appropriate response pattern and improves consistency.

✅

❌

Context & Task Definition

Let AI Agent imitate your prompt

2. Context and Task Definition

1. Structure prompts

2. Assign a Role

3. Put instructions at the beginning

4. Provide Context

2. Context and Task Definition

❌

✅

• Structure prompts​​

  • XML tags – <instructions>, <input> to separate content clearly

  • Separators – ### or """ to distinguish instructions from context

2. Context and Task Definition

❌

✅

• Structure prompts​​

  • XML tags – <instructions>, <input> to separate content clearly

  • Separators – ### or """ to distinguish instructions from context

2. Context and Task Definition

2. Context and Task Definition

✅

• Assign a Role​

  • ​Give LLM a role in the beginning of the prompt

  • Think about decode mechanism and corpus

* KISS: Keep It Simple & Stupid

* YNGNI: You Ain't Gonna Need It

2. Context and Task Definition

• Instruction Placement

  1. Short prompts: 

     • Instructions first

     • Early tokens have more influence

  2. Long prompts: 

     • Context first, query/instructions at the end

     • This helps the model focus on the question after reading context

2. Context and Task Definition

• Instruction Placement

  1. Short prompts: instructions first

  2. Long prompts: context first, query/instructions at the end

Short Prompt

Long Prompt

2. Context and Task Definition

❌

✅

• Provide Context

  1. Explain the background or motivation behind your instructions

  2. Helps the model understand your goals

  3. Leads to more targeted and relevant responses

Output Control

How you shape the response of AI Agent?

3. Output Control

1. Using Leading Tokens

2. Use XML format indicators

3. Consistent Formatting​

3. Output Control

❌

✅

• Using Leading Tokens

  • Prefix hints guide the model toward the desired output

  • Example: AI Auto completion

3. Output Control

❌

✅

• Using XML format indicator

  • Wrap content in XML tags to guide the model’s output structure, tone, and format.

  • Use XML tags as anchors for context or task references.

3. Output Control

❌

✅

• Consistent Formatting

  • Align your prompt’s format with the desired output (e.g., Markdown → Markdown) to improve reliability, especially in few-shot examples.

Guide the thinking

How to Make an AI Agent Think Like You?

You

Co-worker

Assume you need to assign work to a very smart colleague.

• They are capable, but don’t know what you want.

• You need to guide your colleague how to complete the task.

Give me a hand to write some code

✋

???

4. Guide the thinking

4. Guide the thinking

1. Guide with Chain-of-Thought

2. Guide with Few-Shot Prompts

3. Guide with Positive-Pattern

4. Guide the thinking

• Guide with Chain-of-Thought

4. Guide the thinking

• Guide with Chain-of-Thought (CoT)

  • Hard-coded CoT – Manually break complex tasks into steps.

  • Manual CoT (fallback) – Ask the model to think step by step when automatic reasoning is off. (<think>)

  • Self-check – Ask the model to verify its reasoning and final answer before output.

4. Guide the thinking

• Guide with Chain-of-Thought (CoT)

  • Hard-coded CoT 

  • Manual CoT (fallback) 

  • Self-check 

example prompt:

4. Guide the thinking

• Guide with Few-Shot Prompts

  • Use examples effectively (Few-shot) – Provide examples that illustrate the desired pattern; explain the reasoning behind each example if possible.

  • Prompt strategy – zero-shot ➡️ few-shot ➡️ Fine-tuning

example prompt:

4. Guide the thinking

• Guide with Positive-Pattern

  • Tell AI Agent what to do instead of what not to do

❌

✅

Example Template

Let AI Agent imitate your prompt

Example Template

Anti-Overeagerness Template

Claude tends to be overly eager. Use this example to reduce that behavior.

Suggest Order

Hallucination

Don't Guess, be Honest.

Hallucination

Student

Teacher

• Due to the loss design of LLMs, the model tends to guess answers that maximize its score, even when it is uncertain.
• This behavior is a major cause of hallucination.

🧑‍🏫

🧑‍🎓

Reduce Hallucination

1. Allow “I don’t know.”

2. Use direct quotes when RAG sources are provided.

3. Self-check

Strategy

Let AI Agent imitate your prompt

Text

8. Prompt Development Strategy (How prompts evolve)

6. Prompt Strategy & Iteration

9. Advanced Prompt Architectures

10. Model Configuration (Outside the prompt itself)

Quick Start

How to use google-genai api?

1. Clear & Direct Instructions

• question
• task
• entity

Current Works

--

output payload generator

Target CVE

Final Code

• AI Agent has no sufficient information to generate.

output payload generator

Target CVE

Final Code

• The code only output a payload that doesn't change.
• Want to detect the changeable part and can replaced with other pattern. (For latter use, like training data.)
• Desired Result:

Search Target CVE and summarize

all

output pattern & payload generator

Target CVE

Final Code

• I gave 30 RAGs to generate. But AI Agent only use first RAG for generation.
• Tried lots of methods, but use the most naive way in the final. That is - Pass every RAG one by one for generation.

Search Target CVE and summarize

all

output pattern & payload generator

Target CVE

Final Code

• It's very hard to do lots of things in one interaction. Thus I tried to break down the steps.

Search Target CVE and summarize

each

1. Generate pattern

Target CVE

Final Code

• Although it'll generate new pattern in (Stage 1. Generate pattern), but it may not used in (Stage 2. Generate code),
• What's worse, Stage 2. may use the pattern not showed in Stage 1. or existing pattern.

Search Target CVE and summarize

each

2. Generate code

Pattern Example

1. Generate template

Target CVE

Final Code

• In (Stage 2. Extract pattern), we expect AI agent can do these steps:
  1. ​​Given existing pattern, try to find the new pattern.
  2. When a new pattern found, try to generate the template and add it to DB later.

Search Target CVE and summarize

each

2. Extract pattern

Pattern Example

3. Replace pattern

1. Generate template

Target CVE

Final Code

• The result of pattern and code need to be fine-tuned.
  • Pattern issue:
    • Validation: is it exist? can be replaced with existing pattern? should it be a pattern?
    • Naming ability is very bad.

Search Target CVE and summarize

each

2. Extract pattern

3. Replace pattern

2b. Generate pattern codebook

1. Generate template

Target CVE

Final Code

• The result of pattern and code need to be fine-tuned.
  • Code issue:
    • Using wrong or not existing pattern.
    • Cannot execute.
      • Wrong count punctuation (){}'"\

Search Target CVE and summarize

each

2. Extract pattern

4. Replace pattern

3b. Generate pattern codebook

3. Pattern alignment & validation

1. Generate template

Target CVE

Final Code

• Missing about the target CVE.

Search Target CVE and summarize

each

2. Extract pattern

4. Replace pattern

3b. Generate pattern codebook

3. Pattern alignment & validation

5. Code

auto-fixing

2. Generate template

Target CVE

Final Code

• That is the current workflow

Search Target CVE and summarize

each

3. Extract pattern

5. Replace pattern

4. Pattern alignment, validation, maintainance

6. Code auto-fixing

1. Summarize All RAG

all

RAG

Retrieval Augmented Generation

Lost in the Middle

Lost in the Middle: How Language Models Use Long Contexts 

1. Summary each block

* self-check

2. Call separately

Installation

generate_content

Simple Usage

• Query

• Response

• We can use                                                       to manipulate the reponse.