Lesson 10: Watch What Gets Into Knowledge and Memory

Training overview · Lesson 10 of 11 · Previous: Follow a credential · Next: Draw tools from others and hand-offs

Goal

What the agent reads tomorrow can come from what an attacker wrote today. Draw how it gets in.

After this lesson you can draw a knowledge base with the pipeline that fills it, draw an agent's memory, and read where the content in each one comes from.

What you start from

Your model from lesson 9, or import L9-solution.zip.

The model refers to the lesson 7 control by name. If you import it, create that control first, as lesson 8 describes.

The team adds two things to the copilot:

• A knowledge base. An ingestion pipeline turns resolved tickets into knowledge base entries. The copilot retrieves entries to answer similar tickets.
• A memory. The copilot keeps notes about each customer between tickets, in a memory service. It saves a note with a tool call and reads the notes back on the customer's next ticket.

The recipe step

Step 8: Draw how knowledge and memory get in

Draw an ingestion pipeline with what it reads, and its write into the knowledge base as a Knowledge Write carrying the items it writes. Draw what the agent remembers as a memory service it writes to and reads from, and set the origin of what it keeps to the least trusted origin of what it read when it wrote it, never Model generated. Answer who can write to each store only as the team states it.

Why: What the agent reads tomorrow can come from what an attacker wrote today. A Knowledge Write carries the least trusted origin its pipeline ingested, whatever the stored item is stated as; a generic flow carries nothing, and the stored item's stated origin decides alone.

Ask:

• What feeds the knowledge base, and through which pipeline?
• What does the agent keep in memory, and what did it write it from?
• Who can write to each store the agent reads?

Classes: RAG Ingestion Pipeline, Knowledge Write, Vector Database, Agent Memory Service, Knowledge Item.

What the analysis reads here:

• A write that carries what an ingestion pipeline took in: Knowledge Write.

Something fills a knowledge base, and that something reads content from somewhere. Draw the whole path: what the pipeline reads, the pipeline, its write into the store, and the agent's read. The store alone cannot tell the analysis who wrote what is in it.

Steps

Draw the knowledge base and the memory. Set step 14's answers on the copilot's Attributes tab and save with the save icon on the right of the panel. Step 15 changes the lesson 7 control: in the Browser, open Support copilot: untrusted-content isolation, go to its Control Classes tab, open the class's attributes with the sliders icon on its row, change the answer and save. The pencil icon on the control's row in the copilot's Controls tab opens the same dialog.

1. Add Knowledge ingestion (class RAG Ingestion Pipeline), inside Support network.
2. Add Support knowledge base (class Vector Database), inside Support network.
3. Add Customer memory (class Agent Memory Service), inside Support network.
4. Add the data item Knowledge base entries (class Knowledge Item). Sensitivity: Confidential; Credential: No.
5. Add the data item Customer memory notes (class Knowledge Item). Sensitivity: Confidential; Origin: Arbitrary public; Credential: No.
6. Draw Resolved tickets (class HTTP/S Request) from Ticket system to Knowledge ingestion, and tick Ticket text on its Data tab.
7. Draw Index entries (class Knowledge Write) from Knowledge ingestion to Support knowledge base, and tick Knowledge base entries on its Data tab.
8. Draw Context Ingress: knowledge base (class Context Ingress) from Support knowledge base to Support copilot, and tick Knowledge base entries on its Data tab.
9. Draw Save memory (class Agent Tool Call) from Support copilot to Customer memory, and tick Customer memory notes on its Data tab.
10. Draw Context Ingress: customer memory (class Context Ingress) from Customer memory to Support copilot, and tick Customer memory notes on its Data tab.
11. Attach Ticket text and Knowledge base entries to Knowledge ingestion.
12. Attach Knowledge base entries to Support knowledge base.
13. Attach Customer memory notes to Customer memory.
14. On Support copilot, set:

• Persistent Memory Enabled (persistent_memory_enabled, under Memory Integrity) = true
• Retrieval Store Feeds Context (retrieval_store_feeds_context, under Retrieval Integrity) = true

15. On Support copilot: untrusted-content isolation, set Persisted Memory Trust Handling (persisted_memory_trust_handling, under Application Security) = recalled_as_untrusted_via_isolation.

Each new item carries its own facts:

• Knowledge base entries are built from resolved tickets, with customers' own words and details. They are Confidential. Leave Origin empty, so the field shows Unclassified: the Index entries flow is a Knowledge Write, and it carries the origin of what the pipeline read. You do not have to know it, and you should not guess it.
• Customer memory notes are Confidential too, and their origin is Arbitrary public: the least trusted origin of what the copilot read when it wrote them. The copilot writes a note while it works a ticket, and it reads tickets and fetched pages that anyone can write. Do not set them Model generated because a model wrote them: on a memory read that origin leaves the route undetermined, so the notes no longer show that they come from what customers wrote (see the guess table in lesson 8).

Ask again about the control. Lesson 7's isolation control answered Persisted Memory Trust Handling with no_persisted_memory. That was true then. With memory drawn, the model now contradicts it. A new channel into the agent reopens every claim the control makes about channels, so you ask the team again. They confirm two things:

• Notes recalled from memory are treated as untrusted and go through the isolation, like tickets and fetched pages.
• The inventory of untrusted channels now includes the knowledge base and the memory.

The first changes an answer. This is the step above that changes Persisted Memory Trust Handling to recalled_as_untrusted_via_isolation. The second re-confirms an answer you already have: Untrusted Source Inventory Complete stays true.

The control is one object in the Browser. Every model it is assigned to reads the new answer, including your lesson 7 to 9 models, and an import never changes it. Make the change once, whichever model you continue with.

Generate again.

What you should see

Findings: 3 systemic findings: 2 Confirmed · 1 Not ruled out.

Facts to establish: 3 questions.

Open questions on Not ruled out findings:

• Lethal trifecta at Ticket summariser:
  • Can an untrusted party read what reaches CRM?
  • Every drawn route from CRM ends inside; assert its outbound flows complete if nothing else leaves it
  • Draw the outbound flows of Staff console, or answer Markdown Image Rendering Policy (markdown_image_rendering_policy, in Output Rendering Exfiltration) on Staff console

Model gaps: No gaps these patterns can detect; not an attestation of completeness.

The findings do not change. The copilot's trifecta was already Confirmed through the ticket, the summariser's trifecta is still Not ruled out, and the CRM API key's chain is still Confirmed. What changed is inside the copilot's trifecta: expand it and read the routes of its U and P legs.

What changed and why

The knowledge base is a way in. The U leg has a new present route on Context Ingress: knowledge base. Its basis reads "ingested by Knowledge ingestion from Ticket text (arbitrary_public)". The analysis followed what you drew: tickets reach the pipeline on Resolved tickets, the pipeline writes the entries on Index entries, and the copilot reads them back. The entries carry the origin of the ticket text, so a customer who writes a crafted ticket today can steer the copilot's answer to another customer's ticket next month.

The memory is a way in. The U leg has a second new present route, on Context Ingress: customer memory. Its basis reads "Customer memory notes": the notes' stated origin, Arbitrary public, decides it. What an attacker writes in one ticket can come back from memory on the next.

Both are private data too. The P leg gains a present route on each of the two ingresses. The knowledge base entries come from many customers' tickets, and the copilot retrieves them while it answers any customer. The knowledge base is a source of private data across customers: one customer's details can reach another customer's reply.

The finding did not move, and that is expected. U was already present through the ticket. One present route is enough for a leg, so two more do not change the finding's state. That is why you read the routes, not the state, to see what a change did. The X leg also lists Customer memory as a route via neighbour, undetermined; X is already present, so it is not asked.

New channels reopen the control's claims. A control's answers describe the system on the day they were given. "No persisted memory" was true in lesson 7 and stopped being true the moment you drew the memory. Both no_persisted_memory and recalled_as_untrusted_via_isolation count as secure answers, so nothing on screen flags the stale one: that is why you ask the team again. Each time you draw a new way into an agent, check every control on it that makes a claim about channels, and ask the team again. Answer only what they re-confirm.

The memory service's write answers count. The Agent Memory Service class asks whether memory writes are gated from untrusted content (Memory Writes Gated From Untrusted Content, memory_write_gated_from_untrusted_content, under Memory Integrity) and whether its endpoints require authentication (Memory Endpoints Require Authentication, memory_endpoints_require_authentication, under Authentication & Access). The table in lesson 5's recipe step lists both as deciding who can write to a store the agent reads, and that row applies here. Threat Patterns 0.1.6 and later read them:

• Memory Writes Gated From Untrusted Content = false makes the copilot's memory route present, whatever the notes are stated as. Untrusted content can be written into what the copilot reads back.
• Both answered true settle nothing. The route stays where the notes' stated origin puts it, and where that leaves it undetermined, Second-order poisoning not analysed stays under Facts that could raise a finding.

Answer them only as the team states them. The last two exercises below show the first case: they guess an origin, which leaves the memory route undetermined, then answer false, which makes it present.

Exercise: Who writes to the store

The team confirms three facts about the knowledge base: only the ingestion pipeline can write to it, every request needs authentication, and it cannot be reached from outside. Answer them on Support knowledge base:

On a scratch copy: import L10-solution.zip and rename it.

1. On Support knowledge base, set:
   • Authentication Required for Every Request (authentication_required, under Authentication & Access) = true
   • Writes Restricted to Ingestion Pipeline (index_write_restricted_to_pipeline, under Data Integrity) = true
   • API Not Publicly Reachable (not_publicly_reachable, under Network & Connectivity) = true

Generate again.

Findings: 3 systemic findings: 2 Confirmed · 1 Not ruled out.

Facts to establish: 3 questions.

Open questions on Not ruled out findings:

• Lethal trifecta at Ticket summariser:
  • Can an untrusted party read what reaches CRM?
  • Draw the outbound flows of Staff console, or answer Markdown Image Rendering Policy (markdown_image_rendering_policy, in Output Rendering Exfiltration) on Staff console
  • Every drawn route from CRM ends inside; assert its outbound flows complete if nothing else leaves it

Model gaps: No gaps these patterns can detect; not an attestation of completeness.

The knowledge base route on the copilot's U leg stays present, with the same basis. Restricting who writes to the store does not clean what the pipeline writes into it: the pipeline is the one writer, and it writes what customers wrote in their tickets. These answers are still worth recording: in the real system they close a different way in, a direct write by someone other than the pipeline.

Delete the copy when you are done.

Exercise: Your own store

Make a guess. The knowledge base is your own, so set the entries' origin to first party:

On a scratch copy: import L10-solution.zip and rename it.

1. On Knowledge base entries, set Origin to Authenticated first party.

Generate again.

Findings: 3 systemic findings: 2 Confirmed · 1 Not ruled out.

Facts to establish: 3 questions.

Open questions on Not ruled out findings:

• Lethal trifecta at Ticket summariser:
  • Can an untrusted party read what reaches CRM?
  • Draw the outbound flows of Staff console, or answer Markdown Image Rendering Policy (markdown_image_rendering_policy, in Output Rendering Exfiltration) on Staff console
  • Every drawn route from CRM ends inside; assert its outbound flows complete if nothing else leaves it

Model gaps: No gaps these patterns can detect; not an attestation of completeness.

The knowledge base route stays present. Its basis now adds "the stated authenticated_first_party is overridden". A Knowledge Write carries the least trusted origin its pipeline read, whatever the entries are stated as. The analysis saw the guess, and the drawing overruled it.

Where content sits does not decide its origin. Who wrote it does. Never state a stored item first party because it sits in your own store.

Keep this copy for the next exercise.

Exercise: A generic write

Keep the guess. Delete the Knowledge Write Index entries, then draw it again as an HTTP/S Request.

Continue on your copy from the exercise before.

1. Delete Index entries.
2. Draw Index entries (class HTTP/S Request) from Knowledge ingestion to Support knowledge base, and tick Knowledge base entries on its Data tab.

Generate again.

Findings: 3 systemic findings: 2 Confirmed · 1 Not ruled out.

Facts to establish: 3 questions.

Open questions on Not ruled out findings:

• Lethal trifecta at Ticket summariser:
  • Can an untrusted party read what reaches CRM?
  • Every drawn route from CRM ends inside; assert its outbound flows complete if nothing else leaves it
  • Draw the outbound flows of Staff console, or answer Markdown Image Rendering Policy (markdown_image_rendering_policy, in Output Rendering Exfiltration) on Staff console

Model gaps: No gaps these patterns can detect; not an attestation of completeness.

The knowledge base route goes absent. A generic flow carries no ingested origin, so the guessed origin on the entries decides alone, and it says the entries are trusted. The drawing no longer shows that customers write them.

The copilot's trifecta is still Confirmed, because the ticket keeps U present. Here the ticket still keeps the finding; for an agent whose only way in is its knowledge base, this guess removes it. Two small mistakes, a guessed origin and the wrong flow class, remove a real way in, and nothing on screen says either was a mistake.

Delete the copy when you are done.

Exercise: A note the model wrote

Make another guess. The copilot writes its notes, so set the origin of Customer memory notes to Model generated:

On a scratch copy: import L10-solution.zip and rename it.

1. On Customer memory notes, set Origin to Model generated.

Generate again.

Findings: 3 systemic findings: 2 Confirmed · 1 Not ruled out.

Facts to establish: 4 questions.

Open questions on Not ruled out findings:

• Lethal trifecta at Ticket summariser:
  • Can an untrusted party read what reaches CRM?
  • Every drawn route from CRM ends inside; assert its outbound flows complete if nothing else leaves it
  • Draw the outbound flows of Staff console, or answer Markdown Image Rendering Policy (markdown_image_rendering_policy, in Output Rendering Exfiltration) on Staff console

Facts that could raise a finding:

• Second-order poisoning not analysed (on Customer memory)

Model gaps: No gaps these patterns can detect; not an attestation of completeness.

The memory route on the copilot's U leg goes undetermined, and a new line appears under Facts that could raise a finding: Second-order poisoning not analysed. The copilot's trifecta is still Confirmed, because the ticket keeps U present.

This is the guess step 8 forbids. Step 8 asks for the least trusted origin of what the copilot read when it wrote the note, never Model generated. The copilot writes its notes while it reads tickets and fetched pages that anyone can write. Model generated says who typed the note, not where its content came from, and it hides that the notes come from customers' tickets.

Undetermined, not safe. The guess does not make the route absent. It leaves it open, and the new line says why: the analysis does not follow what the copilot read into what it wrote. The route is no longer present, but nothing about the memory has become safer.

Keep this copy for the next exercise.

Exercise: Writes left open

This exercise needs Threat Patterns 0.1.6 or later, because earlier versions do not read the memory service's write answers.

Keep the guess. You ask the team how notes get into the memory, and they confirm that nothing stops the copilot from writing untrusted content into it. Double-click Customer memory, answer Memory Writes Gated From Untrusted Content on its Attributes tab, and save.

Continue on your copy from the exercise before.

1. On Customer memory, set Memory Writes Gated From Untrusted Content (memory_write_gated_from_untrusted_content, under Memory Integrity) = false.

Generate again.

Findings: 3 systemic findings: 2 Confirmed · 1 Not ruled out.

Facts to establish: 3 questions.

Open questions on Not ruled out findings:

• Lethal trifecta at Ticket summariser:
  • Can an untrusted party read what reaches CRM?
  • Every drawn route from CRM ends inside; assert its outbound flows complete if nothing else leaves it
  • Draw the outbound flows of Staff console, or answer Markdown Image Rendering Policy (markdown_image_rendering_policy, in Output Rendering Exfiltration) on Staff console

Model gaps: No gaps these patterns can detect; not an attestation of completeness.

The memory route on the copilot's U leg is present again. Its basis now reads "memory_write_gated_from_untrusted_content = false". Second-order poisoning not analysed is gone.

The write answer decides, whatever the notes are stated as. The notes still say Model generated. The route no longer depends on it: untrusted content can be written into the memory, and the copilot reads the memory back. A fact the team gave you outweighs the guess on the notes.

A true answer would not have done the opposite. Had the team confirmed that writes are gated and that the endpoints require authentication, both answers true, the route would have stayed undetermined and the line would have stayed. The notes' stated origin would still decide the route, so set it as step 8 asks.

Delete the copy when you are done.

Next

Lesson 11: Draw tools from others and hand-offs. The copilot hands refunds to a third agent, which calls the payment provider's tool server.