Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI

발표자: Daniel Chalef (Zep AI)
원본: https://www.youtube.com/watch?v=H7puB0RwJMM
업로드일: 2026-07-23

1계층 — 핵심 한 줄 요약

LLM이 여러 소스를 종합해 생성한 지식(요약, 팩트, 구조화된 기록)은 원본에 그대로 존재하지 않는 경우가 많고, 이런 "종합" 과정에서 출처와 생성 근거를 잃어버리기 때문에, 대규모 에이전트 메모리 시스템에서는 출처(provenance)와 계보(lineage)를 데이터 구조 자체, 즉 지식 그래프에 내장해야 한다.

2계층 — 20줄 요약

1. Daniel Chalef는 LLM이 여러 소스의 데이터를 끌어와 요약·추출·구조화하는 능력은 뛰어나지만, 이 과정이 비결정론적이라고 말한다.
2. LLM이 만든 결과물은 원본 입력에 그대로 등장하지 않는 문장이나 팩트를 포함할 수 있다.
3. 이런 종합(synthesis)은 출력물이 어떻게 만들어졌는지를 보여주는 paper trail을 파괴한다.
4. 본 강연의 주제는 provenance로, 아티팩트가 어떻게·왜 생성되었는지를 추적하는 것이다.
5. Provenance는 법률적 컴플라이언스뿐 아니라, 디버깅, 신뢰 판단, 삭제 대상 결정에도 유용하다.
6. Chalef의 팀은 오픈소스 Graffiti라는 temporal graph 프레임워크와 기업용 에이전트 메모리 플랫폼 Zep을 만들었고, Zep은 Graffiti 위에 구축되어 있다.
7. 고객은 채팅, 음성, 이메일, 비즈니스 데이터 등 여러 접점에서 컨텍스트를 얻는데, 이들은 모두 출처 추적 문제로 어려움을 겪었다.
8. 강연자는 의료 시나리오를 예로 든다: 에이전트가 환자에게 페니실린 알레르기가 있다는 팩트를 보고할 때, 이는 EHR, PDF 검사 결과, AI 접수 챗봇 기록 세 가지에서 종합된 것이다.
9. 에이전트가 팩트의 출처를 명확히 보여주지 않으면, 원래 환자가 직접 입력한 불확실한 정보가 의사를 오도할 수 있다.
10. 단순히 "팩트에 source ID만 붙이는" 방식은 데이터 웨어하우스에서는 잘 작동하지만, LLM 기반 컨텍스트 파이프라인에서는 깨진다.
11. LLM 파이프라인에서는 여러 소스가 합쳐지고, 엔티티가 중복 제거되며, 새로운 데이터가 기존 팩트를 무효화하기 때문에 계보가 계속 변해야 한다.
12. 출처와 파생물 사이의 링크 집합은 그래프의 관계로 모델링할 수 있다.
13. Graffiti에서는 원본 데이터(에피소드)에서 추출된 엔티티와 팩트(엣지)가 그래프 트리플로 표현되며, 팩트를 출처로 추적하는 것은 그래프 traversal이다.
14. 엔티티 병합 시 두 엔티티의 모든 소스 링크를 보존해야 하고, 그렇지 않으면 계보가 조용히 사라진다.
15. 새로운 데이터가 기존 팩트를 반박할 때, Graffiti는 무효화된 엣지에 invalid date를 기록하고 원인이 된 에피소드를 남긴다.
16. 메타데이터 투영(metadata projection)으로 에피소드에 태그(예: EHR)를 붙이면, 이 태그가 파생된 모든 엔티티와 팩트에 상속되어 신뢰 기준에 따라 필터링할 수 있다.
17. 다만 하나의 팩트가 여러 부모를 가질 때, 에이전트가 비즈니스 규칙을 적용해야 한다: 알레르기는 모든 부모가 검증되어야 하지만, 수술 동의는 모든 부모가 검증되어야 한다.
18. 데이터 보존 정책이나 잊혀질 권리 요청으로 일부 소스만 삭제할 때, Graffiti는 남은 에피소드가 지지하는 팩트만 살리고, 삭제된 소스에만 의존한 팩트는 제거한다.
19. Provenance를 그래프가 아닌 사후 로그로 기록하면 규모가 커질수록 관리가 불가능하므로, 데이터 구조 자체가 그래프여야 한다.
20. Graffiti는 전체 소스를 원문 그대로 보관하고, 모든 파생물을 소스에 다시 연결함으로써 컴플라이언스, 검증, 디버깅, 삭제를 동시에 지원한다.

3계층 — 상세 내용

1. 왜 LLM 출력의 출처가 중요한가

LLM은 여러 문서·대화·데이터 소스를 받아 하나의 요약이나 추출된 팩트, 구조화된 기록을 만든다. 이 결과는 원본 어디에도 똑같이 존재하지 않을 수 있다. 즉, 모델이 새로운 텍스트를 생성하면서 원래 소스와의 연결을 끊어버린다. 이는 다음 문제를 낳는다.

• 법률 및 규제: 데이터 출처와 처리 이력을 요구하는 규정(GDPR, HIPAA 등)을 충족해야 한다.
• 신뢰 평가: 이 팩트를 믿을 수 있는가? 어떤 소스에서 나왔는가?
• 디버깅: 에이전트가 왜 그런 결정을 내렸는지 설명해야 한다.
• 삭제: 원본 데이터가 삭제되거나 잊혀질 권리 요청이 들어왔을 때, 파생된 모든 컨텍스트를 정확히 처리해야 한다.

Chalef는 이를 provenance 문제로 규정한다: “아티팩트가 어떻게 만들어졌고, 왜 만들어졌는지 추적하는 것.”

2. Zep과 Graffiti의 배경

Zep은 기업용 에이전트 메모리 인프라이며, Graffiti는 Zep이 기반으로 하는 오픈소스 temporal graph 프레임워크이다. 고객은 채팅뿐 아니라 음성 녹취록, 이메일, 비즈니스 데이터에서 컨텍스트를 끌어오는데, 이 모든 데이터가 provenance 추적을 요구한다. Chalef는 이 문제를 해결하기 위해 Graffiti에서 다음을 가능하게 했다.

• 원본 데이터를 그래프의 에피소드(episode) 노드로 보관.
• 에피소드에서 추출한 엔티티와 팩트를 **그래프 트리플(subject–predicate–object)**로 표현.
• 팩트에서 출처로의 이동을 **그래프 탐색(graph walk)**으로 단순화.

3. 의료 시나리오: 출처가 없으면 위험하다

강연의 핵심 예시는 다음과 같다.

팩트: “환자가 페니실린 알레르기가 있다.” 이 팩트는 세 소스에서 종합되었다:

1. 긴 전자건강기록(EHR) 텍스트
2. PDF 검사 결과
3. 환자가 AI 접수 챗봇에 입력한 내용

에이전트가 의사에게 단순히 “페니실린 알레르기”만 보고하면, 의사는 이것이 임상적으로 검증된 사실인지, 환자가 직접 말한 추정 정보인지 구분할 수 없다. 이는 치명적인 오류를 유발할 수 있다. 따라서 에이전트는 팩트뿐 아니라 그 팩트의 출처와 신뢰 수준을 함께 노출해야 한다.

4. 왜 source ID만 붙이는 방식으로는 부족한가

전통적인 데이터 웨어하우스나 데이터 레이크에서는 팩트에 source ID를 붙이는 것으로 충분하다. 하지만 LLM 기반 파이프라인에서는 다음 네 가지 이유로 실패한다.

1. 다중 소스 종합: 하나의 팩트가 여러 소스에서 나온다. 어느 소스를 택할 것인가?
2. 엔티티 병합: “J. Smith”와 “John Smith”가 같은 사람으로 병합될 때, 병합된 엔티티는 양쪽 소스 링크를 모두 보존해야 한다. 하나라도 빠지면 계보가 손실된다.
3. 시간에 따른 무효화: 새로운 데이터가 들어오면 기존 팩트가 틀려질 수 있다. 예를 들어 “Daniel이 아디다스 신발을 좋아한다”는 팩트가, 나중에 신발이 불량이었다는 불만 메시지로 무효화될 수 있다.
4. 변화하는 저장소: append-only 로그는 대규모에서 관리가 어렵고, 엔티티와 팩트의 변화를 직접 모델링하지 못한다.

결론적으로 lineage는 정적 포인터가 아니라 진화하는 링크 집합이어야 하며, 변형(mutation)을 견뎌야 한다.

5. Graffiti의 그래프 모델

Graffiti에서는 원본 데이터와 파생 데이터를 하나의 그래프로 모델링한다.

• Episode 노드: 원본 데이터(대화, 문서, 이메일, 검사 결과 등)를 그대로 저장.
• Entity 노드: 에피소드에서 추출된 개체(환자, 페니실린, Daniel, 아디다스 등).
• Edge / Fact: 엔티티 간 관계. 예: (환자) — has allergy — (페니실린).

이 그래프 트리플은 다시 자연어 팩트로 hydrate될 수 있다. 따라서 팩트의 출처를 추적하는 것은 단순한 그래프 순회이다.

6. 변화에 강한 계보: 병합과 무효화

엔티티 병합: 두 엔티티가 동일인으로 판명되면 병합된 엔티티는 양쪽의 모든 source episodes 링크를 보존해야 한다. 그렇지 않으면 팩트의 출처가 조용히 사라진다.

팩트 무효화: 새로운 데이터가 기존 팩트를 반박하면, Graffiti는 기존 엣지에 invalid date를 추가하고, 무효화를 일으킨 에피소드들을 기록한다. 예를 들어 Daniel이 아디다스 신발을 좋아한다는 팩트가, 나중에 신발 반품 및 불만 메시지로 무효화되면, 그 변화 이력이 그래프에 남는다.

이러한 구조는 lineage를 데이터 구조에 내장하는 것이며, 사후 로그가 아니다.

7. 메타데이터 투영과 비즈니스 규칙

Graffiti는 metadata projection을 통해 태그를 그래프로 전파할 수 있다. 예를 들어 EHR에서 유래한 에피소드에 “EHR” 태그를 붙이면, 이 에피소드에서 파생된 엔티티와 팩트도 그 태그를 상속한다. 이를 통해 “검증된 임상 소스에서 나온 팩트만 검색” 같은 필터링이 간단해진다.

하지만 태그만으로는 충분하지 않다. 같은 모양의 팩트라도 비즈니스 규칙에 따라 다른 평가를 받는다.

• 알레르기 팩트: 하나라도 검증되지 않은 부모가 있으면 처방을 금지해야 한다. (모든 부모가 검증되어야 안전)
• 수술 동의 팩트: 하나라도 검증되지 않은 부모가 있으면 수술을 금지해야 한다. (모든 부모가 검증되어야 동의로 인정)

Graffiti는 태그와 부모를 노출하지만, 최종 결정은 에이전트가 비즈니스 규칙을 실행하여 내려야 한다. 이는 도메인별 정책이 그래프 엔진에 하드코딩되어서는 안 된다는 의미이다.

8. 삭제, 보존, 잊혀질 권리

데이터 보존 정책이나 잊혀질 권리 요청으로 일부 에피소드를 삭제해야 할 때, Graffiti는 다음 규칙을 적용한다.

• 팩트가 삭제된 에피소드에만 의존하고 있었다면, 해당 팩트도 삭제한다.
• 팩트가 남은 에피소드에 의해 계속 지지된다면, 팩트는 살아남는다.

예시에서 환자의 AI 접수 챗봇 데이터를 삭제하면, 페니실린 알레르기 팩트는 EHR과 PDF라는 두 다른 부모가 있어서 살아남지만, 접수 챗봇에만 의존하던 연락처 선호 팩트는 삭제된다. 이는 출처 링크의 존재 덕분에 자동으로 가능하다.

9. Zep의 추출 파이프라인과 비용 최적화

Q&A에서 Chalef는 추출 파이프라인을 개략적으로 설명한다.

1. Single-shot extraction: LLM이 한 번에 엔티티, 관계, 후보 팩트를 추출한다. 팩트는 subject–verb–object 구조로 표현된다.
2. De-duplication & de-confliction: 기존 그래프에 같은 엔티티가 있으면 병합하고, 새 팩트가 기존 팩트를 반박하면 무효화 처리한다.
3. Reflection: 추출된 내용의 정확성을 검증하고, 왜 팩트가 변화했는지 더 풍부한 계보를 남긴다.

이 파이프라인은 LLM을 많이 사용하지만, 중복 제거와 같은 부분에는 전통적인 정보 검색, NLP, 엔트로피, sim hash 같은 기법을 사용하여 비용과 지연 시간을 줄이고 결정론성을 높인다. Graph construction는 본질적으로 비싸기 때문에, Zep은 이 부분을 최적화하는 데 큰 노력을 들였다고 한다.

10. 파일 기반 메모리의 한계

Q&A에서 Chalef는 markdown, wiki, 파일 기반 메모리 시스템에 대해 비판한다.

• 파일 안의 개별 줄이 변경될 때, 그 변경이 어떤 소스 때문에 일어났는지 추적하기 어렵다.
• 다중 에이전트, 다중 사용자, 다중 소스 환경에서 파일 기반 시스템은 확장에 취약하다.
• 데스크톱이나 단일 사용자/단일 에이전트 시나리오에서는 유용할 수 있지만, 기업 환경의 provenance 문제를 해결하기에는 부족하다.

그는 Zep/Graffiti의 접근법이 바로 이런 파일 기반 메모리의 한계를 극복하기 위한 것이라고 강조한다.

11. 질의응답 핵심

• 50개 이상의 엣지에서 가중치 변화: Zep은 일부 tracking을 별도 데이터 구조로 하며, 그래프 자체만으로 모든 provenance를 담지는 않는다.
• 엔티티와 엣지 타입 생성: LLM이 추출하며, vector similarity search, full-text search, BFS 등을 통해 에이전트가 그래프를 탐색할 수 있다.
• Temporal 특성: Graffiti는 시간에 따른 팩트의 유효/무효를 명시적으로 모델링한다.
• 파일 기반 메모리와의 비교: markdown은 provenance에 취약하다는 인상을 주었다.

4계층 — 인사이트와 실무 적용

1. Provenance는 옵션이 아닌 핵심 요구사항이다. 특히 의료, 금융, 법률, 기업 내부 지식 관리에서 LLM이 생성한 팩트의 출처를 알 수 없으면 책임 소재와 신뢰성이 무너진다.
2. Lineage를 데이터 구조에 내장하라. 로그를 사후에 남기는 방식은 대규모에서 실패한다. 원본 데이터를 노드로, 파생 관계를 엣지로 모델링하는 그래프 구조가 본질적 해결책이다.
3. 메타데이터는 전파되어야 한다. 태그, 신뢰 수준, 데이터 소스 유형은 상위 에피소드에서 파생된 엔티티·팩트로 상속되어야 필터링과 정책 적용이 쉬워진다.
4. 비즈니스 규칙은 에이전트가 실행한다. 그래프는 링크와 태그를 노출할 뿐, “이 팩트를 신뢰할 수 있는가”와 같은 최종 판단은 도메인 규칙에 따라 에이전트가 내려야 한다.
5. 삭제와 무효화를 자동화하라. 원본 데이터 삭제나 새로운 반박 데이터가 들어왔을 때, 파생 팩트의 생존/삭제를 링크 기반으로 자동 결정할 수 있어야 규제 요구를 충족한다.
6. 비용과 지연 시간을 전통 기법으로 줄여라. 그래프 생성은 비싸다. LLM을 모든 단계에 쓰기보다는 중복 제거, 전통 NLP, sim hash 등으로 결정론적이고 저렴한 부분을 분리해야 운영 가능하다.
7. 파일 기반 메모리는 제한적이다. 단일 사용자 또는 단순한 시나리오에는 적합하지만, 기업级 에이전트 메모리에서 provenance를 요구하면 그래프 기반 메모리가 필요하다.
8. Zep/Graffiti의 교훈: 에이전트 메모리는 단순한 검색/요약이 아니라, 시간에 따른 변화, 출처, 무효화를 지원하는 구조화된 지식 그래프로 설계해야 한다.

---

부록 — 원본 자막 전문

<details> <summary>원문 자막 펼치기/접기</summary>

So, LLMs are really great at pulling So, LLMs are really great at pulling together data from many sources. Uh, but together data from many sources. Uh, but they do so non sorry, they do so they do so non sorry, they do so non-deterministically. They interpret and synthesize data They interpret and synthesize data generating a summary, an extracted fact, generating a summary, an extracted fact, uh a structured record. And this output uh a structured record. And this output artifact may not appear verbatim in the artifact may not appear verbatim in the source inputs. source inputs. Synthesis often destroys the paper trail Synthesis often destroys the paper trail of how these outputs were originated. of how these outputs were originated. And I'm going to be talking today about And I'm going to be talking today about provenence, which is tracing how an provenence, which is tracing how an artifact was built and why. Legal artifact was built and why. Legal compliance often demands provenence, but compliance often demands provenence, but it's also useful for debugging. Deciding it's also useful for debugging. Deciding which sources you trust and which which sources you trust and which artifacts to delete. And solving this at artifacts to delete. And solving this at scale presents a real engineering scale presents a real engineering challenge. If I can get my next slide going here. If I can get my next slide going here. So my team and I built Graffiti uh the So my team and I built Graffiti uh the open-source temporal graph framework and open-source temporal graph framework and Zep, our enterprise agent memory Zep, our enterprise agent memory infrastructure is built on graffiti. Our infrastructure is built on graffiti. Our customers derive context or agent memory customers derive context or agent memory for many user touch points. Those could for many user touch points. Those could be chat, but not only chat. Often it's be chat, but not only chat. Often it's voice transcripts, email, business data. voice transcripts, email, business data. And our customers have struggled with And our customers have struggled with provenence. Where did this fact come provenence. Where did this fact come from? What is the veracity of this fact? from? What is the veracity of this fact? And over the next few slides, I'll share And over the next few slides, I'll share how we engineered solutions to this how we engineered solutions to this problem. So here's an stylized failure mode. An So here's an stylized failure mode. An agent retrieves context about a patient. agent retrieves context about a patient. So this is a healthcare scenario. So this is a healthcare scenario. And what comes back is a clean confident And what comes back is a clean confident fact. Patient has a penicellin allergy. fact. Patient has a penicellin allergy. And the context was synthesized from And the context was synthesized from three sources. A lengthy EHR record, three sources. A lengthy EHR record, electronic health record, a PDF lab electronic health record, a PDF lab report, and something a patient typed into an AI and something a patient typed into an AI intake chat. intake chat. If the agent presents the fact to a If the agent presents the fact to a doctor in treat in a treatment scenario doctor in treat in a treatment scenario without clearly indicating the source without clearly indicating the source was from the patient themselves, it may was from the patient themselves, it may mislead the doctor. When an agent mislead the doctor. When an agent retrieves context, can we point to the retrieves context, can we point to the exact source and its veracity? exact source and its veracity? For complex agent applications, the For complex agent applications, the answer is often no. So, I can imagine you're probably So, I can imagine you're probably thinking, um, but can't we just store like a um, but can't we just store like a source ID on the fact? source ID on the fact? This can work well in structured data This can work well in structured data warehouses or data lakes. A pipeline warehouses or data lakes. A pipeline outputs one value copied or mutated outputs one value copied or mutated deterministically, and the sources are known and easily and the sources are known and easily marked. But with context pipelines run marked. But with context pipelines run by LLMs, this breaks in several ways. by LLMs, this breaks in several ways. You prompt an LLM with several sources. You prompt an LLM with several sources. Many facts are each synthesized from one Many facts are each synthesized from one or more of the sources. or more of the sources. Somebody like J. Smith and John Smith Somebody like J. Smith and John Smith are merged into a single entity, one are merged into a single entity, one identity. And John's facts are derived identity. And John's facts are derived from many different places. from many different places. So new data might invalidate old facts. So new data might invalidate old facts. The store keeps changing underneath your The store keeps changing underneath your pointer and an appendon log which often pointer and an appendon log which often might you know might come to mind here might you know might come to mind here gets very hard to manage at scale as gets very hard to manage at scale as there's so many changes occurring. So there's so many changes occurring. So lineage needs to be an evolving set and lineage needs to be an evolving set and survive mutation. So sets of links between facts and their So sets of links between facts and their sources can be modeled on a graph as sources can be modeled on a graph as relationships. So provenence in a context store So provenence in a context store containing facts is a knowledge graph. containing facts is a knowledge graph. We have three in this example we have We have three in this example we have three source data in graffiti that three source data in graffiti that turned episodes. We have two entities turned episodes. We have two entities extracted from the episodes patient and extracted from the episodes patient and penicellin and an edge between them. and an edge between them. This graph triple the two entities and This graph triple the two entities and the edge can be hydrated as a fact. the edge can be hydrated as a fact. Patient has a penicellin allergy. Patient has a penicellin allergy. Tracing a fact to its source is just a Tracing a fact to its source is just a graph walk. So it's pretty simple and easy to map So it's pretty simple and easy to map source to fact on the first right source to fact on the first right but keeping it correct while the graph but keeping it correct while the graph changes can be really hard when new data changes can be really hard when new data uh so for example when two entities uh so for example when two entities merge the merged entity needs to keep merge the merged entity needs to keep all source links from both otherwise we all source links from both otherwise we silently drop a source and we lose silently drop a source and we lose lineage. And when new data contradicts existing And when new data contradicts existing data, mutating it, we need to capture data, mutating it, we need to capture this lineage too. this lineage too. In the rightmost card, a fact is In the rightmost card, a fact is rendered invalid by new data. And in rendered invalid by new data. And in graffiti, an invalid date is added graffiti, an invalid date is added to the mutated edge. And the source to the mutated edge. And the source episodes that resulted in the edge episodes that resulted in the edge mutating are noted against the fact. So gaining graffiti, the relationship So gaining graffiti, the relationship between source data and derived between source data and derived artifacts such as facts is easily artifacts such as facts is easily modeled on the graph. modeled on the graph. With metadata projection, we can also With metadata projection, we can also model classifications that span many model classifications that span many different episodes. and facts derived different episodes. and facts derived from them. And so I'll give you an from them. And so I'll give you an example here. example here. In in the prior healthc care scenario, In in the prior healthc care scenario, episodes may originate from an EHR episodes may originate from an EHR record and have an EHR tag associate record and have an EHR tag associate associated with them, but not all associated with them, but not all records are. And so on ingestion, we tag records are. And so on ingestion, we tag the episodes with the EHR tag. All the episodes with the EHR tag. All subsequent entities and facts derived subsequent entities and facts derived from the episode inherit the tag. And so from the episode inherit the tag. And so if the agent wants to retrieve if the agent wants to retrieve only facts from verified clinical only facts from verified clinical sources, it's very simple to filter for sources, it's very simple to filter for the appropriate tag as we walk the the appropriate tag as we walk the graph. So one tagging action at ingestion So one tagging action at ingestion supports evaluating the veracity of a supports evaluating the veracity of a fact. But what if the fact is three parents or But what if the fact is three parents or more? Here we have a verified flag as more? Here we have a verified flag as our metadata and in this case one parent our metadata and in this case one parent is not verified. is not verified. So is the fact verified So is the fact verified for the allergy flag which could be a for the allergy flag which could be a life and death situation. the agent life and death situation. the agent missing it, missing that particular flag missing it, missing that particular flag could be a deadly mistake. So not could be a deadly mistake. So not retrieving the fact and any source of retrieving the fact and any source of the three should block the three should block that prescription being issued. that prescription being issued. But for something like a consent But for something like a consent on file for procedure fact, the mistake on file for procedure fact, the mistake is operating on unverified is operating on unverified consent. So the patient hasn't actually consent. So the patient hasn't actually given their consent and every parent given their consent and every parent needs to be verified. So every single needs to be verified. So every single episode should have that tag. episode should have that tag. So the facts are very similar shapes. So the facts are very similar shapes. They have three three parent episodes They have three three parent episodes but opposite policies. And here but opposite policies. And here graffiti or the underlying store exposes graffiti or the underlying store exposes that choice. It exposes which of the that choice. It exposes which of the episodes have the gra the particular episodes have the gra the particular tag, but your agent needs to execute or apply but your agent needs to execute or apply your business rules. So that's not your business rules. So that's not necessarily something that we bake into necessarily something that we bake into the graph. [snorts] the graph. [snorts] It's situational. It's situational. Another situation where lineage is really situation where lineage is really important, we may have to delete source data we may have to delete source data due to retention policies or right to be due to retention policies or right to be forgotten requests. So privacy forgotten requests. So privacy compliance and this is really tricky because if we and this is really tricky because if we have context derived from multiple have context derived from multiple sources, how do we do so? how do we do so? Mapping lineage here is really useful. Mapping lineage here is really useful. We know which facts are derived from the We know which facts are derived from the source data we intend on deleting. But source data we intend on deleting. But what if only some of the source data what if only some of the source data needs to be deleted but not all? needs to be deleted but not all? So in this example, we need to delete So in this example, we need to delete the intake chart data. So what the the intake chart data. So what the patient filled in, which is only one of patient filled in, which is only one of three source data. In graffiti's model, three source data. In graffiti's model, the allergy fact survives the deletion the allergy fact survives the deletion and that's because there are two parents and that's because there are two parents parent episode still supporting the parent episode still supporting the fact. However, the contact preference fact. However, the contact preference fact is deleted as it was derived solely from is deleted as it was derived solely from the deleted source data. the deleted source data. So, the rule is pretty simple here and So, the rule is pretty simple here and it's easier to apply because the link it's easier to apply because the link exists. A fact is only deleted if no exists. A fact is only deleted if no remaining episodes support it. So to sum it all up, So to sum it all up, deriving context is lossy and deriving context is lossy and generative. Lineage needs to be built in to the data Lineage needs to be built in to the data structure, engineered into the data structure, engineered into the data structure, which is a graph, not logged structure, which is a graph, not logged afterwards. And in graffiti we keep the sources And in graffiti we keep the sources verbatim and we link everything back verbatim and we link everything back everything derived from those sources everything derived from those sources back to the source. back to the source. And provenence offers many benefits to And provenence offers many benefits to users of graffiti. users of graffiti. You have compliance built in which makes You have compliance built in which makes your chief compliance officer very your chief compliance officer very happy. You can verify a fact based on its You can verify a fact based on its sources so you understand veracity. Should I so you understand veracity. Should I trust this fact? trust this fact? It's easy to debug where something came It's easy to debug where something came from. So, why do I have this fact? How from. So, why do I have this fact? How was it generated? And also determining was it generated? And also determining what to delete. what to delete. And most of what I've covered today is And most of what I've covered today is in the graffiti framework. So, you can in the graffiti framework. So, you can go to uh the graffiti repo on GitHub and go to uh the graffiti repo on GitHub and I have a little uh QR code QR code later I have a little uh QR code QR code later that you can zap um and try it out. that you can zap um and try it out. So, by the way, as an aside, So, by the way, as an aside, lineage and provenence is expensive. lineage and provenence is expensive. Graph construction is really expensive Graph construction is really expensive uh in the way that graffiti does it. And uh in the way that graffiti does it. And so we've put significant effort into so we've put significant effort into reducing cost and latency of generating reducing cost and latency of generating graph artifacts. And I'd be happy to graph artifacts. And I'd be happy to speak to how we've done that uh in in speak to how we've done that uh in in the Q&A. the Q&A. So thanks for attending. Um if you'd So thanks for attending. Um if you'd like to learn a little bit more about like to learn a little bit more about Zep or Graffiti, you can zap the QR Zep or Graffiti, you can zap the QR codes. Uh Zep is on the left and codes. Uh Zep is on the left and graffiti on the right. And I don't know graffiti on the right. And I don't know if we're doing Q&A here or outside. if we're doing Q&A here or outside. Okay, happy to do Q&A. Okay, happy to do Q&A. >> Just repeat the question. >> Just repeat the question. >> Yeah, >> we have time. >> we have time. >> Anybody have a question? >> Anybody have a question? >> Yeah, one right from the front. >> How do we mutate the graph >> How do we mutate the graph at the edge? >> Oh, to account for weight changes in >> Oh, to account for weight changes in relevancy. Um that is some structure that we've that is some structure that we've actually built into Zap not into actually built into Zap not into graffiti and graffiti and um what we do is we do have for that a um what we do is we do have for that a some tracing that we do which is kind of some tracing that we do which is kind of x of the graph. So not all of the x of the graph. So not all of the um provenence is in the graph. Sorry, um provenence is in the graph. Sorry, you >> if you have 50 edges, >> uh, sorry, is was that a question? If >> uh, sorry, is was that a question? If you have 50. Oh, there you go. you have 50. Oh, there you go. >> I said I I use 50. >> I said I I use 50. >> You use 50. >> You use 50. >> Yeah, >> 50 edges. Um, so so you know, Zep is >> 50 edges. Um, so so you know, Zep is able to uh look at provenence across able to uh look at provenence across those but using a separate data those but using a separate data structure from the graph. structure from the graph. >> Okay. Thank you. Thank you >> Okay. Thank you. Thank you >> for that particular problem. >> So uh does the agent also create the >> So uh does the agent also create the edges edge types and the entities itself edges edge types and the entities itself or how does it resolve those edges or how does it resolve those edges types? >> So in graffiti you can search across the >> So in graffiti you can search across the entire graph. Uh it has um vector entire graph. Uh it has um vector similarity search against various similarity search against various textual artifacts textual artifacts um full text search as well as graph um full text search as well as graph relational operations things like BFS relational operations things like BFS um it depends on the underlying uh graph um it depends on the underlying uh graph database that's used and so your agent database that's used and so your agent can walk the graph it can search can walk the graph it can search semantically etc. semantically etc. And obviously from anywhere you hit in And obviously from anywhere you hit in the graph you're then able to understand the graph you're then able to understand the provenence of a particular artifact the provenence of a particular artifact that you've hit. Uh I find it very uh fascinating this Uh I find it very uh fascinating this temporal support out right out of the temporal support out right out of the gravity database. Can you uh tell just a gravity database. Can you uh tell just a little more little more um under the hood what are those really um under the hood what are those really the the episodes are nodes in the graph the the episodes are nodes in the graph just like the other nodes and how do you just like the other nodes and how do you extract so I saw the API that let's say extract so I saw the API that let's say I give some uh ad ed episode how do we I give some uh ad ed episode how do we extract information from the ad episode extract information from the ad episode call under the hood like large language call under the hood like large language models and and edges models and and edges >> yeah yeah so uh yes episodes are an >> yeah yeah so uh yes episodes are an entity on the graph uh or a node on the entity on the graph uh or a node on the graph. Um it makes sense to model them graph. Um it makes sense to model them that way. Um in Zep and graffiti we have that way. Um in Zep and graffiti we have various derived artifacts that are um various derived artifacts that are um nodes on the graph as well because they nodes on the graph as well because they too need to have lineage and we need to too need to have lineage and we need to understand how they were derived. Um, understand how they were derived. Um, and in terms of how the ad episode and in terms of how the ad episode method works, method works, there's a pretty complicated there's a pretty complicated uh, pipeline that gets run on um, uh, uh, pipeline that gets run on um, uh, episode ingestion and I'll just give it episode ingestion and I'll just give it the very high level uh, uh, outline for the very high level uh, uh, outline for you. So there's a structured extraction you. So there's a structured extraction extracting entities and the extracting entities and the relationships between them and candidate relationships between them and candidate facts and those are the materialized or facts and those are the materialized or hydrated fact triples. So two entities hydrated fact triples. So two entities and a fact. Um and it's structured as uh and a fact. Um and it's structured as uh a fact is structured as um um uh subject a fact is structured as um um uh subject well subject uh verb object. Um and after subject uh verb object. Um and after that there is a uh deconliction process that there is a uh deconliction process that runs dduplication and deconliction that runs dduplication and deconliction process. We dduplicate entities and we process. We dduplicate entities and we deconlict facts because there might be deconlict facts because there might be existing facts in the graph existing facts in the graph that are going to be mutated by a new that are going to be mutated by a new learned fact. So Daniel loves Adidas learned fact. So Daniel loves Adidas shoes. Three months later, Daniel's shoes fell Three months later, Daniel's shoes fell apart. He sends it back to those the apart. He sends it back to those the shoes back to the return application and shoes back to the return application and he sends a nasty gram along with it. We he sends a nasty gram along with it. We now Daniel returned shoes as a fact and now Daniel returned shoes as a fact and Daniel was unhappy about Adidas. We need Daniel was unhappy about Adidas. We need to invalidate the Daniel loves Adidas to invalidate the Daniel loves Adidas shoes fact and so that is part of that shoes fact and so that is part of that pipeline as well. A lot of what we do pipeline as well. A lot of what we do uses LLMs but we try very hard not to uses LLMs but we try very hard not to use LLMs in this process as well. So use LLMs in this process as well. So where we're able to deploy more where we're able to deploy more traditional information retrieval traditional information retrieval techniques, more traditional NLP techniques, more traditional NLP techniques, uh looking at things like techniques, uh looking at things like entropy and a bunch of other, you know, entropy and a bunch of other, you know, using um sim hash and a bunch of other using um sim hash and a bunch of other approaches to ddupe, uh we do so uh far approaches to ddupe, uh we do so uh far cheaper, far faster, deter more more far cheaper, far faster, deter more more far more deterministic. more deterministic. So hopefully that answer your questions. So hopefully that answer your questions. >> Yep. Uh thank you for the great talk. Um just Uh thank you for the great talk. Um just wanted to ask a question. So it seems wanted to ask a question. So it seems like a common theme these days in uh like a common theme these days in uh memory systems is more filebased memory memory systems is more filebased memory and wikis and knowledge bases and uh I'm and wikis and knowledge bases and uh I'm just wondering have is Zep working on just wondering have is Zep working on something like that and also uh could something like that and also uh could some of the ideas here be represented in some of the ideas here be represented in that paradigm? that paradigm? >> Yeah. uh markdown suffers from provenence. markdown suffers from provenence. File-based um file-based uh memory File-based um file-based uh memory starts to break down with provenence. starts to break down with provenence. It's very difficult when you m mutate It's very difficult when you m mutate lines in a file to understand the lines in a file to understand the lineage or the provenence of why those lineage or the provenence of why those changes occurred. changes occurred. Um, not only that, but in multi- aent, Um, not only that, but in multi- aent, multi-user, and multi-source scenarios, it can be and multi-source scenarios, it can be very challenging to manage markdown very challenging to manage markdown files at scale. I think they work really files at scale. I think they work really well um for desktop usage. Uh, they sometimes um for desktop usage. Uh, they sometimes work well in uh agentic use cases that work well in uh agentic use cases that are server based, not necessarily are server based, not necessarily desktop or single user, single agent desktop or single user, single agent scenarios. Um scenarios. Um but what we found is that it's um they but what we found is that it's um they just break down with the types of just break down with the types of enterprise problems that we solving in enterprise problems that we solving in particularly in particular provenence as particularly in particular provenence as an example. Does that answer your an example. Does that answer your question? Oh more I don't know how much time we Oh more I don't know how much time we have left but maybe one more have left but maybe one more >> uh Daniel thank you. Um question on the >> uh Daniel thank you. Um question on the right bag. How do you do that explicitly right bag. How do you do that explicitly or implicitly? Um, how do you create the or implicitly? Um, how do you create the facts? Do you ask the LLM to summarize facts? Do you ask the LLM to summarize the conversation or at every turn you do the conversation or at every turn you do that? And and which component does it, that? And and which component does it, Zep or? Zep or? >> Uh, >> yeah. So, so we do um as part we >> yeah. So, so we do um as part we actually as part of the extraction, actually as part of the extraction, we've managed to get um a singleshot we've managed to get um a singleshot extraction working that extracts extraction working that extracts entities and the relationships between entities and the relationships between them and facts. them and facts. and we're able to do so really cheaply and we're able to do so really cheaply as a consequence. Um, as a consequence. Um, and so yes, we're using an LLM for that. and so yes, we're using an LLM for that. Uh, we do have a reflection step or some Uh, we do have a reflection step or some reflection built in to ensure that the reflection built in to ensure that the things that we've retrieved um are things that we've retrieved um are actually accurate actually accurate um as well as to do some other stuff um as well as to do some other stuff around uh uh more more richness to around uh uh more more richness to lineage. So why did something change? lineage. So why did something change? Not just this was related but also why Not just this was related but also why did it change. Uh that's uh partly in graffiti partly Uh that's uh partly in graffiti partly in zap. in zap. Yeah. All right. Well, thank you Yeah. All right. Well, thank you everybody.

</details>