Meeting notes provided by Gemini
Jun 17, 2026
AI Agents, Graphs, and Tabletop Worlds – Transcript
J. Langley: We can start off a little bit with uh, you know, a little, you know, some updates. Um, I know we we’d been talking about it a little bit on Discord. We are uh we are looking at well actually not looking. We are we are going to move off of the Tech 256 Discord um over to our own channel, our own server. I think server is the right way to say that. Um that’s uh some of us have been working on it. Uh I think David set it up. So thanks a ton for all of that. Um but we’ll have our own channels. Um, we’ll have things set up for meetups and for, you know, papers and, you know, things like you would expect from a, you know, instead of just one channel that’s hanging out on someone else’s server.
J. Langley: U, so, uh, I talked to Tyler, um, who actually owns that server. Um, and we’re looking at, uh, when we do make the cut over, actually dropping a link, uh, from that Discord over to the new one. Um, so that should, you know, that should help anybody that winds up over there accidentally. But, uh, you can look for that coming up probably next week.
David Showalter: Yeah, I’ve been I’ve been slacking lately. Lauren’s been doing a lot. Others have too. So hopefully in the next week.
J. Langley: Um,
David Showalter: Um don’t let it turn you off. We have a standard uh accept the rules to become a member. But as soon as you’ll get filtered through that, just agree to be nice to people and then you’ll get full access to all the different channels.
J. Langley: so there’s that. The other thing that just came up today, I got an email from Scott uh over at Hudson Alpha introducing me to their chairman of the board and wanting to have more discussions about possibly having us host over there again.
J. Langley: So, we’ll see um if they’re willing to actually do the, you know, they they want to do the community building, that kind of thing, and they just couldn’t quite get out of their policy way of how do they authorize somebody that’s not actually an employee of Hudson Alpha to host meetings at Hudson Alpha, which I I get it, you know, it’s not a um most most companies don’t just let other other people just come use their stuff, you know, so I get it. But uh also appreciate Deote um who is currently where we’re hosting our our in-person meetups. Uh so I believe that one will be next I think we’re next week uh again. I think or are we? Yeah, we are. Okay. But with that um tonight uh we got some pretty cool stuff. Um I may have seen some of this uh pieces maybe uh already. Um, but with that hand it over to Josh and I will go off
Josh Phillips: All right. Uh, thank you, Jay. Uh,
J. Langley: camera.
Josh Phillips: so tonight I’m going to be going through this talk, AI agents, graphs in tabletop worlds. Uh, this is one I’ve been kicking around for a while. Um, and it’s going to be, uh, should be pretty open, so I want this to be pretty fairly interactive. Uh, we are going to go through a bunch of demos. We added a little bit of extra time to this one because I’m going to have some stuff up for us to kind of go and explore. Uh there are three papers that we are going to cover towards the end. Uh but the big thing with this one is we’ve done a few of these where they’ve been uh just as much kind of an intro to topic. So intro to world models, intro to reasoning and reinforcement learning. This is kind of the intro to knowledge graphs that we have not done yet uh in the Huntsville AI series. um and try to kind of wrap it in a fun topic uh of these tabletop games. This is a this is a kind of my hobby side project that I spend all of my time on.
Josh Phillips: Um and so um it’s kind of a fun way to to share something I’m really passionate about which is knowledge graphs. And I haven’t talked about that with this group yet, but um I think it’ll be very cool to see how these things are used. A lot of people are talking about knowledge graphs, especially now people are starting to really see the the use of these things. you know, people that like hooking up in a simple way, you know, little Obsidian apps where they’re doing the these linkages and stuff like that. Um, and they’re starting to see really the value of using this as a way of doing agentic stuff and memory based stuff. So, I wanted to kind of lay out what are graphs, what does all the jargon mean, and really more focus on what can you do with them. Uh so I’m gonna have a demo app that has a whole bunch of different ways that we use graphs to do interesting stuff and uh hopefully it’ll be be pretty fun uh to go through that. Uh so I guess first things first uh just to kind of get a lay of a land for topics for how how familiar people are with things.
Josh Phillips: Who here has uh worked with knowledge graphs or um have heard of them, have never heard of them? I think kind of a vibe on the room.
J. Langley: Well, I’ve heard of them because we work together.
Josh Phillips: Sure. Okay. So, it sounds like uh a lot of people are aware of them.
J. Langley: So
Josh Phillips: Um so, that’s good. How about uh tabletop gaming? uh DND, uh that sort of thing. Uh who here is a a kind of a tabletop gaming pild,
J. Langley: yeah, I was not allowed.
Josh Phillips: you’re not allowed. All right, cool. All right, looks like we got a short handful. Well, uh fair warning, you do not need to understand tabletop gaming to get stuff out of this. Um so that uh we are going to go through a lot of things. You’re gonna say a lot of nerdy words. Um but um hopefully that won’t turn you off too much because there is some interesting stuff here. All right, so what we’re going to be talking about tonight is a campaign called Tolus.
Josh Phillips: Uh this is a campaign uh that’s in the D&D universe. So you got these different editions of Dn D uh like 3E, 5e. Right now they’re on 5e. Uh this is an interesting one uh because it has a lot of of very very intricate detailed stuff inside of it. And so it’s a joke that if you want to run this campaign, you need to be very friendly with your spreadsheets. Uh just because it’s this giant giant world uh where they’ve basically got uh you know this giant book here. Here’s my prop. So I actually have the tollist book. This is my disclaimer too uh before uh I I do own this book. So, all the things that you see here is is fair use uh for my personal use. I am not making any money off this money cook if you ever see this. Uh but uh it’s this enormous book here. And so inside of this book, they’ve got all of these little bits and bobs that are in the the nooks and crannies of it.
Josh Phillips: Uh and in order to get through it and kind of uh uh utilize it, you’ve got to spend a lot of time researching this thing. And they’ve got they even got like these little like, you know, it’s like I’m looking through the King James version of a nerd Bible. Um, and so one of the things that I wanted to do was to use some of these uh language models to build a system out that’s going to help me uh run a game that’s using this sort of thing and explore some different uh sort of patterns, you know, because we have all these characters. They all have their own intentions. They have the places that they’re at, uh, their own in inventories and stuff and and what would it have to happen for me to effectively simulate that world? for me to go, you know, 5 seconds at a time and and try and have all of those little bits and bobs come to life and and work without me having to do anything. You know, as a DM, uh, if I want to to try and sort of, uh, make the world feel alive, I’m basically scrambling at the edges to fill out just enough detail as my people go somewhere.
Josh Phillips: But it’d be nice if if I could, you know, maybe in my own time explore that world on my own so I can do that more in the moment or, you know, maybe have some additional things that let me explore this sort of area. And so that’s what we’re going to be looking at tonight. Uh, and so just to give you an idea of, uh, some of the stuff we’re going to look at, um, I have this simulation here. So this is that big map and I’m just going to to to narrow it down just to give you an idea of how dense this thing is to one little area. There’s this place called Delver Square and so you can see that here this is about 42 of the people that are inside of this this um this simulation. Now it is a fairly dense spot uh but you can kind of see the density of of these characters. And if you scroll out, uh, imagine what that would take to try and make this thing, uh, feel alive, to have these things going after their attentions over time.
Josh Phillips: And so it has a few different components. Uh, and so this one right here is a tickbased simulation. So as we go through this, uh, it gives each of the characters little actions. And every single one of them have their own, uh, little system. And you see as they move along, they start moving and moving to their own little areas and start, you know, going and and progressing uh along their way. And so you got to ask yourself, you know, how is it that I’m going to make uh something like this where I’m trying to connect all these webs to know where these characters go? Because I don’t want to this just be random, right? Uh, I really want these people to have a life. I want them to have a personality. Uh, something that that makes them act in a realistic way. So, if I’m popping into their world, then, you know, they’re going to act in a way that that causes authentic experience, uh, that that is going to create amazing emergent game gameplay, uh, and is going to, if I’m driving these things with LM, it’s not going to give me the same story over and over again where they kind of all end the same.
Josh Phillips: And so I want to be able to go into say for instance this uh this guy. So this is Iltamar Shawn. Uh so this is uh somebody who is uh working for the blacksmith who is inside this little town that we’re at. Well, what does he want? You know, he’s uh concerned about uh uh uh things that are are cur he’s curious about the world. He cares about being uh having status in society. Uh he has things that he believes. He has uh things that are happening to him over time that change the way that he acts. And as he goes through the world, he’s going to form intents and desires to do things. Uh you know, we want him to be able to effectively model whatever his personality is and model the things that he knows. And so, um that is part of what we’re going to be going through, uh as we we hit all of this stuff, uh through the work. All right. So, the other thing too is to think about is how do these if we’re doing a giant simulation like this where we have lots of agents and actors together, how do we effectively simulate them talking to one another?
Josh Phillips: So, that guy, he has his own goals and his own things that he does. Um, but what does he think about this guy right here, Iltimar or tell her herdsman or you know this person right here? If they end up talking to one another, what does their interaction look like that is going to be meaningful? At what points would they talk to one another? And how would they meaningfully engage? And how would that change their relationship? Uh where in some simulations they do meet, some simulations they don’t meet. Uh and where that’s going to be readable and be believable and interesting more than anything uh across the time. And so to to in order to do that you have to have a very meaningful deep and and progressive progressive in in in the sense of uh uh it unfurs as it needs to uh sort of of depth to your world. Um, and so yeah, that is I think enough to to hit the high level for what it is we’re doing. We’re going to go back to these things in a little bit.
Josh Phillips: Um, but uh I’m going to talk a little bit about the toll scenario too to kind of give a little bit more understanding of of what it takes to do uh this sort of thing and why I chose this one specifically. Uh and so the idea behind this thing was um the uh Monty Cook was the guy who one of the guys who made the third edition of Dungeons and Dragons. And he basically made this world for his own campaign. And he he wanted like one city that had all of the mechanics inside of third edition. And so it has like all the different classes. It has all the different races. that has things that go up, things that go down, things that you know, go into the forgotten planes and, you know, necromancy and all the all the sorts of bits and bobs that you would have in a game like this. He basically wanted just a nice compact area so he could see how all the systems work together. Well, um, that’s great.
Josh Phillips: It makes a very interesting campaign. Uh, it also is absolutely wonderful for me as well as somebody who is is kind of wanting to play with AI. So, I want I want to experiment and understand how far can we push these things. uh in in the world. How you know if I want to have an agent sort of interact with these systems, you having them in this close proximity where they’re going to be in conflict with one another where they’re going to have to work with one another kind of makes this wonderful uh playground uh for me to experiment with for my own research. Uh, and so to give you an idea of some of the problems that you have to do, because what I want to do basically is I want to take this giant corpus of stuff that, you know, I’ve bought, but I want to get it in a way that works for me. And so we got to do this sort of rag sort of thing. And so I need to find some way to get all of this gorp that is in this very very ugly format uh and kind of take this and how do I break this into little tiny claims and nodes that can kind of relate to other things inside of a what you will see to be a graph-based structure.
Josh Phillips: Basically, I want to to find some way to get these down to a small enough component that I can piece and part and deal with small things inside of this text uh as they become relevant to the system. And so, uh to do that with a normal sort of like rag is very difficult because you can see here it’s got stuff that’s in a single column. So, a lot of times if you’re doing some sort of a uh an extraction, it will read across and not have a good time with that. Sometimes if it’s just a simple two column thing, that’s fine. But this thing has the addition of adding stuff in the margins where there’s like consistently a margin that will go off to other pages. It will, you know, have little tiny links, have things that are very unformulic. Sometimes it’ll be images. Uh, and so how do you effectively get information out of that? Um, and so you can see here some more examples of this sort of stuff where they’ve just got these like so this guy it has really kind of like a structured table sort of thing where it has like here this character it has a stat sheet and it has certain things filled out and there are other ones that’ll just have like one little little blur about them or it’ll mention an item that they have but it won’t have a stat sheet.
Josh Phillips: So how do I work back from that? Because if I want all these things to work together, I have to be able to effectively fill in information that’s missing, too. And so, yep, more stuff. And some of this, you know, if you throw this something like a very small LLM, this was going to make it go crazy. And so, you have quite a problem here of how do you you get to all this extraction just because that’s all gorp that is going to make it go nuts. And so this book right here is actually only one of them. So this this is 672. There’s also a bunch of extra ones. And so it ends up becoming quite a large uh bit of knowledge uh that we are going to do. Um, and so for that, um, I guess I’ll stop a little bit and talk. I just kind of question folks, you know, have any of you, uh, dealt with a problem like this where you’re trying to get a bunch of information like this, uh, into some sort of area, you know, dealt with pipelines or things like that?
Josh Phillips: I don’t know. Any any thoughts before we move on to the next area? I don’t want to blabber too
Lorin Bales: Yeah, Josh, I was just curious,
Josh Phillips: much.
Lorin Bales: um, did you, um, given the massive amount of context here, did you fine-tune the model a little?
Josh Phillips: I did not. No.
Lorin Bales: Ah, cool.
Josh Phillips: Yeah. So, it’s all it’s all going to be uh straight. Uh yeah, Walter, what’s
Walter Freeman: So I you obviously know about the things that we did together,
Josh Phillips: up?
Walter Freeman: but I played around with some of this stuff on my own. And um one of the the principles was that I was working with a world that wasn’t completely filled in yet. That the all of the actors were not yet known to the database because you know as as comprehensive as the Polus book is, it probably doesn’t tell you about the life of the baker’s granddaughter’s friend,
Josh Phillips: Right.
Walter Freeman: right? Um, does does your like you have this corpus of of stuff in the book and characters and agents and places and so on.
Josh Phillips: Absolutely.
Walter Freeman: Do you also have a facility for um some agentic model saying, “Hey, I need to invent a new agent and and stick it in something that wasn’t written in but it but needs to be there to make the
Josh Phillips: Yeah.
Walter Freeman: story work.
Josh Phillips: Yeah. And and there’s the president. I won’t talk a ton about that. We’ll get a little bit to it. Um because we’re going to talk a little bit about how do you ground. So So you always want to start with um what is known. So you have something that is as strong as possible. And so we’re going to talk about something called evidence graphs. Uh that basically says here’s all the things that are grounded. So don’t change that. So we have you know somewhere solid to start from. Uh but then obviously you do want to be able to backfill things and so I have a whole pipeline that allows me to do that that basically says you know hey all right so this thing is of of a fifth level it says that that um actually we’ll talk about this when we get to uh Jevica which is one of the people that we’re going to be looking at and I I’ll kind of talk through that there.
Phil Bording: Josh,
Josh Phillips: Yep.
Phil Bording: it’s it’s Phil. You know when we we look at human beings and let’s just use height and
Josh Phillips: Hello.
Phil Bording: weight. We have some sort of normal distribution of character and structure for height and for weight. All right. So how how do you take the attributes of the characters and look and see if they
Josh Phillips: Mhm.
Phil Bording: have just absolutely the same level of or do they have some normal distribution? I mean, what’s the statistical analysis of the
Josh Phillips: Uh so actually Uh,
Phil Bording: characters?
Josh Phillips: it is something that that there’s a whole bunch of So, if I go into um trying to think of where it has the best view. So, I think this has the best view. Um, I’ll go find one that has a character. Here we go. So, so it actually does have uh things like, you know, what is their eye color, what is their height, you know, what are um yeah, their appearance sort of things.
Josh Phillips: uh because that does matter for some of the the checks that happen. Uh and so it I actually do have a system that goes in and if it doesn’t have information specifically, it will look at the core rule books and say like, “Oh, this person’s a human and they’re usually this height to this height. They’re usually, you know, this sort of appearance um and they have these different backgrounds because this thing gets it’s all this stuff gets very deep as far as like what they’ve defined out.” And so it will will take that and I have a series of agents that will then come up with a reasonable uh sort of thing to put there for them. Now I don’t dip it to kind of ensure diversity um I don’t think that’s necessary um until I I find otherwise. I think I think just because of how this thing is uh set up uh it’s going to be diverse because everything is so close together. Uh which is why I like this scenario. I don’t know if that answered your question, but well,
Phil Bording: I’m okay with that.
Josh Phillips: that’s good. All right, so um we’re going to take one little microcosm. So, just like we were talking about this, uh giant area uh is, you know, kind of a melting pot. There’s a smaller melting pot that we’re going to be looking at, too, which is this Delver Square. This is kind of like your tutorial island, uh for this this campaign. It’s got, you know, it’s got an inn. It’s got a blacksmith, a a sort of magic vendor. Underneath it, goes into like an unders city and beneath that is an endless dungeon. Uh, and so it’s kind of like this little hub that you could stay inside here for an entire campaign and have a pretty good, you know, year or two of adventures if you really uh went downwards quite a bit uh with the Delver’s Guild. And um it’s great for us because now you know uh part of this is all of what I’m going to show you today uh it’s all 100% local and so I’m doing all of this on local models that I host myself.
Josh Phillips: Uh that does mean that uh I do have a some constraints about what I can reasonably generate even though I do have fairly good hardware for this sort of thing. Um and uh the nice thing about this is that I can get a meaningful you know uh subset in order to do se a reasonable amount of simulation. Where do I here? Let me get back to here. Uh just with this tiny group of people here which gives me 42. So every time I want to do a turn I have to do 42 individual turns and whatever that means which in this current case is about eight LLM calls in total. Um, and so that’s why we’re talking about Delver Square and we’re exploring these folks and they’re all kind of in close proximity and you’ve got a lot of the hooks into the bigger Tollus plot too. Uh, and so we’re going to talk about one person especially throughout this talk is this person called Jebica. It’s just uh one of the the characters that’s inside of the starting segment who kind of leads you around.
Josh Phillips: Uh, she’s super highle mage. She’s connected to a lot of things. We’re going to talk about this place called the inverted pyramid. She’s also uh part of a city council. Think of the inverted pyramid as like your your your Gandalf and your Sauruman sort of secret wizard society. Uh I think is the the most common way to to put that. Um then uh she is uh in the ghostly minstrel which is the tavern that we’re going to be be in a whole bunch that’s inside of Delver Square. She gives lots of uh stuff to adventurers to kind of start a big quest, one of the main quests in the series. And so she has some secrets and we’re going to talk about that too because one of the interesting things about uh Dn D is it’s a very interesting form if you think about it. If you think about it from a terms of knowledge graphs and and sort of what one person knows versus what the entire rest of the table knows where the game master has all the secrets.
Josh Phillips: He knows everything and it can take you know a year for the players to essentially earn that secret that lets them get to the next part of the narrative. And so if I want to hook agents up to this thing, I’ve got an interesting problem where I want the thing to be useful enough to to, you know, be interesting to play with without the LLM blurting out the end of, you know, that Luke dies or, you know, Luke gets his arm chopped off at the uh the middle of the Star Wars movies or something like that. So um she’s got some secrets and that’s why we’re kind of choosing her too. And so you can see here we’re going to start getting into this concept of of what a graph is. And so a graph the big thing is that um we’ve got these little thing these little nodes and these edges and they go to different areas. And so a Jeepa basically just think about this as as how connected she is to other things that are happening inside of the system.
Josh Phillips: And so this is her neighborhood which we’ll talk about what that means in a little while. So she’s connected to something called the pack lords to this inverted pyramid. I said earlier that that is something that she is a member of. And so here we’ve got the pack lords and we’ve we’re saying that she is opposed to that. It’s got this fancy little dotted line. What does that mean? Um there’s this thing called the Iron Mage, Nevada, Bane Warren. So she’s got all these things that she’s connected to. They don’t really match up, but we’re still connecting them and showing them here. And so we look out, we say, okay, those are all the things that are connected to her. Uh we can also uh mess up our mouse a little bit. Also ask the question so what are connected to the things that are she’s connected to? Uh we can ask other questions like like what are the uh pieces of information that she has? So what are the the secrets that sort of live around here?
Josh Phillips: So there’s this question of is there a mole inside of the inverted pyramid and does she know that? And uh are we supposed to know that it has to do with the Bane Warren’s key? Um and I’m very sorry if anybody wanted to play this. I’m going to to minorly spoil a single quest line. Uh but I’m going to try and keep it to that. So I hope you all will forgive me. I think I think we’ll be okay. But um we can use graphs to to model some of this complex information uh sort of relationships in a way that allows it to progressively unfold. Uh we can also ask questions like okay so I’ve got this concept of uh this iron mage. So where did that come from? So I’ve got this thing in this graph and so here I have something leading back to where this information was sourced from. So this is from this campaign journal. You know I’ve got this just reser reversal and four-fold ostracism.
Josh Phillips: These are two spells and so they’re described here apparently on this page. So this might be something that’s interesting to know. Um, and then yeah, just more things about secrets. And so I want I want you to think about too, you know, as we’re going through this now, obviously we’re going to be talking about uh the fun little D and D game, but think about this in your own world, the sort of places that you work with. You know, this graph structure is is is very very useful uh for for other things. So, you know, maybe for here where I’m grounding it, uh maybe I am somebody in finance and so I’ve got, you know, some sort of uh receipt and I have something that is the physical copy of that receipt that ties to my piece of bookkeeping. You know, maybe I’m running a uh a test for performance for my website and I want to lean that back to uh some sort of metric that proves that I run that test. So all of these things can kind of uh go into other realms as well.
Josh Phillips: Uh yeah, we have a
James Chung: Um yeah so this is a directed graph.
Josh Phillips: question.
James Chung: So what what do the arrow means? Is it is it like a causation or uh if you can give me a context of what the direction means then that would be great.
Josh Phillips: Yeah. So this is uh kind of an RDF triples here. And so, uh, it’s gonna we’re going to go into ontologies, too, and semantic graphs. Uh, but basically these are tied to a semantic graph that I have for this system. And so right here it’s saying that Jebica is watching the Iron Mage. Uh, and this isn’t what my actual ontology is, but uh, yeah. Well, can can I hold off on that for one second and then we’ll kind of go into to what these are these are. Okay, cool. But yes, uh, we will get to that for sure. All right. Um, I already went through that. And so, yeah, the other thing that I wanted to I think I I talked about it a little bit, but something to remember too is just this concept of, uh, graphs can be a lot of different things.
Josh Phillips: Uh, so you can have graphs that serve as a source of truth where the graph is trying to be absolutely true and grounded. Uh but you can also have graphs where you’re trying to use them to um create a fuzzy world. And so for instance, I might have the GM who has, you know, absolute knowledge of the world, but I want to gradually piece together this player and their knowledge of the world as it piece and parts around and they learn new things and they build a graph that might contain false information, but we want to model that false information. And so how do we deal with that with with graphs that are knowingly have imperfect information? This kind of gets into like the the world of imperfect information games and that whole that whole area uh where this is something very much we want in a game like D and D because that’s where the fun is. Everybody knows the plot from the beginning. It’s there’s nothing to do. You’re uh you’re watching a movie. See?
Josh Phillips: All right. So now I’m going to go into a little bit of what a graph is. Um and so graph uh is formed out of nodes which are these sort of things. This uh uh inverted pill pyramid Jebica they contain an edge and an edge basically stands for a relationship. It’s some sort of uh of of connection between uh those two nodes. As far as what that connection is, it’s could be very open and we’ll get into what all of those are. Uh, and then a lot of times there are properties that are also stood off of these things. And so I could have Jeepa here and it has her whole character sheet in it. Or I could have uh a member of inverted pyramid and I could hang a property off of this that says for five years or in bad standing. Uh, and so you can do a lot of different uh sort of schemas for these things in order to make different interesting shapes uh to do uh stuff with. And so the big thing with these is that where a a graph datab a relational database is is usually using joins to do the kind of things that we do with um a graph uh sort of structure.
Josh Phillips: And so it’ll look for the relation in sort of a tabular format with a key and find some information and then tie that to some other uh table that also has a key and join those things together. And that join is the act action and it’s done at runtime uh and sort of uh dynamically uh so you have to to to do something in order to make that join occur whereas a graph it just has it uh and it that is like the structure upon which you query uh out of the gate without that additional joining and well yeah sure you can get there at the end of the day it massively changes how you interact with it at scale. uh and so for areas where we we care more about the relationships between things and unfurling a network uh graphs become very useful. So uh every graph is formed from a triple that contains the subject relationship and the object and it looks like this. Uh it can have a direction. So Jevica is a member of the inverted pyramid.
Josh Phillips: Uh an ontology is basically the rule book. the vocabulary uh for those triples at least in this case you know there are a lot of different ways to use ontology generally uh we’re talking about this uh this is kind of how that comes out so it is valid for Jebica to be a member of the inverted pyramid uh it is valid for her to guard the sealed key based off of this ontology but it’s not valid for the Delver square which is a place to be a member of the inverted pyramid there a lot of ways that you can define the ontology Um, I generally like to think of them as the semantic graphs where you have a meta model. We’ll look at that in a second of uh basically a graph that describes another graph is is kind of how it works. And so we’ll have another structure that says actors can become members of organizations and you kind of do a weird semi-inheritance composition uh thing from there. But that’s the idea of the ontology is that it is a rulebook of sorts.
Josh Phillips: Um, and so one of the things too like what you’re talking about uh James was uh this would be where we and it sounds like you probably already know this but this this would be where we define um that directionality. So you at the ontology layer you could define that uh you know an actor be can become a member of an organization but an organization cannot be a member back of an actor that sort of thing. uh and so among graphs I think there’s probably examples that do not fall into this but most uh of all graph structures fall into this tree right here which is a undirected graph and then directed graphs uh and then within that there are uh directed a cyclic graphs uh which is basically the graph goes one way and it never uh goes backwards and then cyclic graphs that are kind of uh they can go every which way and the other. Uh and they all have their structures. None of them’s better than the other. Um we’re going to just kind of go through and get an idea of what the different types of graphs are and and what how we can compose these things uh in order to make interesting data structures.
Josh Phillips: So an undirected graph I think uh one of the easiest ways to think about this is uh a lot of things with proximity. It’s very hard to have a directed um graph with location. You can for sure, but generally, you know, things are where they are in the world and in relation to one another. Uh, and so, you know, Buffalo, Buffalo and Albany are 290 miles away from each other. And that’s just it. That’s that’s how how far away they are from each other. You can’t change that. Uh, if I, you know, my house is next to my neighbor’s house, my neighbor’s house is next to my house, they can’t my house can’t be any more next to my neighbor’s house than my neighbor’s house is to mine. Um, so that’s the idea of kind of a a undirected graph. Uh, there’s probably a lot more to this. I don’t work with these a lot. Um, generally I I am I am I I’m working in the world of directed graphs.
Josh Phillips: Uh, so I can talk a little bit more about these. Uh, so the uh next graph we’re starting to get into uh that directed graph area is a hierarchy. So a hierarchy is a parent has a children. So uh Delver Square is part of Tlus. The ghost of ghost ghostly minstrel is part of Delver Square. So it’s it’s sort of a nesting sort of relationship where the direction is going uh down the tree. Um you can also have these be these could be uh directed a cyclic graphs. So you could have it that uh a child can only be uh a child of one parent or you can have it be something where it kind of like moves up and you have kind of a jagged hierarchy. Uh and so uh both of those fall under the directed graph area them. Um you can also have a directed asyclic graph here uh where you’re thinking of something like a pipeline. And so a lot of uh data pipelines are going to be directed asclic graphs where you have kind of your first stage and something happens to it and it gets turned into something else.
Josh Phillips: And so here is kind of a loose very high level way that my pipeline worked for this which is um you know my page 18 gets chunked into a chunked and we extract stuff from the chunk and then we resolve the things that we extracted and try and merge them into an entity and then we admit that into the graph and so that’s a directed asyclic graph. It does not go backwards. There’s no uh you know send back. If something gets canceled that’s the end of it basically. Um there are other ones where we can do that sort of like as an evidence sort of graph. So the key is sealed and this is something that is about the key and it’s backed by this. Um where Jessica vouches for it. So it’s about this person and it’s backed by that where I don’t have some sort of backwards relational thing. Um, but it’s somewhat off on its own. And you can also do it with something like page citations. You know, generally pages come I guess you could get into some weird weird thing where you know maybe somebody does a revision for a paper that then references a paper that referenced it.
Josh Phillips: But in most cases um I don’t think that’s a thing. Uh and so you can see like you know the three papers that we looking at tonight they all site uh graph rag which was a paper from Microsoft I think in 24 or something like that. Uh and that all sites rag which is a paper from 22. Uh so stop a little bit. Has anybody worked with either of these types of uh graphs before? You know directed graphs uh undirected graphs that sort of thing.
J. Langley: Yeah, I’ve definitely done directed graphs and some some some probably undirected um
Josh Phillips: Is anybody here realizing they have worked with these things and didn’t realize it before right
Lorin Bales: So Josh,
Josh Phillips: now?
Sam B: A lot of medical data is in graph format.
Lorin Bales: I uh Yeah, I’ve used it in cyber.
Josh Phillips: Yeah. Oh, yeah. It’s It’s huge. And any sort of network analysis stuff.
Lorin Bales: Yeah.
Josh Phillips: Um All right. And so cyclic same sort of thing where the the other one was directed as cyclic graphs they do not go back on one another.
Josh Phillips: Cyclic graphs can go back on one another. Uh so those can be semantic graphs. Of course I forgot to put the cyclic nature of the semantic graph uh in this one. Uh but just imagine a line going back to one of these and then you’ll have what I meant to do. Um you can have uh knowledge graphs which I think of knowledge graphs it’s it’s a very fuzzy one but uh semantic graphs I I view knowledge graphs as a a realization of a semantic graph. It’s kind of how I think of it. So you’re usually applying some of some sort of ontology uh from some some viewpoint because you can’t have a true ultimate you know golden knowledge graph. there’s some sort of perspective that that you’re starting from a from a knowledge graph in order to uh to instantiate uh a semantic graph. Yeah. What’s up, Walter?
Walter Freeman: So I see this and I love your example this right um so I see you have a birectional an explicit birectional connection between Cheva and Nana right and that just opposes
Josh Phillips: Yeah.
Walter Freeman: Nana um is in general if you don’t have that
Josh Phillips: Mhm.
Walter Freeman: is information only allowed to flow one way so for instance in in your let’s take an undirected arrow rate between Jevica and the pyramid Jeva is a member of the pyramid Would we be able to find that relation from probing either node either by like can you ask a node in the graph what points in as well as what points
Josh Phillips: Yeah,
Walter Freeman: out?
Josh Phillips: I mean you can do anything you want. I mean that’s the sort of thing because you’re you’re working from Legos, right? Which of of all all you need is that you have uh I have an a a node and then I have a let’s see go back here. You have a node. Where did I find it? This triple that’s all composed of triples like just like atoms. I have a node, a relationship, and another node. And then on top of that I can label a ontology which then becomes my rules for for what that means of that semantic graph.
Josh Phillips: And so this semantic graph that says an actor can live at a place you know and then I have another one that says a place can be owned by an actor. Uh if I put that into my ontology then that defines that. Now I could also you know maybe have a loose ontology. So, I I try and move things into one direction, but I let people kind of go loosey goosey and and and uh you can have some extra stuff. I wouldn’t suggest that uh as a long-term thing, but that could be useful for like some data scraping sort of uh sort of stuff. Um you probably want to iron it into a more strict ontology at some point. And then as as far as um how do I then take uh that ontology and do like what you’re asking which is I want to then be able to query effectively given that um that is a topic that I almost added into this but we’re already very dense uh which is called like an executable ontology uh and this actually how my thing works and So I actually have an executable ontology in this system where it enforces you know that a um a claim so things that I call claims which is kind of these like assertion of something about the world that must uh have a some sort of source page some sort of hard artifact behind it and if it doesn’t that’s some sort of a violation that flags my pipeline to do something with it.
Josh Phillips: Um I don’t know does that answer your question?
Walter Freeman: Uh yes that’s you’re you’re kind of explaining what the um in answering the question you’re saying that the agents I guess who are driving these entities have complete information of the graph structure at least in their
Josh Phillips: Right?
Walter Freeman: neighborhood.
Josh Phillips: So they have complete information about their graph structure. So so Jebica has her own graph. The game master has complete information about everything in theory,
Walter Freeman: Yeah. depth.
Josh Phillips: you know, and that’s actually what this is right here. So So one of the the major content here, I’ve talked about it a bunch, but we have this contention of the world graph against the character graph. And so Jebica uh she’s guarding the key and there’s a mole that wants that key. That’s that’s the actual situation. Uh but she does not know that. And so the process of her learning that is the struggle that makes games fun and that we want to track. And so yeah, they can be typed.
Josh Phillips: We kind of talk through this. That’s a big thing. Uh graphs can also be weighted where each edge carries some sort of a number. This can be a multiplier. weighted graphs, you’re getting real close to actually how neural networks work, too. So, so this concept of a a a directed asyclic graph uh with weights and activations and biases and suddenly you’re in a neural network. Uh but this also be things like you know the the level of confidence inside of an item you know what is the degree of that relationship if you have an ontology that makes that make sense. uh they can be signed where the links are positive or negative. So you can have a a a onedirectional graph that uh through polarity uh you assign additional attributes to it. Um and you can also have temporal graphs which we will actually see a temporal graph in our thing where I do since I’m doing a simulation simulations have ticks and so we are watching the change of a graph over time and that’s a very important thing for sort of the work that I’m doing.
Josh Phillips: And so here we see a beat three uh Jebica is just a member of the pyramid. And then here something happens between her you know I I haven’t defined it but something some sort of relation has emerged here between Jebica and the key and uh Jevica and Nana. Uh so you mentioned before this concept of something have multiple connections that’s really called a multigraph when there’s more than one link per pair. These are getting into things where you know these are just attributes that you can use to compose an interesting graph structure. Uh and you can also have a hyperraph where you have one link with many endpoints. So you have kind of something that that serves as a a locus. And uh this is actually somewhat the topic of one of our papers uh where we’re thinking about this one is thinking about this where the hyperraph is important uh but we’re going to be actually looking at a paper that’s interesting in the other way where what if you have basically one link with many endpoints where the link is not important.
Josh Phillips: You have a liinal it’s almost like a liinal space uh that holds the graph together. Uh so there’s a bipartite. I won’t talk about this. Uh so property graphs we talked before all a lot of what I’m I’m showing are are RDF triples. I actually use the property graph more but it doesn’t it doesn’t show well in slides. So you can also you can have it where everything is uh defined through triples. Um and so anything that you want to define has to be defined through triples. And this is how a lot of like the older bas databases are. Uh but then uh a property graph you can hang metadata off of that which obviously uh makes a lot more sense to a lot of folks including myself. I think it probably I think this has some some performance issues. It’s the issue with property graph but I don’t care because I’m lazy and it makes sense to me. Um let’s see. So yeah a hop. We sometimes talk about hops.
Josh Phillips: That’s a a lot of times you’ll actually hear multihop uh reasoning is used in kind of non-graph based stuff, but hop means that I’m going from one node and going to its relationship and the next hop is a is a secondary relationship. A neighborhood is just everything close by. What’s what’s everything that’s one hop away from me or two hops away from me? You know, up to some area to some bounded neighborhood around me. It’s one of our papers we’ll look through. It’s something about neighborhoods. A degree uh is how many things are connected to me. So, you know, you have a pretty good hint that something is wrong. If you have a node with a degree of zero, you’ve got an orphan there or a node, two nodes with a degree of one. You got an island. And so, a lot of especially when you’re working with LLMs is that they’ll spawn out, you know, all these little nodes and you’ll have a dot dot dot dot dot dot dot dot dot dot dot and it’s not hooked to anything.
Josh Phillips: It’s a big pain in the butt. And so a lot of the work with these uh data pipelines here is figuring out how to federate those those islands into the main graph. Yeah. What’s up,
David Showalter: Hey,
Josh Phillips: David?
David Showalter: so I’m guessing the other side too. If every node has 500 connections.
Josh Phillips: Yeah, it’s probably not good either. Yeah.
David Showalter: Yeah.
Josh Phillips: uh you you probably uh you probably that is probably a ontology problem
David Showalter: Yeah.
Josh Phillips: where you you basically it’s really easy to do that if if your thing is like this node is related to this node your ontology is related to you’ll see stuff like that uh which that’s no good either that or your pipeline sucks for some other reason yeah what’s up uh James
James Chung: Uh so this is a slightly different analogy but we can think of the directed graph as some kind of a flow of the information. And so if we if we say that the information is flowing one direction to the other then would that neighborhood matter for the end of the edge because you know the information cannot go the other way around traversing the edge direction.
Josh Phillips: right? So it’s the that is one of our papers. Uh so yes, great question. Um I think the answer is probably uh maybe um uh it can and so you probably need some way to understand about that. So I think that the question is is how well-connected is your graph? And so do you have a lot of these liinal areas where um uh if I remove that that sort of middle area that eventually wasn’t important for my question, it doesn’t get cited uh and it’s not, you know, ranked as being high, but it somehow was the only way that I could do a network hop up to something that was finally useful. Um then maybe it’s important. I don’t Yeah. Yeah. As as far as you know what it is, I’m not sure what the answer is. It’s just a an interesting question. Uh so yeah, with graph retrieval, uh just same way we’ve talked about graph, we’ve talked about uh normal rag a whole bunch where you’re doing the semantic sort of retrieval chunking.
Josh Phillips: I’m sending in a piece of text and I’m doing some magic uh sort of embedding stuff on it and then I’m going to go find something that’s kind of like that. Uh and you’re looking at this stuff can be pretty useful uh but it has a limited sort of of uh of of depth to it. And so if you have really really complex problems uh that that rely on relationships in order to be able to effectively understand that this thing uh is actually related to this other thing um then uh normal rag falls apart and so that’s why it it works most the time but then it just totally whiffs. Uh something like graph rag can help with solving that. You got to put in the work for the ontology to make sense. Uh so it can kind of unfurl uh some things that wouldn’t normally get there. All right. So, um let me think of how to go from here next. Um so, I’m going to talk through uh a little bit of the papers.
Josh Phillips: We’ll go through a little bit of the papers that we’re going to hit and then I think we’ll go back into our demo and look through a little bit of the stuff in action. Um and so the scout rag here um we have three papers here and they’re mostly focused around uh when I have this structure what are the questions that I can ask about my information and the flow of data uh inside of my system that I can’t effectively ask with a relational database in the same way uh and get uh get very far with it. Uh and so scout rag here is I think my favorite of the three papers that we we’re going to go through. Uh and this is the idea of how do I deal with a graph rag across a network uh with multiple domains uh where I cannot for some reason have all of that information inside of one graph. And so this could happen for a lot of reasons. So obviously we have our situation here. We have the imperfect information gain.
Josh Phillips: That’s the point. The game master knows everything. the players don’t know everything. I still want the players to be able to interact somehow with the graph and do useful things with it. Uh and so we’ve got that that sort of uh idea. That’s the siloed domain concept. This can also be uh the case of you know a lot of us work uh in areas where we have to have um bipart uh partitioned information uh due to security reasons. It could be for health reasons. HIPPA, you know, you can’t co-mingle uh data between different uh sort of uh jurisdictions. Um if you are dealing with anything with the government, uh obviously you understand all of that. You have uh partitions across across need to know across different systems, different companies that are working maybe on the same project and have to integrate together that you want to be able to use these sort of things with. Uh so how do we handle with those sorts of situations? uh while keeping the the quality as high as possible.
Josh Phillips: Uh and so they’ve got this sort of uh system called Scout Rag. And I’m just going to talk through a little bit of the pipeline here so you can see the shape of how this uh could uh potentially roll out. And so the first thing that happens here, uh we’ve got this key query that goes in. So it’s like your normal rag sort of thing. And that goes into uh this DRAA. Uh so this is this uh let’s see do I have the acronym? It’s this the grouper is basically domain relevance assessment agent. And so what this is going to do is it’s going to look across the entire corpus of everything that you give it access to and it’s just kind of kind of eyeball it and it’s going to say hey what’s the general idea of all this stuff? And so this is something that uh uh is generally done. There’s lots of ways to do this but an easy way to do it is the uh communities. Uh, and so I’m going to go here into our explorer.
Josh Phillips: Um, and I’m gonna go here so we can kind of see what a community is. So a community is was with a um a neighborhood. A neighborhood has to do with uh what’s kind of near me. And a community is a larger grouping of things that match some sort of um sameness to them. Um, and so here I’m using a community with a Leiden algorithm. Uh, I believe I think that so there’s two major ones that that people do. Uh, is it a Leiden and Lubane um, or I don’t know how you actually pronounce those things, but that’s that’s how I’m going to call them because I’m from the south. Um, and there are some other ones too, but these are are generally what people are using when they’re dealing with with graph. And the idea is that we’re going to take a whole bunch of these things and we’re going to try and put them close together so I can bucket things at a high level. So, I’ve got uh 10,000 entities here in this particular data set that I’m looking at and I want to be able to look at them broadly to understand, you know, Dewey decimal system.
Josh Phillips: What part of the library do I go to? Um, and so here I’ve got this sort of thing and I’ve had my LLM look at all the things inside of here and kind of give each one of them a name, but they don’t have a name generally. So, if you go to like Wikipedia, uh, another way to look at this is, you know, yeah, Wikipedia uh, community graph I think there’s like a big Yeah, here we go. So, this is like where somebody has done this for all of Wikipedia. And so, they’re kind of grouping things together and you can see the bundles and clusters. I’m sure you’ve seen stuff like this. You know, there’s these these, you know, these sort of visualizations that come out every once in a while. Uh well, this is very important for us in our problem because this gives us a way to start. So I want to look at all of these groups and say, okay, it’s not reasonable for me to go and look at every single node here.
Josh Phillips: Uh I’ve got, you know, tokens cost money and I I want this answer before I’m 80. Um and so what this guy here is doing is that he’s going to look at all of these different highle things. just going to scan them and say, “Hey, you know, I think that this uh in this case we’re talking about that that dang key again.” And so we know that the key has to do with the inverted pyramid. So there’s a community that has that. There’s a community that has House Bottom in it. And so I’m considering these to be highly related. There’s another thing where I’m I’m hitting it with the church of Lothian. There’s something in there that makes me think it’s somewhat related. And then there’s something in the undersity market. I don’t think it’s related at all. And so what it’s going to do is it’s going to say, “Okay, for these two communities, I want to look hard. I want to look at everything. I think there’s lots of little things inside of here that are useful, and the more time I spend in here, the probably more I’m going to learn about my query.” Uh,
Josh Phillips: and for the this moderate one, uh, dra is trained. Uh, it doesn’t have to be. No, you can just dr aa it can be anything. Uh, so you don’t have to pre-train a model. probably can. I think you could probably make it very um cheap. So if you like did like a a modern BERT sort of thing, I think this could be somewhere where you can use like a BERT model to classify into something very quickly. Uh but you can just use a normal LLM too if you’re wanting to uh you know be lazy. Um the big idea is that it’s able to take in some sort of some sort of uh criteria and output that classification. Um, and so yeah, for the moderate ones, the idea is that, okay, I’m not going to look at all of this, but I think there’s probably something inside of here that I can glean from. So, I’m going to look inside this community or this bucket with some very specific search terms, and I’m going to constrain my budget for this thing because that’s one of the big things that we care about too when we’re looking at these broad broad domains.
Josh Phillips: Uh, of course I could brute force it and I could just search everything as much as possible and I’m probably going to get a better answer over time, but it might take a long long time. It’s like deep think from uh his writer’s guide to the galaxy. Uh, or it’s just it sits there and it’s just thinking about everything. Think about it. It’ll be with us in a few millennia. Uh, I don’t got time for that. So I want to kind of balance get as correct as I can when I’m dealing with these these these split domains that I cannot do one search on. And so it does all that searching and it goes to another model which is this tagga which is the partial answer generation. So it’s going to take all these things and it’s going to try and come up with answers from those things and it goes to another model and that synthesizes it and it proposes an answer. Um, now this is something I’ve actually implemented in my system and it’s pretty cool.
Josh Phillips: Uh, it works pretty well. Uh, and so the idea then from that seed is that it goes to this other agent who is evaluating it and then we determine, you know, are we happy with this anther answer? Uh, how much time left in my budget do I have? Uh, and if I have more time uh, and budget and I’m not happy, go back and search again. Otherwise, kick it out. And so yeah, the idea here is that we’re trying to to trade off between how much time do I want to spend on this thing while dealing with this very difficult synthesis problem. And we’re looking uh for the uh semantic similarity. So this is one of the ways they do that search, the knowledge richness. So we’re looking at the degree. We talked about the degree earlier. So the relation count that we have uh and then the uh average answer quality from past queries. And this can be from um you know is this domain generally noisy? You know do we trust the the information that’s coming from?
Josh Phillips: Uh then you can kind of have a bunch of different heruristics of how you determine this that’s going to matter to your thing. The idea here is that this general shape of of I’m getting something in. I’m spreading out with a scout. This is why it’s called scout rag. uh and then I’m synthesizing from that scout to come up with candidate answers and then turnurning until I’m happy is a good way of solving this impartial information problem.
Phil Bording: Well, two things, Jay.
Josh Phillips: question.
Phil Bording: I got to go, Josh. I got to go and do some other work. This is being recorded.
Josh Phillips: Uh
Phil Bording: Okay.
Josh Phillips: yeah.
Phil Bording: And a few years ago, I worked with the assoc data association algorithm, but the data association algorithm uh would only allow you to
Josh Phillips: Mhm.
Phil Bording: do one data one data item with another data item. It didn’t allow you to do the best fit of three things. You had to do the best fit of two and then you had to do the best fit of one of those with the third one and then take the third one and do it with the with the second one.
Phil Bording: In other words, you had to do all possible combinations of two best of two items to get the best fit and that gave you a score. Okay. Yeah. Anyway, but this is being recorded. I can look at it later.
Josh Phillips: Uh yeah, we’ll have to remember to post it,
Phil Bording: Okay.
Josh Phillips: but yeah, eventually probably.
Phil Bording: I gotta go.
Josh Phillips: Okay,
Phil Bording: All right.
Josh Phillips: thank you.
Phil Bording: Very nice talk. Enjoying it.
Josh Phillips: Yeah, thank you.
Phil Bording: All right. Bye.
Josh Phillips: All right. So, um, yeah, here’s the kind of what they’re talking about as far as their, uh, their algorithm for deciding to stop. So, we have this idea here of, uh, we’re doing the retrieval and we’ve synthesizing it, seating, and am I good enough to stop? That’s the strategy uh, here. Um, and so that’s this algorithm where they’re basically this is the hyperparameters that you would tune for your own system. uh where how deep do you want to go?
Josh Phillips: Uh uh what’s the breadth? So how wide do I want to to um I have some sort of domain that I’m in and I think that there’s more stuff to gather inside of this domain. I’m getting stuff from it or there’s something in here that I know is here uh and I haven’t found it yet. So how how deep do I want to go in one area? Do I want to back up and do a breath search and say, “Okay, I’m not finding anything in here. Uh I need to look elsewhere. I’m in some sort of local minimum.” uh and I still don’t have the answer that I need. Uh or do I do a hybrid where I’m doing a little bit of both? Maybe I’m going deep one area, I’m going a little deep in a lot of areas. Uh and so that’s the decision that you have to kind of tune for your system. Uh and then kind of what’s what’s your stopping condition? Do you have some sort of verifiable reward?
Josh Phillips: If you have a verifier, this becomes very easy. Um but uh if you don’t, how are you going to get that heristic to stop? Um, and so here’s uh this this is uh this looks bad at first. I promise you it’s not. Um, and so we have here the uh graph rag local and graph rag global and this uh drift c. So these are all from from versions where everything is centralized. So I have one knowledge graph and so uh with this I can uh you know everything is local. So local, remember, in this case means that I’m going to locally search. So I have one knowledge graph and I know where to go look. So if I have one knowledge graph, I know where to go look. I can get it right 53% of the time and it takes me 34 seconds, 11k tokens. That’s great. I like that. Um, but sometimes I don’t know where to look. That’s going to drop my performance a bit.
Josh Phillips: It’s going to take a little bit more time. Uh and then if I do the exhaustive sh search where I I do both the local and the global state uh search. So I just search everything uh it takes me 232 seconds and I do get a bump here. Uh so I’m up to 63 seconds but that is now you know 700 tokens and a whole lot more time. Um, and so the idea with these two is that so what if I don’t have that one database and so now I’m splitting it across, you know, lots and lots of different databases that are disperate. Uh, and so now um I can use this sort of decentralized search or these are questions that are designed for this decentralized database. And so if I do that exhaustive brute force just read everything, my numbers go way up on this sort of data set. but it takes me 415 seconds and 880k tokens. The nice thing about Scout Rag is that I’m able to recover even inside of a decentralized database a lot of that um performance.
Josh Phillips: So, I’m able to even do better than if I know exactly where to look. It does take more time, but I I think I’m kind of trying to get into a more acceptable area. So, that’s the idea. So you can’t take I mean this is basically on whatever set that they had this is their data set uh and their problem set but you can take the the general shape of this problem and take it to whatever you have. So yeah that is the the big paper that we’re going to be looking at. I guess any thoughts on this one,
James Chung: Well, in terms of metric, um, oh, so scout rack is run on one centralized data or is it
Josh Phillips: uncentralized, so distributed. That’s the big thing is that if this were just,
James Chung: okay?
Josh Phillips: you know, centralized, then this would be really bad. But it’s dealing with the the imperfect information game problem, too.
James Chung: Okay.
Josh Phillips: Um
James Chung: And then how would it perform on a one centralized data against the the global and the local?
James Chung: Do you have any idea?
Josh Phillips: yeah, my guess is that it probably it’s kind of a you know bringing a flamethrower to a a candle lighting. It might be too much because you are you’re handling a lot of complexity here. Uh so if you can just do it with a local that’d be good or you might do it with something a little bit simpler. So I think you know uh if you have communities inside of a centralized database you know you could still do this DRAA thing I think that’s probably a good idea but maybe this you know giant complicated parallel retrieval thing you don’t have to do that I’m not sure you
James Chung: Yeah, as long as I uh the graph looks sparse enough,
Josh Phillips: know.
James Chung: then I can I think this is very applicable.
Josh Phillips: Yeah. Yeah. I think I I mean I’m generally always going to do because I want to get my prefilter down. So if I can deal with less nodes, I want to do that as soon as possible.
Josh Phillips: So I’m always going to do something like that. Um, agreed. All
James Lawlor: Is there uh any requirement that the decentralized parts
Josh Phillips: right.
James Lawlor: of the data are using the same like information schema or ontology?
Josh Phillips: No, not at all.
James Lawlor: Hey,
Josh Phillips: You could have you could have it be completely um I think this would even work if you had a mixture of graph and non-graph databases honestly. Um so you could even even do some sort of merge there. Um because you’re if you’re just taking you know zooming back out and even taking out the graph component of it you know you can do it partially on the graph and then do that merge with you know non-etworked connections to do some
James Lawlor: heat.
Josh Phillips: sort of a hybrid search. Um, and that’s actually pretty close to actually what I do. So I I do a mixture of BM25 and embeddings uh along with a graph hybrid search.
J. Langley: So probably really good with extremely messy data.
Josh Phillips: Yes. Yes.
Josh Phillips: Also known as data.
J. Langley: Yes.
Josh Phillips: I think the messier your data is and the messier your access to that data is, the better it’s going to be. All right, I have two more little papers here. Um, and uh, uh, these are just kind of like more interesting things getting an idea of of kind of what do what are the problems you’re thinking about when you’re looking with graphs. Um, and so here’s one where, you know, I’m doing this graph sort of structure, and I’ve kind of already uh uh given given away the uh uh the story on this one. Uh, but I’m going through and there are things that I end up citing. So, you’re doing uh the deep research with with Chad GBT and it searches up 300 articles, whatever. It’s getting crazy now how many they they do. They it cites maybe 20 of them. But the question is is if it didn’t site if it didn’t go through all of those articles that it didn’t site, would it have gotten to the answer?
Josh Phillips: So, was it necessary uh for it to actually do those things? And is it safe for me to prune those those sort of things that it’s not citing? And so, um so we go back here. So, one of the things that was inside of this was um you know, do I think historical performance uh are the things that I’m finding inside of this domain or this part of the domain uh useful for my questions? And if I’m starting to go through this domain, it’s got a lot of data that I spend a lot of time spending all my my um my query time, my network time querying through and I never actually site or use that data. Uh I should just delete that data, right? So I’m not wasting so much time uh because it’s noise. Um, but uh it might be the case that that is actually something that’s very critical. I I don’t know. Have have any of you guys uh heard of the pineapple story in Team Fortress 2? Um, this is a fun one.
J. Langley: No one
Josh Phillips: Okay. One person who has heard at the time.
J. Langley: Charlie.
Josh Phillips: So, this was a very funny one.
David Showalter: It rings. It rings a bell, but I can’t place it.
Josh Phillips: Yeah. So, uh, let me actually see if or the coconut. Sorry, it’s the coconut in in TF2. So, yeah, this was a um Oh, no. Am I learning this was a myth? Oh, no. Well, let’s not look at that because it ruins my my example here. But apparently uh there was a thought that there was a a a coconut JPEG that was inside of Team Fortress 2 that if you deleted it, it crashed the game because it was so important uh to you know don’t touch that thing. You know, don’t don’t flick that light switch or suddenly, you know, the power to my computer doesn’t work. Uh even if this one is not uh true, I’m sure that we’ve all been in that sort of uh redneck engineering moment uh in our lives.
J. Langley: Oh, I have print F statements with comments around them for do not remove.
Josh Phillips: Yeah. Yeah. Yeah. So, so the question is you know it can that be the case for for our graph neighborhoods too. So if we if we are doing this network sort of thing uh are there some circuits that that are liarally important. This is really interesting because this isn’t really a problem that you’re going to have in a actual relational database a tablebased you know SQL thing. it’s less likely because it you’re not bounding through those networks. But whenever you’re working with a network uh those sort of problems can uh happen more which I mean if you think of a if you think of a computer uh program as an an abstract syntax tree of what you’re executing is a graph structure to a certain extent. Um so uh it’s that is actually why you know to a certain point uh the problem is there too. So yeah uh the question is you know uh whenever you’re doing this sort of thing they wanted to isolate it.
Josh Phillips: So say if I take those those answers out uh and I don’t give them that that that stuff but I give them the cited things they got to is that enough work or is that enough uh to answer it right? So basically I’m not doing the network search. I’m just giving it the citations. Um, and then uh uh did I even need those citations in the first place? Those the first two ablations you want to do uh to kind of uh narrow down the the cases uh that were important. And then after that, we’re going to try and make it search again, but we’re going to take away the liinal network and see if it can get there. I don’t know if that does that all make sense what what they’re we’re trying to do. I’m saying it poorly. I know. Okay, see they’re basically trying to to put their their arm, you know, their their boxes around it to make sure that this is actually a problem. And what they found is that the site is see you do need that cited evidence generally at least with what they are are testing.
Josh Phillips: So you can’t just take it away. The model didn’t just know it. Um but a lot of times if you do take away the middle space, uh it collapses. Um, so thought that was an interesting sort of unique problem. I don’t have much more to say about that. I think it’s kind of obvious what the thought is here. Um, any other thoughts on this uh paper? All right. And so now I have a third one. Um, and I think this one is going to get to some of the stuff that you asked Walter earlier on. um which is how do we verify our triples against the schema and source ground evidence uh whenever uh we have some of a closed world uh sort of uh loop which here we do so in this case um my closed world loop is this I have this book this book says what this book says and so there’s a certain class of information that I want to be able to say uh whatever I’m saying does or does not agree with whatever is in the official corpus.
Josh Phillips: Now, I might also say that it’s totally okay uh to to not be um you know, going along with this book uh because, you know, it’s fun to live life instead of, you know, be a rules lawyer. But I do want to know when I’m deciding to diverge because that might, you know, start to make things happen over time that I hadn’t anticipated before. um especially when I’m adding a bunch of LLMs that like to hallucinate a lot and lose the plot. Um and so so how do I understand uh whether a claim or something inside of my system is able to ground back to some sort of source of truth? And so here we can say, you know, we’re using the the old key thing again. Um we’re guarding the Bane Warren key uh and and is that something that’s actually valid? And so I’m going into here I uh inside this loop they’re using a lot of trajectories and memory banks. Uh so some way of of uh connecting to past uh sort of areas and past groundings that have that have occurred and seeing if it can get any information from that.
Josh Phillips: We talked a little bit about this in our our memory paper series I think last September. Uh so it’s using that trajectory sort of uh thing. Uh and one thing too if you’re using a knowledge graph too as you are improving the data inside of your knowledge graph uh you can start using that uh as this sort of memory system uh for your agents as well uh in an interesting way that helps. Okay, so it’s looking at that. It’s looking for some hints uh of memory that I can go and use to to search this wide graph. I’m going to come up with a plan for how I’m going to query it. And then I’m going to think and uh then uh decide, you know, if this thing’s true, cool. I’m good. Uh I can get out of the loop. And if it’s not, I’ve got to figure out how to go and find uh the information to either prove that this thing is true. So what is all the information that that supports this claim?
Josh Phillips: Uh but more importantly, what are the defeaters for this claim? So what are all the things that disagree with this claim? um and then uh use some combination of that loop in order to come out with your your ultimate decision point. That’s kind of obvious, but um they got sort of a loop here that you can implement, which I think is useful. Um and so uh what they’re using here for the the internal structure is making sure that if you have any metadata on that claim, you know, where did this claim come from, where was it sourced from, who uh made this claim? So yeah, I think Jay, I think we’ve talked about this a few times. You know, there are sometimes uh there are issue tickets that depending on who made that issue ticket, I have a certain level of belief in the uh the truth of that issue ticket. Uh you know,
J. Langley: Yep. We’ve done that. Um, our IT folks know that if one of these five people come to you,
Josh Phillips: so yeah,
J. Langley: that that is a real It is actually on fire.
Josh Phillips: right. Jack says it’s a problem. It’s a problem. Yeah. So um see your metadata. So things like that. So so reputation, you know, where it’s coming from. Uh you know, what are your neighbors that I can look at? So here’s this thing. What are other things around me? Where do those that are valid site to? I should go look at where those things are to to start my neighborhood trip. So we’re almost kind of back into your scout rag again. You’re just doing it in a different way. Uh and then can I do like a bre first search to uh find interesting relationships in a useful way? The nice thing here is all this is that I can use a a model. They’re actually really good if you’re the tool calling versions. I’m basically using a model to do all of this and holy moly, it is really cool.
Josh Phillips: I’ll show some of the stuff that that it came up with. Um, and uh some other things that it’s useful uh is to connect it to external tools. This is going to be external to your knowledge base but internal to your company. So you can just like do an internet uh an intranet. Um but you can also use it to go out to you know the wider internet. And so I did a version of this where I was kind of doing it. Um we had this this uh case of um I think this is a good time to talk through the character sheets. And so if I go here, let me find I need to get better with my titles because now I have all of my titles are named the same There we go. Oh, we don’t need this anymore. Okay. So, I’m going to go here to my characters. Um, trajectory. And so, I’m going to go to my Jevica sort of things. I’ve got a whole bunch of runs for her.
Josh Phillips: And early on, the very first few runs, I think her character sheet was very light. Yeah. So, here we go. So, early on, uh, I had basically I I had mined everything I could possibly get about this character out. And so, I was only using things that were were solid claims about, uh, you know, I could I could mine directly from the source. And so, I knew that she had this inventory item called Jebica’s arm, and it did some stuff. And she was able to attack with that arm. And I knew that she had this much gold. And I knew that she had two spells that were to her name because, you know, you have this entire book, it’s going to mention the interesting things about her and it’s up to you to kind of figure out what the rest is. And that was useful. Uh, but then, you know, in Dn D, they’ve got uh each uh uh spell that you have ties to a spell slot. And I think both of these are like a level five spell slot or something like that.
Josh Phillips: Uh, but she has nothing else to find for any of the other ones. and my GM was doing a good job and it was checking and making sure she had the spells she said she had and I didn’t have anything filled out and so I had lots of hallucination struggles with this where they’re basically fighting with one another. Uh let me see if I can find some examples of that. Uh, and so basically what this is, um, part of my, uh, thing, and we’ll get into this now that we have time, we’re kind of at the end of this, um, where I have the ability, uh, to have my my AI play a a um, a game master and another one play one of the characters that I have, following all their intents, kind of in an unbounded loop, and see where they go. Um, and so here, uh, you can see, uh, they’re kind of going through and she ends up like trying to cast a spell later on and the GM ends up getting very salty with her and saying like, “You don’t have anything prepared, so you can’t do that.” And it keeps like they keep kind of arguing back and forth and it leads into it kind of
Josh Phillips: devolving. Um, and they they can’t get past it basically and the thing falls apart. And so I really needed at this point to find a way to Okay, so I’ve got I want to use things that are grounded, but I also need a way find a way to um get back and and fill this in in a way that is interesting and going to be helpful without introducing a lot of hallucinations. Yeah. Uh what’s up,
David Showalter: Uh,
Josh Phillips: David?
David Showalter: I don’t see a problem. You completely modeled the accurate like high school level D&D campaign. So, it’s 100% accurate. You You nailed it.
Josh Phillips: Yeah. Yeah. Uh and so yeah, uh what I ended up doing is I did very much what we’re we’re seeing here. And so I built an agent that will go through and it number one I have a llinter. So I actually have a JSON format that is the full 5e data sheet. Uh and I have I wrote my own llinter that takes in that JSON.
Josh Phillips: And so let me uh I don’t have a good way to to pull it up here. I’m not going to pull up to Jason. But um uh that’s that’s not a good talk thing. Bad programmer. Uh but basically it has uh you know all of the things that are missing. So I noticed that you’re like a spellcaster and you’re missing spell slots. Uh I’m not going to allow you to commit this to the database until you fill that in basically. And so then it basically gathers all of those things up. What’s not filled in? Uh, does it not have the height? Does it not, you know, have that they have blue or brown eyes or no eyes because it’s D&D? Um, um, then how do I efficiently go and find what to fill in without it becoming sort of mud, you know, where everyone’s named Lara and, you know, they got, you know, the the things that all these LLMs come up with that’s the same basically. And so I want it to be interesting and to have diversity uh from a a narrative standpoint to have people with with interesting uh and hope hopefully conflicting interests because that’s how you get interesting uh narratives.
Josh Phillips: Um and so you can see here uh it it will go out and basically come up with all those things. It’ll mine as hard as it can. So it looks harder and harder and harder. Is there anything that I could find? you know, maybe something that is indirect and I couldn’t originally gather that has to do with this person. Can I go look online for how people build stuff? So, like for Jebica, um, I’m going to look into something that’s a little bit further in, a little bit more recent. Um, we can see now that she’s got everything filled in. So, it went in and said, you know, okay, what do do mages at this level, what do they kind of have? Uh, you know, what’s their proficiency generally? you know, what sort of things does she end up doing that might have led to her there? You know, what are things that she likely has given her background? So, I’m going to go look up at builds online of how other people have built this.
Josh Phillips: So, finding some way to ground it in a way that makes it sort of interesting uh and kind of giving them a nice spread so you have a a viable character um or in some cases a nonviable character is perfectly acceptable too uh if you give it the flexibility to do that. Um, and so yeah, you can see here it’s done that and then it’ll mark mark tick tick all the stuff down. Uh, and same thing with like the inventory. It goes and finds things and then it will make up stuff uh to that kind of match the narrative. Uh, which I like a lot because that’s kind of what you do with these, you know, you don’t stick to just the book. The book’s a starting point. Um, and yeah. Uh, what’s up
James Chung: Um, so, uh, the agent or no, it’s not an agent, but LLM made a response based on the information. Um, and I I forgot the name of the character, but so but she has tons of information when we look at the graph.
James Chung: So what is the context? What pieces of information is the LLM given to make up a decision?
Josh Phillips: Yeah. Uh, so here’s a this is kind of getting into stuff that I’m not talking about tonight. Uh, well I in the papers, but I will talk about right now. Um, is uh I generally have a loop in all of these LLMs. uh for my specific case because I am doing a tick based system where I want to update state and I’m also I want the the agent to act in character. So I don’t want to give the agent too much to do uh because I want it to not be focusing on like I need to call this tool, I need to update my character sheet, oh and I need to to to give a a you know authentic performance too. Uh, so I actually attach agents on either side of their response. And so she has a a I call it the chronicler and the the memory. And so before there’s this recall step where it basically goes out and I have something that goes and uh gets all of the relevant information.
Josh Phillips: And so there it’s kind of sparse. But let me go to a more recent one. I’ll go to the shaorn one here. So I’ve added a bit to this and so it’s actually going out and its entire job is just like the scout rag to go and fetch information. Uh, and so I do I always do something where it’s the it does a query first and there’s a non-agentic complete you know go get me BM25 a bunch of stuff go get me uh a graph network first call sort of stuff to kind of seed the world get me the most relevant communities maybe you know based off of the stuff and then I hand that to my scout and then my scout has to decide what else to get and I give it additional tools so I have it give it the ability to read this sort of campaign journal where it’s looking at the high level sort of of stuff that’s there. Uh some sort of a chronicle. And right now it’s at the very beginning.
Josh Phillips: As you can see here, all these things are empty. Um but I’m going to hop now to a little bit later and it’s going to call those things again and it’s going to start surfacing extra information uh inside of here. Let me find one where it’s actually calling stuff. Here we go. So now you can see it’s got the campaign. And if I look into this campaign, it has the scene that we’re currently in. So that’ll persist, you know, across compactions, uh, and different stuff like that. Um, it’ll have the roleplay journal. So this like has like little snippets of stuff that has happened. Uh, so you can kind of recycle your your context to keep cost down and stuff like that. Uh, it has the ability to go grab the character sheet. So, if it’s like in a roleplay moment, it can go grab, you know, spells. It can go grab information about the other characters, uh, within reason, uh, stuff like
James Chung: Okay. So I just want to make sure if I understood correctly.
Josh Phillips: that.
James Chung: Um so you keeping track of the trajectory of the past actions and
Josh Phillips: Mhm.
James Chung: then store it in forms of say just a markdown file and then you’re going to make a curate to the scout database uh no scout rack system in order to retrieve some additional information and then use those two pieces of two both of those things to make uh to generate the response from the LLM. Is this the correct way of inter interpret inter? Is this the correct
Josh Phillips: um so it’s actually the the the um trajectory I do store as a graph.
James Chung: interpretation?
Josh Phillips: Um so you can store that as you know a graph and it leads to an artifact and that artifact actually I usually do JSON uh right now but um not markdown files for sure it is it is giving a markdown response here just because it’s it’s an LLM talking to another LLM and I want to kind of summarize all that but it’s generally if we go look at so I can go here and it’s there’s a reason it’s got a toggle off but you can see it’s just kind of doing uh it 86,000 input tokens that goes into it in total and its job is to take that 80,000 86,000 and form it into you know what does the LLM need to know and this just happens every time under the
Josh Phillips: covers.
James Chung: Okay, thank
Josh Phillips: Yeah, that is uh kind of what happens here. Uh and it looks pretty uh uh I’m just going to kind of show now uh just to kind of see it in action. Um one thing that I won’t have is I won’t have that full uh input on uh on my end. Uh so it’s missing that that that initial input, but it does have an after uh whenever I’m doing it interactively. But I can show a little bit of what uh this looks like here. And so I’m going to uh start a new campaign here. And uh can you guys see the screen enough to read it?
James Chung: Yes.
Josh Phillips: Okay. Oh, I know what uh All right. And so it’s gonna start me into a new campaign here. Um, and uh I’m just gonna say I want to uh set my uh play as one. Let me turn my sound off. The NPCs Okay. And so it’s going to have the 42 NPCs that I’ve got here.
Josh Phillips: And each one of these, you know, none of them had a full character sheet defined out. Uh, and it’s basically backfilled all of these with that full sheet and kind of pulled a a starting situation for them. So at the start of day one, Majora’s Mask style, you know, where are these guys at and building out that trajectory for them. Um, and so there’s lots of different folks here. We’ve been talking about Jevica, but I’m going to play Dorg here since the first one up here. Uh, so, uh, Midor, we’re going to start getting it into into gear. And so now it’s going to go pull that sheet for me. and start just filling things
Christopher Bales: Is that a warrior?
Josh Phillips: out.
Christopher Bales: Is that an ogre warrior? Yes.
Josh Phillips: The idea with all this is that I want to um uh be able to to have all of these these characters play like they’re the main character. you know, you have the main character syndrome in a lot of these things, but what if everybody had their own things uh that it is that they want to do?
Josh Phillips: Um, and so it’s now setting this up. Uh, so DOR has uh, you know,
Walter Freeman: Was each of those
Josh Phillips: its wants and needs. Yeah. What’s up?
J. Langley: Did you turn your volume
Josh Phillips: I see I see
Walter Freeman: called character lines uh a separate agent call
Josh Phillips: hands.
J. Langley: down?
Walter Freeman: with uh with a system prompt and um MCP
Josh Phillips: Oh,
Walter Freeman: overhead?
Josh Phillips: man.
J. Langley: That’s what it was.
Josh Phillips: I I didn’t have my volume on that entire time.
J. Langley: That’s what I was trying to tell you.
Josh Phillips: Thank you. Thank you. Sorry if I Sorry if I talked over anybody. I’m sure I did. What’s up? Sorry.
Walter Freeman: Oh, so there were a bunch of lines that said character colon something. Were those like tool calls where one one model was spinning up an agent and invoking
Josh Phillips: Uhhuh.
Walter Freeman: another model with MCP overhead and system prompt overhead and so on or were they like Python scripts?
Josh Phillips: Uh, no, it’s tool calls. Uh, so all of this entire system is agents all the way down.
Walter Freeman: Okay.
Josh Phillips: Uh so so um I’m actually I my uh my system where I’m I’m simulating it against each other is actually two open code uh servers that are talking to one another through a proxy. Uh so they think that they’re,
Walter Freeman: Okay.
Josh Phillips: you know, they’re both the uh the the AI, which is
Walter Freeman: Yeah. So, so,
Josh Phillips: kind
Walter Freeman: so all of those calls were a each one was a separate model
Josh Phillips: um it is interled.
Walter Freeman: call.
Josh Phillips: So it’s not a separate not kind of but not really. So there’s an exeutor that’s able to uh incrementally uh execute calls off the stack. Um, but it’s still one sampling call, so to speak. So, it’s kind of weird, but so yeah.
Walter Freeman: Okay.
Josh Phillips: So, we have DOR here. Um, and so yeah, Dorg is doing stuff. It looks like he’s got some shop.
Josh Phillips: Uh, so Dor growls a stranger. Yeah, something like that. The door is a troll. uh or an ogre or something like that. Yeah. And so it’s going to do some stuff and basically what’s happening in the background, so you don’t see it in the actual thing, but all of this sort of stuff that is occurring in here where I’m showing it sort of fully out, uh this is happening in the background for these agents. And so the the the question is is how can we kind of make all this stuff work together in a cohesive way? Um and so It’s put him like straight into a situation that matches his actual situation. So, he is a a guard that is charmed uh inside of uh to guard a potion shop, you know, against his will essentially. Uh a troll who’s had like a hex or something cast on him. Uh and uh yeah, he’s staring at the stranger’s hand. And yeah, I’m not going to spend too much time uh doing the art RP stuff here.
Josh Phillips: Uh but I’m going to uh kind of just do nothing. And so we’ll see another area here of of what you’re thinking about with the the game master’s uh uh intent. So So DOR has its intent, you know, the things that it wants. It wants people not to steal stuff from the shop. The game master’s intent is for things not to be boring. And so it’s going to start causing problems. uh if problems don’t happen. Uh so I’m going to say uh doorboard and so there’s some some problems that are starting to appear and so it’s injecting that there are some guys around um and you know there’s some some interesting things that are forming and generally it’s going to be pulling the stuff from what’s actually around. That’s the nice thing about the network is that uh it can actually know, you know, these NPC NPCs are near DOR here. And so each of those NPCs have their own things that they want as well. Um, and so it’s going to keep moving uh like that sort of thing.
Josh Phillips: And that’s kind of what this simulation is here. So that’s door, that’s one guy. Uh, but here I have that loop running for all 42 at once. And so that’s what this little chronicle thing is here where they’re going through and how do I make them sort of integrate with one another? Because the idea of what I want is for me to be able to uh you know be playing DOR here and the world moves around me in a way that’s interesting and immersive uh and and it creates you know infinite story generation. Uh I want to be able to step into say okay so that’s Dorg. I want to be able to step into Iltamar. I want to be able to step into this person’s life. I want you to step into this person’s life and have it feel like a whole new world that I can just, you know, it I could probably spend a lot of time just playing every single character in the city and find something new and interesting. And I think it’s very hard to do without something like these graph structures.
Josh Phillips: So
James Chung: Um earlier you mentioned that you are processing the 42 characters with just eight LLM calls and I was wondering about that because what I
Josh Phillips: Yeah.
James Chung: initially assumed was each you’re going to have like 40. So if there are 42 characters then you’re going to have the 42 LLM agents where each one of each each LLM represents one character but um so how does how is this structured so that how how does it like 42 character actions are represented by the eight LM
Josh Phillips: right?
James Chung: calls.
Josh Phillips: Well, it’s eight LLMs each or eight LM calls each for each turn. It’s the same way.
James Chung: Oh,
Josh Phillips: So,
James Chung: for each.
Josh Phillips: you know, Yeah. So,
James Chung: Okay.
Josh Phillips: it’s kind of like I talk about here. So technically a turn for this one character is is I think seven calls. So the the call here for this one character turn is there’s the preall so the scout then the actual agent and then the chronicler. The chronicler’s job is always to just like write what happened uh so that that it gets persisted.
Josh Phillips: And then there’s the the game master’s scout, its response chronicler, and then some other stuff that it does. The uh the the pot stirer agent is in there as well. Um and then same thing for these guys. Uh with them, because there are so many, you know, I’m really letting a big model do a lot of work on the one-on-one thing. But here, I’m trying to kind of batch it out. So, I’m doing lots of little tiny calls because I kind of want to give everybody the ability to uh set their intents. And so, I’m going to start a line here. So, I’m just going to have it advance now. And it’ll take, you know, a little while to do it, but we can you can see it kind of start uh going through stuff. And so, there’s 42 agents that are starting their watch. And there’s going to be an agent that kind of orchestrates all of it at a high level uh to say like okay I’m going to decide what are the latent conditions that are going to start happening at this beat.
Josh Phillips: Uh and you can actually see it in my atlas here. Uh I added the concept of pressure which is basically you guys are being boring. I’m bored. And it uh starts adding in here. You can see this little little tiny red dotted line. might be hard to see, but it adds in an area of instigation where anybody inside of that area uh basically starts getting, you know, starts doing stuff. So, it starts nudging them in in order to do interesting stuff. And I’ve got another thing called heat. You know, what are the areas of heat that inte integrate with the pressure? It’s just sort of sim stuff that you’re doing. Um, and so it’s going to start deciding how to do those things. And so we can see here in our my queue I’ve got it so that I have I think 16 at a time that are running. And so I have my beat orchestrator and he’s just kind of always looking at stuff. And right now you can see these agents are all doing their memory scoring.
Josh Phillips: And so they’re looking through their memory and having somebody go and kind of look through the things that have happened in past turns. And so let’s see where our goodie Dorg what’s Dorg been up to in this in this uh run. So we can see here that you know at beat five Dorg is is uh smelling so yeah he has the keen smell and so he’s sniffing around for thieves uh at beat five and that’s in his recent memory and so he’s going to be probably it’s going to lead him to kind of probably you know messing with somebody tracking somebody he’s suspicious the next person he interact Max is probably not going to have a good time. Dor is not going to be in a a a nice friendly mood is my guess. Um Dor, I need to get There we go. And so yeah. Um and yeah, I’ll leave this up so you can So you can see uh deliberation. So I have somebody’s he’s now trying to figure out what to do. So I’ve got my memories.
Josh Phillips: What do I want to do right now? This can be where they’re resetting intent. See, I have this intent system here where it’s kind of what are they deciding what goals they want. This can last for many turns. You know, this can last for 20 25 turns if it’s something really really important or it can be something like I’m going to go move over to that door. I’m going to stand right here. I’m bored. It can be a lot of different things. You don’t you don’t always want everything to be, you know, the dieh hard action movie,
Walter Freeman: Come on.
Josh Phillips: but it also has a a win condition. So, something has to happen in order for them to come out of this or um uh Yeah. What’s up,
Walter Freeman: Do you have is is a turn or a beat the same um amount of
Josh Phillips: Walter?
Walter Freeman: time for all characters? Like for instance, suppose there’s a combat going on and and two characters go into rounds, you know, as you would do in a a traditional D and D game.
Walter Freeman: Does the system force every other character in the world to also take actions in sync or can you do something like okay these two characters are going into combat you know we will we will do turn by turn um and then a bunch of other characters who are off camera um aren’t getting rapid updates but then might have a coarse grained thing to bring them up to the future as soon as action touches them
Josh Phillips: Yeah, absolutely. I mean that that’s absolutely I think we’ve talked about this I think at one point too of you know having that you know I’m not looking at it right now so we’re going to do that a very low fidelity simulation out there.
Walter Freeman: Exactly.
Josh Phillips: Yeah.
Walter Freeman: Exactly.
Josh Phillips: So absolutely this is going to happen um uh where you kind of do that catch up you do that high level stuff you might do like a super quantized model uh you know something that’s really really fast too. So, there’s lots of different ways that you could uh um kind of get away with that.
Josh Phillips: Um but yeah, so right now I’m not doing the hard time uh dilation or sort of time step things. If I start going into um uh you know, lots of combat focused things, then yeah, that’s obviously going to be important. But for me right now, I just want to, you know, have it be interesting. You know, have it have it have interesting stories. I wanted to to explore and say like, okay, so what if Dorg ends up crossing ways with Jebica and somehow in this version
Walter Freeman: Mhm.
Josh Phillips: they’re best buds. She undoes his charm and he joins her instead,
Walter Freeman: Yeah.
Josh Phillips: you know, and suddenly we’ve got like, okay, well, this is interesting. And that’s I I want to optimize for interest right
Walter Freeman: Yeah.
Josh Phillips: now. All right. Yeah, we’re kind of in the the loose loose moment, so I guess any uh last Oh, yeah. What’s up, Chris?
Christopher Bales: Hey, can you hear me? Hey, so um what you’ve done here is insane.
Josh Phillips: Yeah.
Christopher Bales: Uh but uh from a from a practical standpoint outside of this insane project you have going on. um what I see when I look at a neural graph is extremely interesting, but I also find that I can’t get um sort of like a a quick shot information space from it in a meaningful way that I understand. So could you just speak briefly on how you can take a neural graph which has you know the way the relations between objects and how I can extract what’s the basic extraction from that layer that you can put on an agent for example if that makes
Josh Phillips: Um so it it it obviously yeah it does uh it obviously matters quite
Christopher Bales: sense.
Josh Phillips: a lot on what your ontology is and what you’re doing but I can give some examples that um are are relevant in my life and I’ll try I’ll try and do a few things that are not related to this. So so think about um a graph. So you you have your code uh codebase is at the end of the day it does end up being something that’s kind of like graph structure.
Josh Phillips: Uh and so I have certain parts of the codebase that are uh related to bugs and bug reports that I have. So so are there changes that that occurred inside of my codebase that are tied to uh a a inordinate amount of bugs? Are there places in my codebase that haven’t been changed in 20 years? There are no bugs. I suddenly have the ability through my graph structure to query and get a heat map of problem children that I needed to potentially go refactor or something like that. Uh and so you take the same thing and and take that to uh maybe I have the same thing with a certain breaker in my house. Uh and I I have the ability to kind of get my power information. I have some sort of node inside of my cluster that always is failing and a lot of things can be tied back to that thing. I think Lauren uh talked a good bit or mentioned uh security. So network security. Do I have a lot of of of un unauthenticated weird behavior that’s coming from a single user or a single region or a single access pattern?
Josh Phillips: Um, this is very hard to find whenever you’re just looking at a table query, but if you know how to query it from a relational standpoint, you can get a lot better data. Um, as far as, you know, making sense of it visually, you know, the nodes and edges are nice. They’re kind of cute pictures, but, uh, it’s just the underlying data source, too, that’s very
Lorin Bales: Yeah, Josh, my uh my background is actually in the codebase itself.
Josh Phillips: useful.
Lorin Bales: So, the ontology that you brought up earlier with the abstract syntax tree. So, like if you’re searching for vulnerabilities in the code, it’s all about like how did you modify that one variable would be a good example, right? There’s only three lines that do that. Well, which one is really the the source of the issue, right? So, you’ve got a huge graph, right, when it comes to a codebase,
Josh Phillips: Yes.
Lorin Bales: but you know from looking that the v you know where the vulnerability is from the start because you’ve scanned and you can get down to those three nodes.
Lorin Bales: That’s that’s kind of how I would think about it. You can drill the query inadvertently drills all the way down, right? Yeah.
Christopher Bales: I I guess uh so are are you at the at the onstart on the at the onset are you setting up like a visual graph is that like you’re like when you set up like the interesting nodes or I guess the interesting um edges there there’s a relationship between the objects and you like I guess you you look at the edges that interest you the most for a problem you’re trying to solve and Then from there you visualize it. Is that kind of like where you start?
Josh Phillips: Uh so when I’m actually solving problems I generally don’t visualize it. The lot of times that I visualize it is when I need to explain what’s happening. And that’s usually something where you know whenever I’m taking this this thing to somebody who doesn’t know what it is and so you know if you just do a blank visualization a lot of times so like this is me uh so for instance I did for these papers I took them and I turned them into a graph and I said hey what are the things where you know these papers have uh things that are relevant across the whole set of them and so you can see here you know here’s the why neighborhoods matter thing here’s
Josh Phillips: sharp and here’s three that I didn’t uh cover that I was you know but are still important to me in this area which is the ontology to tools executable oncologies and the game knowledge management system and so you know this thing right here is not super useful for me other than for me to say okay I can see that there are clusters here and so now I can go look and I say okay there’s something here obviously there’s a fan out why is there a fan out here Okay, there’s lots of things going to textual diversity. This is a paper about textual diversity, I guess, uh, in some sort of case. Um, and so that’s sort of useful for telling a story quickly. I want to kind of like do a graph rag on a whole bunch of things and not look real hard. Uh, it’s it’s kind of really if you have a good pipeline, you can get that stuff. Um, but the other area is for going back to here. is I’ll do stuff like you know this.
Josh Phillips: So I will I will use it to to do a curated meta model of of whatever the problem is that I’m trying to solve. It’s almost like when you’re doing uh you know software architecture diagrams and stuff like that. You’re trying to tell the story making understand of what this highle sort of graph structure is. Um but you know if I’m you know I don’t look at the queries that are going uh on the back end for this very often. Uh a lot of times I’m looking at actually sort of tabular data. So I take I take the graph return and I actually turn it into a tabular piece of data sometimes for me to look at and aggregate those metrics up. I don’t know if that answers your question but it’s a That’s what I got.
David Showalter: This was great. My uh brain’s a little fried tonight, so I didn’t follow everything, but thank you. And I’ll be looking over it some
J. Langley: Yeah.
David Showalter: more.
J. Langley: Don’t forget to bug me to post the video.
Josh Phillips: Hey Jay,
J. Langley: Yeah,
Josh Phillips: post the video.
J. Langley: I know. This has been
David Showalter: What’s been some of the most most fun errors you’ve had in uh trying this for your campaign?
Josh Phillips: Oh god. Uh oh, there’s so many. Uh it’s definitely it’s definitely LLM hallucination because I’m using the LLM a lot in the pipeline and there was this whole uh so we’re talking about the time where I was backfilling uh the information on these these guys and I basically I found like the first I was like I want to focus on these 20 you know the first subset of the Delver Square group and I said I want to fill in the other people. Let’s fill in some information. and I need some bench depth uh to to my thing. So So look at the books and go find the next 20. And it came back and said, “Oh, I found the perfect 20 and this is great. I’m going to build some bench depth for you.” And so I said, “Wow, it’s really doing this. I don’t have to do much at all.
Josh Phillips: I love the Gemma models. They’re so good at this.” And it just kept going. It kept going all the way to 100. And I was like, “Man, this world’s great.” I showed it to uh to Birdie and Charlie who know the setting. They’re like, “Oh, wow. I never heard of these guys. We didn’t guess we didn’t do very much of the setting. I was like, “Yeah, they they’re going through it.” Because the first 20 were great. They were grounded. They’re perfect. You know, everything was solid. And I said, “All right, we’ve got a system. We’re going to let it go.” 100% fabricated. Every single little thing about it, but it it labeled it bench depth. It’s a tier three bench depth pick. Thank you, Claude. Thank you so much. So, it took me about two weeks to get all that crap out of there. I’m still fine. I still find the word bench depth. I have I have a search term for bench depth, so it doesn’t appear anymore.
J. Langley: So,
Josh Phillips: Uh
J. Langley: so when you read articles about you can do that, but please don’t talk about the trolls or whatever, you can see how that kind of what the heck in that training data somewhere just swung it,
Josh Phillips: yeah.
J. Langley: you know. Well, for you, you kind of told it, but that’s kind of
Josh Phillips: Yeah.
J. Langley: fun.
Josh Phillips: I think that’s the most painful. But there’s there’s a lot cuz I mean I’m I’m I’m using these in ways that you shouldn’t because it’s for fun, you know? So I get to do some of the dumb stuff. Uh what’s up,
Lorin Bales: Yeah. I’m just curious about like your hardware,
Josh Phillips: Lauren?
Lorin Bales: your tech stack. Like you said, you use the Gemma models. So like what are what are the what are the the big pieces of this entire architecture? Like I don’t know if you’re using rag or using lang chain, you know?
Josh Phillips: Uh so the big uh the big pieces um I’m not using lang chain.
Josh Phillips: I’m using basically my own own system as far as all of that. Uh so raw realm hooked up. Uh I do use open code a good bit uh for some of some of the side stuff just because I like being able to
Lorin Bales: Okay.
Josh Phillips: hook up against it. It’s got lots of little plugins that I can use to basically completely control the runtime. Um, so I use that. Uh, then I have lots of different little servers. Uh, it’s all Kubernetes. It all runs on a Kubernetes cluster. Um, I have I think five GPUs that are dedicated to the thing. Um, and uh, it’s using uh there’s an embeddings and a re-ranker and it’s multimodal. So it’s multimodal.
Lorin Bales: Ah.
Josh Phillips: I don’t know if you remember back in July we did the multimodal embedding. So I’m using that.
Lorin Bales: Mhm. Mhm.
Josh Phillips: It’s very useful. Uh and so a lot of the times it’s going to return lots of queries and I will rerank and throw out half of them uh based off the semantic reranking.
Josh Phillips: Um and I’ve got a big model that takes four GPUs and is
Lorin Bales: All
Josh Phillips: a very fast very big but uh then a small little fast dumb model which is my Gemma model. Um, and yeah,
Lorin Bales: right.
Josh Phillips: I have it connected to Foundry,
Lorin Bales: Very cool.
Josh Phillips: too. Uh, this is like a virtual tabletop. This is a fun thing I didn’t talk.
Lorin Bales: Yeah, I saw that.
Josh Phillips: Yeah. So, so it it actually has a a complete virtual so it’s like roll 20. And so all of those things, you’ll notice here’s Jevica. It actually it’s uh I’ve got the map and so it’s pinned to she went to that that thing and it moved her on the grid. There’s lots of little things in the background. I’ve got about uh I think 60 pods in total across this thing. So it has a SAM it has remask the thing I showed where it’s doing the the segmentation. So yeah.
Lorin Bales: Very cool, Josh. Thank you so much for putting all this time in and and talking to us.
Josh Phillips: Yeah.
Walter Freeman: That is super cool.
Josh Phillips: No problem.
Walter Freeman: I look forward to seeing this in person.
Josh Phillips: Absolutely.
David Showalter: Hey,
James Lawlor: How will deal with like source data that doesn’t neatly fit into
David Showalter: I’m
James Lawlor: the
Josh Phillips: Um, so my base ontology,
James Lawlor: ontology?
Josh Phillips: it’s almost impossible for it not to. So I have I have lots of ontology. I I mean there’s each graph that you have, you can project a new one into. And so the very base one is basically like here is a source data. Uh here is a claim from that source data and here is a semantic node that is aggregated from a set of claims. That’s it. And so it’s always going to fit into that. And if it doesn’t fit into a graph with that ontology, I can either change the ontology or just make a new graph. It’s fine. It’s not that big a deal.
David Showalter: Hey, uh sorry to go completely different subject, but uh few minutes ago.
Josh Phillips: Oh, I saw the picture thing.
David Showalter: Yeah. Yeah. So, I’m I’m probably gonna I’m probably going to hop over there and see if it’s pure hype or if they’re actually showing some kind of new hardware. So,
Josh Phillips: Is it an
David Showalter: yeah.
Josh Phillips: orb
David Showalter: Well, it said surprisingly large. So, you know, they said they were working on the Yeah.
Josh Phillips: thing you go into?
David Showalter: Not the thing that goes into you, the poorly worded version. Yeah.
Josh Phillips: I don’t know about that one. Very
David Showalter: Hey,
Josh Phillips: cool.
David Showalter: thank you, Josh. This is great. And uh see you guys uh see you guys soon.
Josh Phillips: All right. Take care. All right. Uh, any last call questions?
J. Langley: No, that was super super cool.
Walter Freeman: Yeah.
Josh Phillips: Cool. Well, thank you folks. Thank you for letting me ramble and show off my nerdiness.
Walter Freeman: Yeah.
J. Langley: All right.
Walter Freeman: That was super cool to see.
Josh Phillips: All right, take care.
J. Langley: See you.
Josh Phillips: Bye.
Walter Freeman: See you.
This editable transcript was computer generated and might contain errors. People can also change the text after it was created.