Notes provided by Gemini
J. Langley: Let’s see. I’m not sure how I know David said he was he was out tonight, so this might be a slim uh a slim crowd. Uh I it was a pretty interesting uh set of things after I got into it and started, you know, looking through, okay, so here are these models that were used in this paper. Um yeah, they kind of sucked at what they were trying to do. Uh so initially I took the set of models um I looked at how they’re actually working you know what is their architecture um what data were they trained on you know things like that um which that I got some really interesting um information on that one um and then uh took a quick look through what are the current state-of-the-art models being used for things like this um and I did come across a bunch of other data sets that might be interesting for anybody else working robotic stuff. Uh there’s a and some of them are agriculture-based uh fruit trees, things like that. And then there were a lot of other uh links to uh other more urban data sets, things like streets or houses or um factories, you know, things like you would see on a on an Amazon, you know, hey, I need a robot to go pick parts or pick stuff and, you know, all that kind of stuff.
J. Langley: So that was that was kind of interesting um to see that. But let me share and then we’ll get going. Let’s see. Share the whole screen. Why not? What could go wrong? We just kind I only have one screen anyway on this laptop. So, um so as usual, um I made a markdown file. Um, and then for those of you that were a little later hopping on, I took the markdown file and I gave it to uh Claude and I said, “Hey, can you turn this into a slide package?” Um, and it converted all of my text into bulleted lists and made it worse. So, I had to go I had to go figure out how to uh go the opposite direction and revert all of that. So, um, a lot of this is based off of the same, uh, it’s based on the paper that went with the data set we covered last week. So, if you were here last week, I think everybody was, um, so we’ll skip that.
J. Langley: A lot of the I’m not going to reread all of that stuff, but, uh, basically a data set, um, with a lot of video feeds, a lot of, uh, uh, oh crap, I’ve already already missed it. inertial measurement unit, I think. Uh anyway, things like that. Okay, I got it right. Um odometer things, uh encoders from wheels, uh things like that. Um so that data is all there. Um, one thing that uh I went and did another look at later that was interesting um is that the GPS they So the question I had was they were they were evaluating where the robot thought it was with where the robot actually was, you know, as they were driving it through. Uh and so they well and of course they ran all that data later to see you know post post data capture what each model was doing. Um the question I had was well how did they know where the robot was if GPS isn’t working um or has a lot of error in it. Um and you can’t you know that’s the whole point is you can’t trust that.
J. Langley: So what did they use? And well, it turns out they used the GPS um plus uh some other filtered thing from I mean it got into a lot of math that I didn’t really understand. So uh that just to say I’m I’m not sure that the entire evaluation is as good as it could have been. you know, if they had some other other way um like like one of the things we had discussed having a flag go off the top of the robot that you could actually track the thing, you know, as it moves or possibly, you know, u putting the GPS PS antenna up above the canopy so that you actually have a track. Things like that may have actually helped. But anyway, um that’s what they did. So that’s what we have. Um so the paper’s main point um is that slam works great indoors. Um it’s got familiar stuff. Uh it falls apart under a crop canopy. Um, and then they the the main point behind it is you got all the leaves.
J. Langley: Uh, you’ve got wind blowing stuff stuff down on the ground in front of you. Uh, lighting gets kind of crazy. Um, and then the other thing that was just the main part the the problem is that one row of corn looks a lot like another row of corn. Um, and a lot of these well basically all of these models uh operate on being able to recognize if they’ve been in a place already. Um, so kind of I mean we do the same thing. If I’m you put me on a street corner somewhere and if based on what I can see, I pretty much can tell you where I’m at. Um, you know, because I’ve been through there a bunch. Um, especially if it’s somewhere that well I’ve I’ve been. Um, so there’s that’s kind of how all of these work and why um it just totally falls apart when you get into the situation where everything looks like everything else. Um, again, we won’t go too far into the platform and sensors. I did figure out that they were using a single camera.
J. Langley: Um, not a not a single lens, but a single stereo camera. So, there wasn’t a second camera. uh they just took the same feed and packaged it once as SVO files and packaged it again as this Ross bag format. So that was something that was a little different. Um one of the things that they talked about was uh the the reason they did this is that some of the other data sets, even the ones that were agriculture, were only done at like one time or at during one week or something. and it doesn’t take into account that these are growing things. So you come back in 3 weeks and it’s going to look very very different than it does this week. Uh so that was one of the main things. Uh the four scenarios they did. Um I took a little a little more time looking at this one. Uh the double loop uh the first one they were basically going around the field. um and just kind of making the corner at the edge of the field.
J. Langley: Um which is kind of interesting. I guess that’d be a good a good start uh starting point. Uh the part that was interesting there and I think it’s the reason they did it is you had cars and buildings and other kinds of things, you know, on the periphery. Um and some of the models actually worked much better there because well, they they have something, you know, that can show up. Let’s see. Uh the second one was just a double loop, you know, um going down and back. Uh and then the four adjacent rows um and early growth and then late growth uh were the main ones they used. Uh here were the main uh system actually the six systems uh they benchmarked. Uh one is called Orb Slam 3. Um we’ll talk about that one later. um Vins Fusion, uh which is the VIO, uh plus slam. Um that’s vision plus I can’t remember what IO stood for um in their head, but uh inertial odometry. It’s a it’s a way to do uh your how far you’ve moved uh plus vision uh along with the uh location and mapping.
J. Langley: Um, and then SV Pro, Open Vins, uh, similar to Vins Fusion, uh, whatever. I don’t I don’t know why you make, uh, algorithms or acronyms you can’t pronounce, but there’s that one and RTAB map, um, which was a graph-based, uh, piece that was a little different. Um so over all of that um their ranking uh they had three that were ahead of the others uh by by a margin. Um it’s uh it got kind of interesting where they compared uh six models across the four scenarios and you would see one model would do well in another and it would do well in maybe scenario three and then SVO SVO Pro would do well in scenario four. And so they wound up doing kind of a matrix thing to figure out who who is actually kind of the better ones. Um so it comes out these three were ahead of the others. Uh they figured out that the the canopy closure um so between the uh scenario I think three where was that scenario three for early growth and scenario four um there was a pretty measurable uh difference um which you could expect.
J. Langley: Um and then uh so some of the other thing the good bit of news was that the parts that use more of the inertial pieces um did better because they’re not you know they’re not actually using as much of the vision which is kind of kind of interesting. Uh so the whole thought is uh adding vision is a is a really good way to help. Um, but if the vision you’re adding is adding a bunch of noise, um, especially when you get into the closure part where everything looks like everything else, um, it’ll throw you. Um, so anyway, um, the whole thing was, uh, it’s not a great outcome. Um, so there’s a lot of work that needs to be done. uh some of these models um their their average uh error uh in meters was around six or seven meters in error. And so that I mean that puts you like five rows over at least. Um so that’s that was kind of the the the end point there is they’re not great. So, what I wanted to do was take a look at these uh the actual models that did well.
J. Langley: Uh pulled the paper for those, of course, threw Claude at them to see, you know, what it thinks. Um and then had a uh this was fun. I had a discussion with Claude uh because Claude refused to to actually pull diagrams from these um and instead uh built its own diagrams uh and then claimed that that was great because those were original works that it had created um and would keep me out of any kind of copyright issues. I was like, “Thank you, my original artist, Claude.” Um but anyway, um we’re gonna pull up the actual paper and look at the actual paper. So, which is what I tried to get it to do, but it just refused. Um so, anyway, um for the Orb Slam 3, um it starts off, let me go to the actual, uh this is I think this is the Orb Slam 3. Yep. Uh, and you’ll see something come up if uh that we’ll get into a little bit later. Uh, I I believe that is distributed bag of words too.
J. Langley: Um, which some of these use uh fairly heavily. Um, and that was a that was a new thing for me. Um, because I was thinking words, why are we in words? This is should be a vision thing, right? Um, so anyway, how do I Nope, no, no. Go back. There we go. Hold on. Let me see if we’ve got I’m having a little trouble tracking the uh Do we have messages? Oh, a graph. Yes. Okay. Yeah, there there were some interesting graph kind of things. Uh there’s a I believe there’s a was more of a hierarchy than a graph maybe uh under the distributed back towards piece. Um, so anyway, this thing is actually working through uh it’s got frames coming in from the camera. Um, it’s got the information coming in from the the inertial unit. Um, it’s extracting what it’s what’s called orbs. Um, it’s putting all of that stuff together. Um, and trying to build a a local map.
J. Langley: Um, and then it’s trying to figure out where to go from there. uh or build a key frame. Um and then a lot of that is going into uh this this kind of a and it’s I’ve seen this in a couple of the other ones. There’s like a local version of the model or or a mapping kind of a thing where you think you are um and then there’s also a think of it as a I don’t know if I want to call this a memory or something or an overall map. Um, and then you try to stitch where you think you are and what you’re seeing into like the overall map. Um, it seems to be similar to some fusion things that I’ve seen in other places. So, um, it’s it’s kind of interesting. Um, so anyway, um, it’s working through that. Let me get back over. But it’s it’s running uh three of these items in parallel which is um also the thing I didn’t mention um is that one of the hard things uh about doing this work uh that that these models have to contend with is doing it fast enough to put on an actual robot that’s autonomous.
J. Langley: So not only does it have to do it, it has to do it fast enough. So you’ll see some of that come into play. That’s why, you know, this this actual um model runs three things in parallel so that it’s trying to do things faster. Um so it’s got an active map. It’s trying to figure out where the current frame that it’s put together um sits. It’s trying to minimize uh a reprojection error. Um it’s also estimating velocity and other kinds of things based on this inertial unit. Um it’s putting these uh key frames and points. It’s pulling it’s, you know, removing things that uh are outside of some kind of a a bounds uh for error or possibly uh older. Um then this uh loop closing and map merging is the main hole is the main point. um it figures out where or it tries in this case poorly um for this other data set uh it tries to figure out if this thing matches up close enough to a place that it’s already seen.
J. Langley: Um and if it does think that they match well now it it assumes that that’s where I’m at and so it tries to put that all back together. Um, so apparently they came up with this thing called the Atlas, which I never I didn’t go back to the Orb Slam 2, which was uh kind of interesting. U some of the older Orb Slam stuff you can’t find because it had some interesting licensing uh associated with it. Um, so apparently Orb Slam 3 was a cut to an entirely like new set of things. Um, they kind of split from the Orb Slam 2 stuff. Um, so it’s they got an English uh, they have an active map and a set of older ones. Um but they and this is this feels similar to some of the stuff we’ve done with multimodal you know embeddings where you may have one set of embeddings that are used in more than one place. Um anyway, that’s what lets it lose. Uh so the part of the issue with these things um that this kind of uh tries to address is uh what do you do or what is this model supposed to do if it’s uh trying to track where you are uh and all that you know that kind of thing.
J. Langley: Um, what happens when it gets lost or it loses where it thinks it ought to be? You know, where do you start from again? Can you start over? Uh, hold on one second. Okay. So, part of the thing is if you go offline is kind of what they call it. Um what what do you do um when you when you come back and now oh okay now I know where I’m at. Um which didn’t happen live but I’m guessing it does. Um so that was interesting. Um which I hadn’t thought of that that part yet. Um the training data and this is this was the surprising part. Um there’s not any um it’s built on handcrafted orb features. uh some adjustment stuff. Uh basically somebody put I mean back to your old feature engineering type thing. Um it’s like if somebody sat down and put together you know a set of images and figured out how to um okay I’m I need immage but pretty interesting stuff.
J. Langley: Um, that’s why if you actually go look at this uh the repo that this is in uh for Orb Slam 3, you will find the vocabulary which is it’s it’s basically a 40some meg file that’s got all of this stuff kind of encoded in it. Um, which I debating a little bit on how much to uh how much to look into some of this stuff because I I think we’re leaving this behind. So, you get into this question where I’d like to learn some of the historic historical ways things were done, but I don’t want to spend a ton of time if I know we’re just going to bail out uh and not ever do it this way again. Um, this one was uh CPU only. Um, and it’s kind of become somewhat of the standard on a lot of this. Um, so uh it’s apparently doesn’t work well on embedded hardware. Um, but there there’s some ongoing work where people have been uh putting this on a Raspberry Pi and some other uh other kind of things. Um it’s I’m not sure if they it I’m not sure it makes a whole lot of sense to throw it towards a Jetson.
J. Langley: Um unless you’re using the Jetson part to do some of the vision uh and pull that off. Uh oh, also this one this came up a few times. Uh, apparently that is the name of a pretty good data set. That’s more of a I was thinking that was more of indust an industrial data set. Uh, but we can come back to that after a bit. Uh, the next one up uh was a Vins fusion model. Um, and this one used something fun called uh I’m going to try to pronounce it as shy Tamasi uh corner features. Um, so somebody came up with a set of features and put them into a library and now that’s what’s
Todd Page: Oops.
J. Langley: used here. Um, so yeah, uh, it’s a lot of interesting stuff. Let’s see if I’ve got that one pulled up. Uh, this one was a was a little easier. Um, this was basically uh doing odometry with a bunch of sensor fusion. Um, so basically you’re uh it’ll work with the mono camera, stereo camera, plus an IMU uh plus apparently other sensors if you can think of other sensors to put on here.
J. Langley: Um, and then it does one thread uh where it pulls all this stuff together uh using these uh uh corner features. So again, I didn’t take a ton of time trying to figure out what corner features look like. Uh considering I think we’re not going to be doing that much, uh you know, moving forward. Uh it also does uh you know, the whole loop closure uh for it’s got it’s got its thought of, hey, have I been in the same place uh again uh or you know, things like that. Um, but again, same as the Orb Slam 3, um, it’s it’s using a bag of word scheme similar to the I’m guessing distributed bag of words. Not sure if the D is distributed. Uh, but it’s also basically pre-built. So, you just take the model. Um, there’s really not a way to train the model or do anything with it uh in it to make it better. Um, I guess as long as the stuff that’s in its uh, you know, in its data set matches things you’re going to see, you may have a better a better better chance.
J. Langley: Micro area. Okay. Yeah. Are aerial vehicle stuff. Uh, there’s some interesting stuff on that data set where people are trying to uh to also include like how wind works. That was kind of interesting. Um, again, this one is a CPU uh front and back end. Um, but apparently since it’s just got one thread, it’s just clocking that one thing, you know, pretty heavy. Um, so there’s some measurement of uh latency um and how many frames per second um it can it can handle. Um the third one.
Josh Phillips: I didn’t want to go.
J. Langley: Yeah. Go ahead.
Josh Phillips: Uh so the all of this sort of like handcrafted stuff. Do you think there’s any sort of value for you know obviously people were crafting this
J. Langley: Yeah.
Josh Phillips: before but this is some sort of a problem that requires domain knowledge of whatever it is that that you’re modeling and building this for. So, you know, could this sort of thing be useful as a world model test sort of thing for some sort of
J. Langley: Oh, that would be a it would be a really interesting thing to see.
Josh Phillips: orchestration
J. Langley: Um the one of the things that you may run into that might so my thought was if you knew the data set that had started they had started off from from either images or whatever to build these uh bag of words or corner features or whatever. It’d be interesting if you could train a world model to build you know if you showed it the same thing what features would it come up with?
Josh Phillips: right?
J. Langley: Um would they match or cloak? But then you’re getting into some of the uh I don’t want to say
Josh Phillips: Yeah.
J. Langley: licensing, but uh I’ve run into this uh on some NLP type stuff back with crap. Who was it that was doing the uh uh it was open NLP before? Stanford NLP, I think. You were some of their models. Well, that’s great, but we need a license fee or something. So, some of it’s open, but some of it’s not quite open.
J. Langley: Um so yeah the architecture is open the weights might not be um but that would be an interesting you know um interesting
Josh Phillips: Oh, another thought too. So, this kind of goes back to because we were talking about that weather balloon pattern where there’s some sort of,
J. Langley: thing
Josh Phillips: you know, there’s a home base and then you have your little scout, you know, drones on the ground. And so, the other thought is what if you had a higher powered, you know, sort of thing computing these predefined handcrafted things and it could send it down to little robot as it’s watching it, you know, things like that.
J. Langley: Yeah. Yeah.
Josh Phillips: I don’t know.
J. Langley: I’ve I have I’m not sure who brought up the uh kind of the pairing of a drone plus a ground robot or a balloon or a ground robot, something like that, but that has been in my head since last week. Uh I’d call him Mut and Jeff probably. Um Not not quite sure. Um but yeah, I mean you could see uh if you had something with a view of where everything is.
J. Langley: Um and then you know some either it doesn’t necessarily it could be at a stationary point. Um you know as long as it can see where everything is. Um probably but I mean that’d be an interesting an interesting thing. Um yeah that’s so uh even so even if we do you know abandon the current approach of having handcrafted bespoke features um things like that um and move to another you know approach there should be some way to evaluate well and I I think you’re right on the it took a lot of domain knowledge to build that stuff, you know. Um, and well, I don’t know. The interesting part of it is I don’t know that the domain knowledge was about where the thing was going to be operating or whatever. It may have been more along the lines of what is it in an image that tells me which way something is moving or that one side is closer than the other or you know, maybe. I don’t know. I’m just thinking. But yeah, good thought.
J. Langley: Good thought. And I heard a sound. I couldn’t couldn’t quite tell if that was a hand raising or a
Josh Phillips: Oh,
J. Langley: uh Okay.
Josh Phillips: that was uh that was me raising my hand. Yeah.
J. Langley: Okay. So, I know what that sound is now. Just could I was just going to blow right through it because I couldn’t quite figure it out. um the RTAB map, which is a more fun thing to say. Um this one was a little uh more interesting um even though it was an earlier model. Um the point here was more along the lines of a much longer running type uh application uh where yeah I’ve got some working memory and I might know where I’m at kind of thing but I need to be able to actually uh take more steps um let’s say uh before I you know before I have to go check with check with memory to see you know am I am I where I thought I was um this one actually brings in a graph optimizer.
J. Langley: Um so you can imagine as you’re as you’re moving and you’re drifting or whatnot is you know because none of this stuff is exactly accurate. Um it’s got a way that after it if it does figure out okay now I know exactly where I’m at. It can actually go back and optimize some things u in the graph that it’s got uh underneath. Um, let me see if I can find I had the paper for this as well. This kind of got a little more um little more interesting uh because you got point clouds, an octa map, a map graph, some map data, you know, all thing all the things. Um, I’m not sure if you’ve got a switch between Oh, okay. So this is either or I guess between uh RGB plus depth images or just a stereo image. Uh then you got odometry. I don’t know what TF is. So anyway, uh you get all these inputs. Uh it synchronizes them for time. Um because some of these may uh a lot of your inertial units are running at like 200 hertz.
J. Langley: Um which I thought was pretty high, but that’s the thing. uh and you may have stereo images coming in at a at a lower rate. You know, your odometry might be coming in at a diff, you know, slightly different rate. So, it puts all that stuff together. Uh tries to do uh proximity detection, loop closure, uh to figure out, okay, where am I based on where I thought I was, things like that. And then it actually is is using its current map and then assembling what it’s calling a global map. I don’t know if Titer flow is even in this. Uh let me see. This was 2019 I think. Let me see if I can go up and look. Nope. Nope. Come on, you can do it. Oh, I missed it. Let me go back one. Transform library. Okay, that makes sense. Oh, this was a whole back in its time. Here were the number of models um or the models that it was pulling uh to try to figure out which ones worked with different cameras.
J. Langley: Uh you got LAR uh starting to come in odometry. Um and then you’ve got uh some kind of uh outputs that it’s trying to build. Um anyway, so this is a transform library that gets and I’m guessing I’m not sure where that comes from. Um I have seen that a lot of these uh if you get into some of the some of the papers on how to actually run them, uh which I’ve I’ve actually tried to build out another uh I gave up because it got it got way more than what we can cover in an hour or so. Um, I want to run some of the newer models we’ve come across uh with the actual uh same data set that uh you know we we looked at last week. Um so then I got into well that’s kind of interesting and then I got back to well what would it take for me to I mean these are all CPUbased models I mean I should be able to run them. So I started looking into well what does it take to actually rerun the same experiments u so I have the data they captured I should be able to run the same model uh download the weights and do the things uh not well download the vocabulary which is the interesting part um uh there is a lot of uh calibration that has to happen which could be where uh this transform piece comes in. Um, it’s got to know things like exactly how far off of your like so I’ve got an inertial
J. Langley: uh measurement. Um, and I’ve got a a a camera. Well, where are those located based on the actual center of mass of the robot, you know? Um, so there was a lot of things going in for that that were kind of interesting. Uh, but that was our tab map. Um, and again, same as the other two, it doesn’t actually learn anything. It’s got a bag of words dictionary. Um, hardware, um, with CPU, um, it was, and again, we’ve talked about disk space, which kind of gives you a clue how far back this was. Um okay. So they were they were more focused on um again longer running. So the bigger the map gets um you could imagine that uh you’re the time it takes to query the map and and do some some matching to figure out hey here’s what I see. Does it match anywhere that I’ve seen before uh in my map? The bigger it gets, you got some interesting trade-offs. Uh as in, well, how do I uh how do I structure that?
J. Langley: Do I structure it where it’s uh takes longer to do an insert, but maybe lookups a whole lot faster or or whatnot? Um but anyway um their comparison table uh pretty much shows you the you know one’s graph based one’s got threads things like that. Uh the loop closure uh two or six do off and then one is a four do off. Um this one uh went into some length to explain that you didn’t need the six. you could you could two of them were calculated anyway based on the third and it’s just like okay that was a lot of math just to say we’re doing four instead of six. Um, let’s see. This one it looks I’m still not quite sure on uh so there’s no training data, but I think this one can actually some somewhat update as it’s going. Um, which is kind of interesting. I don’t I’m not quite sure. I didn’t notice that the first time through. But again, the the weakness on nearly all of them or actually all of them um is loop closure is hard when all the things look like all the other things.
J. Langley: So there’s that. Um and then I had asked Claude, hey, I need to know the neural architecture of of these so I can actually compare them. And it said no. And I was like, what do you mean? Um and it was like, well, they don’t have them. I’m like, “No, no, go look again.” Um, no, really. Um, it doesn’t. Uh, so there’s this thing called orb features that I’m still not quite sure um what exactly that means. Um, so we did go through uh an it tried to explain it um because I asked it to go try to put it in uh plain English. Um, and it did what it the best it could. I think I’m going to bail out on this. Um, and actually have to go back and look and see what really are they. Um, anybody got thoughts on orbs?
Lorin Bales: Yeah. So, I’ve been trying I’ve actually been asking Jim and I about some of this because the moment you said that one, there’s no neural networks, one there’s no training data set and it’s bag of words, I was like,
J. Langley: Yeah, this makes no
Lorin Bales: what is going on? Right. So,
J. Langley: sense.
Lorin Bales: I was also digging as you were talking and and looking at yours and reading Gemini. And so, to me, it seems like I don’t know if you’ve ever played really old MMOs. Chris might be laughing at me at this point, but if you’ve ever played a really old MMO that doesn’t have a map
J. Langley: Uh would would Ultima online
Lorin Bales: and Yeah. Yeah. some something where like you’re exploring a world.
J. Langley: count?
Lorin Bales: Yeah. Or or like Minecraft or something, you know, something that doesn’t have a map. So, you’re wandering around and you’re getting all the images and you’re and you’re being like, “Oh, I’ve seen this before. I’ve been here before.” Right? So you’ve got that orientation, the key point location, right?
J. Langley: Mhm.
Lorin Bales: This kind of thing is going on, but you’re you’re the you know, you’re the bag of words and the map that you’re building inside your mind.
Lorin Bales: And that’s but my first concern my first concern with this entire approach is okay. So, if you’re building the map as you go, what happens when you have like object avoidance going on and there’s danger in your world and there’s areas you shouldn’t go into? Yeah.
J. Langley: Mhm.
Lorin Bales: Cuz because then how how how do you know that the first time the first time you explore an area?
J. Langley: Yeah. That’s why I think I think what this is is somebody put together a large set of things like here are the kinds of things that you may see in Minecraft. You know, this looks like a tree. Here’s what a tree looks like. You know, some things like that. Um, and it didn’t actually build it build you a map.
Lorin Bales: Right.
J. Langley: It just gave you the things that would be on a map. And now that you’re moving through,
Lorin Bales: Mhm.
J. Langley: you see a thing. You’re like, “Okay, here’s a tree. Cool. a tree is here.
J. Langley: Well, now I know I can’t walk through a tree. Um, so you could you could get some kind of a u an idea of and
Lorin Bales: Good.
J. Langley: again that’s that’s going to be I’m not sure how they did that at all. Um, where there there should be things that you can recognize that I have to avoid and there could be some things I need to recognize but I may not need to avoid
Lorin Bales: Mhm.
J. Langley: them. is in let’s let’s say I was building this for an automated lawn mower. You know, I I find tall grass. Guess what I should do? I should not go around it, you know. Um uh but I also need to know where I’ve been. Um, and that that’s that’s got to be another one because if I’ve been there and I’m and I’m actually and that’s another interesting part of this. Um, a lot of uh a lot of these things are meant to operate as if the map is the map. And and this is part of where the the other side that fell off or that that fell apart in the data set was going back through the same place in a different growth stage.
J. Langley: So uh that was where the environment was changing by itself as
Lorin Bales: Yeah.
J. Langley: you know between runs or you know the map is changing without any kind of interaction. Uh but you can imagine if the robot itself is changing the environment as it’s moving,
Lorin Bales: Yes.
J. Langley: you know, if I am if I’m making ruts behind me or if I am cutting grass or clipping things or whatever, I actually expect that to be different next time. So if I go back next time, the uh the problem they ran into uh the first part was hey loop closure everything looks the same. Uh the opposite of that is if you’re actually in the same place but it looks different because you changed it, you know, and you don’t recognize it because well last time it was tall grass. Well, yeah, you cut it,
Lorin Bales: Yeah.
J. Langley: you know,
Lorin Bales: The the warm thermometer changed the temperature of the water, right?
J. Langley: right? Yes.
Lorin Bales: Yeah.
J. Langley: So, it’s just it seems like a non-solvable problem based on how the how it’s being attempted to, you know, um it just it just feels like we’re going to build if I if we keep following this path, we’re going to keep having to put stack on top of stack on top of stack of complexity for keeping up with what I can go through, what I can’t, what I have to avoid versus what I don’t.
J. Langley: Um, if I do modify the environment, I have to either get another image of it or make some notes or keep track of that memory. Um, it’s a lot.
Josh Phillips: I mean,
J. Langley: Um,
Josh Phillips: isn’t this essentially why we stopped doing expert systems?
J. Langley: yes, it is.
Josh Phillips: They didn’t work essentially.
J. Langley: Uh, but, um, if if I needed to, let’s you can imagine how this would work out. We’ve got a model and it works great in this one factory. That’s cool. I need this in my factory. Okay, cool. Here’s the price for us to build the new model that works in your factory. You know, um I could see that. Um definitely.
Lorin Bales: the the other thing I went looking for in the papers is hosting the model because you’re talking about
J. Langley: Um
Lorin Bales: CPU and I’m thinking what is the B even with a CPU what is the battery pull on that right because when you go autonomous and you’re out in a field I sure hope that you can continuously run this this
J. Langley: Mhm. Well, that’s why Yeah. No, that’s why they were I mean, some of these were on like Ryzen CPUs.
Lorin Bales: architecture.
J. Langley: I mean, these weren’t like small Uh that’s why they they were they’ve been uh I mentioned somewhere it’s got I don’t want to scroll up too far. I’m going to lose my place. But uh one of the one of there have been some studies trying to run some of these models on a Raspberry Pi and some other you know either a Jetson or something you know something fairly small um just to just to get them where they’re even available on that kind of you know um because I’m not sure that might be another they didn’t actually answer this one either. Um the the robot they used from Terraentia, it had a Jetson Orin um you know GPU available. They were just using the GPU to do graphic stuff. I mean that’s what they used for the video encoding. That’s all they used it for. Um I don’t even know that uh they didn’t try to run any of these models like online or live you know they just captured the data and then they ran the model offline against the data to see what the error would have been had they run the model you
Lorin Bales: Oh,
J. Langley: know yeah um which good news is that means we can also try
Lorin Bales: wow.
J. Langley: a bunch of different models using that data um you know what I mean it’s it’s kind of I I do love that they captured that much data and shoved it out there for us to use. Um, but yeah, I mean that was that you got you got the uh first off, does a model even work? Second, can I run it on something that I can put on a moving platform um and not run out of power every, you know, 100 meters? Um, but yeah, this one, uh, a lot of this goes back to bags of binary words for fast place recognition and image sequences. This is one of their, uh, I I’ve seen a lot of references back to the initial um, bag of words paper. U, but that was kind of interesting. uh think of it as a uh a way to take an image uh pick certain parts out of the image to kind of create some kind of an embedding for it and then creating some kind of a hierarchical structure.
J. Langley: It’s not necessarily just a 2D kind of a search. It’s actually more of a more of a hierarchy. Um and a lot of times they’re Yeah. One I’ve seen is uh splitting putting them into groups. Uh they like the factor of 10 a lot from what I’ve seen. Um so yeah, interesting stuff. Um this kind of felt fun. It’s like we took vocabulary stuff we used to do for NLP and applied it to images.
Josh Phillips: My favorite uh uh retrieval method that’s not uh semantic search is actually BM25
J. Langley: Uh
Josh Phillips: and that’s back words.
J. Langley: yeah.
Josh Phillips: Really good. It works if you’re doing keyword search.
J. Langley: Yeah.
Josh Phillips: Yeah. Hey man,
J. Langley: It just happened.
Josh Phillips: fat.
J. Langley: Uh they got that one right. Um that’s just kind of fun. Uh skipping on down uh back to the down to more of the newer methods. Um and this was basically using uh Claude as a uh I didn’t do a uh deep research or anything.
J. Langley: I’ve basically said, “Hey, if what are the things that have come out in the last year or two that would be used today for this kind of thing?” Um, and master was a was one that popped pretty hard. Um, I love that first off somebody built a duster model and then somebody else came and gave them one up and built a master um with the funky spelling. Um, it’s still trying to do the same kind of loop closure. So, I’m not sure that I’m not sure anything that’s trying to do a loop closure is going to necessarily work well for what I’m trying to do. Um because it’s just a fundamental thing that when you’re trying to uh na so part of it is just you just need to navigate between where the stalks are. I don’t need to know that I’m in the exact same place I was before. Um or maybe I do. Maybe that’s a whole another problem. But so maybe separating the mapping from the navigation could be something to look at.
J. Langley: Um, so this one actually has uh a bunch of different data sets that it was trained on. Um, I haven’t taken a whole lot of uh of time to go uh to go look through these. Um, apparently none of it is agriculture. Um, it’s, you know, multi- view geometry data, blah blah blah. Full stop. That’s how you know Claude wrote this part. Um, gotta have a GPU. Um, yeah, they ran it with, you know, an Intel with a a GeForce a 4090. Um, but basically if you can hit a 15, you know, frames a second as far as that that you could be autonomous at that point. So that’s good. Uh, and then it looks like somebody took the Master Slam and then I don’t know if this is because they came out around the same time. Um, you have this Master Fusion. Um, hold on one second. Great group. Uh, Wuhan. Okay, this one was don’t know but anyway. Okay, so there’s that.
J. Langley: That’s interesting. Okay, so this one, the the GNSS thing though, that’s the I’m not sure if that would actually work because that kind of depends on knowing your position, which we don’t always. Uh this one uh apparently was the the one to go with uh until the master part came out. Um uh but DPVO uh or DPV slam uh is another kind of a I mean this one was seems to be a little bit different architecture. Um, and of course you can tell Claude wrote part of this. Then you got the VGT anyway. So that basically Yeah.
Lorin Bales: Hey Jay,
J. Langley: Go
Lorin Bales: some of these videos are awesome.
J. Langley: ahead.
Lorin Bales: This link for that master
J. Langley: Yeah. Hold on. I’m trying to see if it’ll let me click it.
Lorin Bales: slim.
J. Langley: Come on. There we go.
Lorin Bales: That’s incredible.
Josh Phillips: Yeah, I know you mentioned that slam’s not super useful for like some of the things that you’re doing. I think anything that I’d be interested in looking at robots be like I want it to do my garden which is you know it can be somewhat known and it’s small you know or you know do my house or something like that.
Josh Phillips: So you
J. Langley: Right.
Josh Phillips: know
J. Langley: That’s why I was looking at some of the other data sets to get out of the agriculture part as well because I don’t want to be, you know, there are there are some things that if you could figure out how to do autonomously um with anything ground um would be pretty interesting. Um and there there’s also the kind of the mindset sometimes of oh well that’s already been done. It’s like well yeah maybe but there there’s actually a need for um you know more than just one operation uh that can do a thing. You know um pick up your yellow pages and see how many plumbers are listed. You know if there was only one well we’d all be waiting. Um,
Josh Phillips: But then you can’t claim that you’re state of the art, Jay.
J. Langley: I don’t
Josh Phillips: And what’s the point of if I can’t brag that I’m state of the art?
J. Langley: care.
Josh Phillips: I should just go, you know, sleep.
J. Langley: Oh man.
J. Langley: Well, you know my rule. One fancy thing. Everything else needs to be boring. Uh, yeah. That’s wild. I say Yeah. Oh wow. Yeah, we may have to play with this some. Um,
Josh Phillips: Is this like using like like the Gaussian splat sort of stuff? That’s almost what it kind of looks like.
J. Langley: yeah, I’ve seen some of these actually did use the word splat.
Josh Phillips: Okay, that makes sense.
J. Langley: Um, oh, and patches or deep patches or I’m trying to think of where that came in. Um, that was it wouldn’t be I think it was Oh, this was the DVPO. Okay. Um, so I don’t know if this one was was uh splats or what. Uh oh, this one, of course, you have to have one that’s got a transformer. Um, why not? encoder decoder uh for custom CNN. Okay. What’s learned versus classical? Okay. Okay. So, it’s got a some kind of a factor graph that it’s already learned or that it, you know, doesn’t get updated that it’s using.
J. Langley: Uh patch tracking correlation is learned. Uh some of these papers are good just to go back and find hey, can I get a copy of that data set to play with? All right. Oh, the slam former. Oh, this one this one was interesting. Uh so looking at, you know, just the uh just the difference between like an Orb Slam 3 and a Master Slam uh when that I found a couple of places uh that actually had some comparison of them. Sometimes it’s apples to apples, sometimes it’s not exactly. So it’s take it with a grain of salt. Um but some of the some of the systems and this not an agriculture one because Orbl Slam was definitely not at a you know.135 meter accuracy on that one but uh taking it from a.13 down to a 086 or a 135 down to a 069 you know that’s that’s some pretty good uh reduction there. Um I don’t know if that’s a I maybe resolution is better better way to frame that one.
J. Langley: Um some of So one side on one side of it you had the accuracy uh change uh but then again here is uh Claude being very honest. Um some of the other stuff uh that came out with this was just how much faster you can run some of these uh you know so what kind of uh you know can I actually get this on a machine that I can put on a let’s say an air platform or a small drone something like that. Uh, let’s go back. Yeah, let’s take a look at the the slam former Yeah, it’s pretty cool. So, some of this I’m just barely getting my feet wet. Some gaps. Let’s see. Inclusion blah blah blah. And again, all the references. Yay. Um, and that’s currently out there in the presentations uh repo. So, where do you think so? thoughts. Um, one of them was uh was Josh’s comment about taking, you know, some of our uh more current models and then seeing if we could use those to go backward um and possibly uh either re recreate or test on some of the features um that had been created.
J. Langley: Um where else do you think we ought to poke at next? you know, if we I’m probably gonna keep keep rolling down some of this um kind of train for a while.
Lorin Bales: Well, I could totally see me spinning off with some of these data sets and uh running the uh Isacson on uh my DGX.
J. Langley: Mhm.
Lorin Bales: Maybe go that way with it,
J. Langley: That’d be cool.
Lorin Bales: but more of that industrial rover perspective I was trying before.
J. Langley: Yeah. Yeah. One of the other things that would be similar enough uh to some of the other thing to one of the other pieces I was looking at for the agriculture part. uh the same mechanism I was looking at to evaluate uh soil nutrients um is also used uh it’s currently used on site surveys uh for certain industrial locations things like that and right now that’s a manual operation of having to actually uh think it’s nearly like a ground penetrating radar type approach where you have to drag this thing over every you know it’s basically making an underground map Um, so just something like that where I where you do actually have good GPS, you know, I mean it’s it takes away one of the major uh constraints from the agriculture side, but it still involves some of the same, you know, underlying mechanisms.
J. Langley: That might be another another thought or another stepping stone.
Josh Phillips: Yeah,
Lorin Bales: And then All
Josh Phillips: all of my problems are spatially boring. So, all the things they’re talking about are super useful if you have to go trekking.
Lorin Bales: right.
Josh Phillips: I’m just like sitting in my cave of of knowledge and and and stuff. It’s like uh like a lot of the things like we we were talking about having something that like can sort cards and stuff like that, but that’s robotics with like you can stick it in a spot and you know, you don’t worry about uh power being so much of an issue.
J. Langley: right? Oh, I’ve got Okay, this might be interesting. We may have to go somewhere else to talk about some of it,
Josh Phillips: Um
J. Langley: but let’s say if you were a drone uh and you were operating with an operator um and for some reason you got into a place where you were no longer connected to your operator right now,
Josh Phillips: Mhm.
J. Langley: I believe they just land, you know.
J. Langley: Um, what if what if you had some record of uh the visual part of how you got
Lorin Bales: Correct.
J. Langley: there and could actually retrace your steps to go back where you came from. Um, especially in GPS and other comms uh
Lorin Bales: Mhm.
J. Langley: challenged places. Um, that’d be kind of interesting.
Josh Phillips: Mhm.
J. Langley: Or what if I could fly a route once um and have some some kind of visual record of that but then offload that and have some other uh you know drone fly the same route. That’d be kind of interesting. But
Lorin Bales: Well, and if you can detect the bubble,
J. Langley: anyway,
Lorin Bales: if you will, you can also kind of know where not to go, right? It’s that still that object avoidance scenario. You can kind of like a Roomba kind of fill out where the couch is,
J. Langley: right.
Lorin Bales: if you will. But Josh, I was curious what you were going to do in the garden. What was your idea there with a
Josh Phillips: Oh, I mean, we just have a tiny little garden.
Lorin Bales: robot?
Josh Phillips: You know, obviously it’d be nice if, you know, go and zap, you know, little things, you know, look at weeds. Who knows? You know, obviously if it’s, you know, uh something that we have, we’d make it for whatever we need to do. You know, they’re little little annoying things that pop up that would be nice to be able to automate that.
J. Langley: Yeah, I’ve come across uh I’ve I’ve been snagging training data sets left and right as I go through things. So, um I’ve currently have a good training set for uh weeds. Um, so visual identification, classification, that kind of thing. Um, and I’ve also got another one for your uh general pests that you’re going to find. Uh, so if you got bugs, if you got different kinds of uh, hey, let’s go find some aphids. uh you know it’s it can do that um uh that would be
Josh Phillips: Is that for picking green beans?
J. Langley: interesting uh or okra um at the right stage.
Josh Phillips: Yeah.
J. Langley: Um that would I I think if you that might be the another really interesting actually that’s it’s probably a lot more people in that market than I would than I would think right now. uh but trying to figure out how to pick fruit or think of where you have a lack of manual labor uh due to certain uh u issues we have at the moment. Um well, how is that going to happen? You know, uh if you could figure out a way to do that robotically and it’s not a there’s a reason it’s a manual task still, you know, I can pick corn with a, you know, a picker. I can pick cotton with a picker. I can pick soybeans, you know, with the harvest, you know, all that kind of stuff. But, um, I can’t do that with apples, Farmbbot. Yeah, I’ve seen that. That’s Oh, yeah. This one was open. Oh, this might be better for
Lorin Bales: Mhm.
J. Langley: Josh.
Josh Phillips: Oh yes, I like the entries.
Lorin Bales: Yeah, it’s on a rail.
Josh Phillips: I have a whole bunch of stuff. Yep. Very
J. Langley: Oh,
Josh Phillips: good.
J. Langley: yeah. Whoops. Watch the video.
Josh Phillips: That is uh Hey Charlie, are you paying attention?
Lorin Bales: So, I think the only I’ve seen this I’ I’ve done a whole deep dive on this, Josh. The only thing that’s the only thing that limits this is the height because it’s kind of it’s on rails and it it’s static, right? So, like if you got a really big squash bush or your green beans grow out of control,
Josh Phillips: Gotcha.
Lorin Bales: that’s it, though.
Josh Phillips: Gotcha.
Lorin Bales: That’s it.
J. Langley: Oh, but if you built, we’re going way off topic, but hey, this is fun.
Josh Phillips: Build your own pantry.
J. Langley: Um, yeah, build your own I mean, manufacture the gantry. Uh, is that the right way to say that? Gantry uh rail with crane thing. Um, yeah, you could uh what if you manufactured green houses with pre-built gantries in them or something and you know that that would be kind of interesting is a I don’t know how much it the cost would be, but uh you know how many people there are buying little plots of land and buying chickens um and we’re going to raise our own food and all that kind of stuff.
J. Langley: Um and then uh uh myself uh by the time I got out of high school, I didn’t want to pick another thing ever, you know. Um purple whole peas, anybody? Uh so I Yes.
Todd Page: Oh
J. Langley: U no let’s let’s not do that,
Todd Page: god.
J. Langley: you know. But um if you had a prepackaged greenhouse that I could sit behind my house that took you know 12 by I’m not sure how you you’d it had to fit on a truck so you’d have to be able to deliver it. So uh maybe 8 by 12 might be the biggest whatever but with a pre-built thing that could water and weed and pick my stuff for me and then go drop it off or package it. That that might be a that’d be interesting. Especially you get into urban gardens and things like that where you’ve got uh you may have a location that you can put this but I can’t go there but like once a week, you know, because I’m working. Um but yeah, if I had something like that that could just do it for me.
J. Langley: Um, and then I show up on the week, you know, on a on a Saturday and all of a sudden here’s Well, the way it usually goes is, uh, gosh, what is it that people always grow more of than they think they need? It’s usually either tomatoes or squash or zucchini.
Josh Phillips: Yes.
J. Langley: I can’t remember which one of those, but you plant a little bit of it and all of a sudden it all comes in at one time. Um, and you’re trying to figure, oh crap, what do I do? Um.
Lorin Bales: We uh we accidentally planted pumpkins. So we we gave pumpkins to our chickens and in the
J. Langley: Oh man, that I bet that
Lorin Bales: fall and then that planted into our garden and then we had unbelievable
J. Langley: worked.
Lorin Bales: amount of pumpkins like 30 40 pumpkins.
J. Langley: Uh, have you hit the egg crisis yet where you’ve got more eggs than you know what to do with?
Lorin Bales: Not yet. We have a new round of a flock.
J. Langley: Okay.
Lorin Bales: So,
J. Langley: Okay. We’ve I haven’t done that, but I’ve got friends that all of a sudden they’ve got buoodles of eggs.
Lorin Bales: yeah.
J. Langley: And it’s like, what are you doing? Well, you and then if you want to sell them, well, now you’re selling food, which brings a lot of other,
Lorin Bales: Yep.
J. Langley: you know, but you can give them away, but then it it’s just really interesting. Um uh I mean, I’m here for it. So, if you ever have too many eggs, let me know. Um I think Ethan came in with peaches last year. um with I think the the orchard he works uh with I think his family has an orchard. Um had too many or something like that. So it was it was interesting.
Lorin Bales: Okay. Well, I will definitely bring some eggs when when that happens.
J. Langley: Oh yeah, Todd was this other link.
Christopher Bales: We
Todd Page: It’s a c able
Christopher Bales: should
J. Langley: Okay.
Todd Page: gantry.
Christopher Bales: we should get about 8 to 10 a day coming up here in about a month.
Christopher Bales: So, no problem with the eggs.
J. Langley: All right. Oh, I’ve got I bet there’s a skip thing. Okay, there it is behind my Yeah, I got it muted. Uh, fishing line Raspberry Pies. Interesting. Okay. Oh, man. I know some parents that would love this. Oh my gosh, this is so fun.
Todd Page: I might not speak much, but
Josh Phillips: That’s amazing.
J. Langley: Oh,
Christopher Bales: Isn’t the industry already using AI to like pick
J. Langley: wow.
Christopher Bales: um certain vegetables using an arm?
J. Langley: I think it is. So, and
Todd Page: Uh, strawberries and grapes.
J. Langley: if
Todd Page: Not grapes. Some other ones. I’ve seen actually where they actually have drones.
J. Langley: yeah, I’ve seen some some things trying to work like fruit trees, you know, things like that. Um, but yes, that that is a thing. Um, I don’t know. Uh, it might be something interesting to look at just to okay, this thing exists. What would it take to make it better or more efficient or more affordable or you know I mean you could you could look at it a couple of different ways.
J. Langley: I’m actually going to finish stop sharing my screen somewhere if I can figure out how I don’t want to present something else. There we go. And then back over here. So, any other any other thoughts on that fun stuff?
Lorin Bales: I have a sidebar
J. Langley: All right,
Lorin Bales: question.
J. Langley: we can close this out and go to sidebar questions and other fun stuff.
Lorin Bales: Okay, so this is my uh chip that one of the chips I use for my heart rate monitor. And so I’ve got I’ve got this idea, but I don’t know how to implement it, which is if I have the STL files, how do I get AI to build the cases around other objects? Have you guys done that? Have you vibe coded a 3D printed object?
J. Langley: Oh, I have not. I believe it has been
Todd Page: I have but not that old.
J. Langley: done.
Todd Page: I’ve actually did a 2D two or three modified.
Lorin Bales: Okay. Okay.
Todd Page: So what are you trying to do?
Lorin Bales: I’m trying to I’ve got I’ve got three STL files and I want to build a case for them.
Todd Page: Okay.
Lorin Bales: I’ I’ve already done it, but it does take me quite a bit of time, Todd. So is is there like a a setup for this? So is there like an IDE like I don’t know I’ve been using anti-gravity and then I tie that with like blender or an on shape API or like what is what is the vibe coding framework but for building CAD models or STL files and whatnot.
J. Langley: I’m gonna type that directly into Google and see what happens.
Lorin Bales: I’ve asked Gemini,
J. Langley: No,
Lorin Bales: but I have no idea.
J. Langley: I did I did I did something like that the other day and it’s totally surprised me what I
Lorin Bales: Yeah.
J. Langley: actually got by
Lorin Bales: Okay.
Christopher Bales: I know this exists.
Lorin Bales: Yes. It’s got to exist. But has anybody done
Christopher Bales: I don’t know how expensive it is
J. Langley: also
Lorin Bales: it?
Todd Page: What about Google banana?
J. Langley: known as vibe modeling. Uh, okay.
Lorin Bales: Yep.
J. Langley: The AI writes parametric code. Uh
Josh Phillips: All I’ve done is is diffusion to mesh which is not a SDL level file.
Lorin Bales: Okay. Okay.
J. Langley: uh apparently it exports the it provides the thing that then you have to go run a secondary tool on to I don’t think it gives you like the STL. I think it gives you the thing that you you go to uh either OpenCAD or something
Lorin Bales: Okay.
J. Langley: and compile into an STL. Not sure what that means. Give me an
Lorin Bales: Yeah, Charlie, there’s there’s some sort of API layer to on shape and that’s what Jim and
J. Langley: example.
Lorin Bales: I was recommending. Agreed.
J. Langley: Oh, that’s interesting. I didn’t know it was a thing. Oh, this almost like screams for needing a lovable or something like that where instead of having you, you know, it does a piece of it, then you have to pick up that piece and then go compile it and then take that piece that was compiled to an STL, load it into a viewer.
Lorin Bales: Yes.
J. Langley: You know what I mean? that it seems like there’s still a few
Lorin Bales: Yes.
Todd Page: See
J. Langley: steps. Um
Christopher Bales: So Jay,
Todd Page: how
Christopher Bales: while we’re on sidebars, were have you uh continued to work on that app that you were talking about
Todd Page: this
Christopher Bales: the other day, the um where you were using Lambda to spin up a bunch of the processes?
J. Langley: Oh, yeah. Hold on. Uh,
Christopher Bales: is
J. Langley: one second. Let me figure out how to stop recording. I’m sure. Stop.