Meeting notes provided by Gemini
J. Langley: So, we’ve talked uh a bit uh about Let me hide this thing. There we go. Uh we’ve talked a bit over time about uh some thoughts I’d had on trying to figure out how to make a little robot go up and down corn rows and other kinds of crops, you know, things like that. And I’ve been looking for a long time actually, uh for some data set that I could actually use to see if I could thinking about a vision model or a world map, something like that. Um, and I came up empty over and over and over again. And I’d pointed Jim and I at it trying to I’ done a deep research project trying to see what is out there. Uh, almost everything I found was some kind of a, you know, a warehouse kind of a thing. And it’s like, that’s great. Um, yeah, this is not a flack. This is not, you know, this is very different kind of a thing. Um, and then I think it was last, no, I know what it was.
J. Langley: Right before I left on a trip uh to go on vacation, I was asking a question to Claude and it said, “Oh, by the way, here’s a data set for this.” Um, now I was actually I don’t know what it was. I was working with another friend that runs a production company and happens to have uh cameras and knowledge on how to do things. I’m like, “Hey, if I can find a corn field in the farm with a little lettuce, would you mind if we take a gimbal or something and run it through?” I’m just looking for some kind of a video thing because I couldn’t find it online. And then I had dropped that into cloud saying, “Hey, I’m trying to find the best approach to make a video going and it’s like, well, yeah, you can do that or here’s a whole data set.” I’m like, “Well, crap, where have you been for the last, you know, because this data set, I believe, is a 2024, I think, which we’ll get into a little bit.” And so, I was able to nyx the whole, “Hey, let me go get dirty in a cornfield.” Um,
J. Langley: because this actually has, I think, over nine hours worth of video. Um, so anyway, there’s that. Um, so we may, if we’ve got time, we may touch on the paper these folks actually wrote as well. Um, but here is under the canopy, the Terraentia data set for agricultural slam, which yes, fun stuff. Uh, let’s see. Can I go next? Next. Yes, please. Okay. So, SLAM, which some of you already know this, uh, some of this is new to some, uh, simultaneous localization and mapping. Um, think of a Roomba maybe, uh, but more advanced. Um, so it’s mature for indoor and urban stuff. Uh, a agricultural breaks most of those kind of assumptions. Uh, so you don’t have a GPS to lean on. uh even your uh very very spec uh what’s the right word? Precise uh GPS. Um RTK I think is the name of it. Um the kind of GPS that typically you can get within centimeters of where you’re trying to get to.
J. Langley: Uh does not work when you start having uh a bunch of leaves and a bunch of other organic material that is also full of water. Um so you got a lot of different bouncing signals from an RF perspective. attenuation and then all of a sudden it works great because you hit a gap and uh we’ll look at some of that. Um so uh you have to you have you have these uh internal uh what is this imus? Um shoot not momentum this inertia momentum may be the word and then unit maybe I’m not quite sure. Um and then of course you know how fast your wheels are spinning hopefully. Um, so the other thing you run into, um, if you’re trying to use something to to look between two different frames and decide whether I saw a cornstck in the last frame at this position, in the current frame it’s now at a new position, so I’ve moved closer, but when you get into it, um, a cornstck looks a whole lot like a cornstck. And it look like it kind of like back to the firefighter thing.
J. Langley: When we all suited up, we all look just like everybody else. I can’t even in my own department it’s hard to tell. Um so uh obstacles um and not things you would and we’ll we we’ll actually show some of this uh after a bit. Um the wind blows that’s kind of interesting. Uh you’ve got sun and other kinds of things that have cameras lit and you know things like that that are hard and then so other parts of that. Um, and this one threw me initially. I I thought I I thought I had converted a video at the wrong frame rate because it was it’s ugly. Um, but yeah, you put a you mount a camera directly on the little robot, have it go screaming up a row. I mean, the thing is just, you know, I don’t know if you’ve ever ridden a four-wheeler over a over a a field before. Um, it gets pretty interesting. Um the other issue is uh if it’s a field or an agricultural field where things are growing, every day is a new day.
J. Langley: Um every week is a different aspect of this thing. You know, you got different stages of growth. Um so yeah, if you if you build a model to work in June, it’s probably going to suck pretty bad in August. Uh so there’s that. Uh so they built this data set. They I believe is the University of Illinois um which we’ll hop into that again in a minute. Um so they’ve got some uh data sets that are already out there. Um of course tractor mowing, you know, things where GPS is pretty available. Um you know, uh over canopy only, low frame rate, you things like that. uh Terra Terraentia uh they built I mean it’s got high frame rates a bunch of different crops uh over several months and then they also give you some different types of of uh data out of this. Uh that’s actually a picture of the little robot that they used. Um the other interesting side of this that I won’t pull too hard on is John Deere then acquired that particular robot.
J. Langley: So, if you want the robot, guess where you have to go. Um, so there’s that. Um, I haven’t actually seen that productized or anything. I don’t know if they bought it to use it or bought it to keep other people from doing things with it. They probably lock you out of fixing that one, too. Yeah, they will. Um, so there’s that. Good news is there’s a boatload of robot stuff out there. Plus, you can make your own. It’s small. Um, so an overview of the data sets. Uh well, a terentia again uh four-wheel skid steering ground robot. Uh they were using the jets and AGX orin um and storage. U it’s at the Illinois Autonomous Farm. Uh oh, University of Illinois Urbana Champagne. Um and again, well, 2022, so this is older than what I thought it was. Uh and then again 135 different sequences of running this robot. So basically what they were doing was taking the robot driving it manually.
J. Langley: Uh some of the videos I’ve got you could actually see the p the shadow of the person behind you know uh but they were capturing the GPS feed the uh all of the stuff and again this is using a robotic operating system or Ross. So, they got all of the topics from the robot itself captured uh with uh well, I think they’re about to hop into another I think I’ve got another image coming up. Uh but just over a terabyte um which caused me some initial problems. Uh I locked up my laptop real quick. Uh and then had to figure out a way to back that off a half a turn. Uh 9.7 hours of video um total. So, it’s not a bad data set. Um, so the they got two different camera feeds that they were using here. Um, and I also learned what zed was. Um, I’m still trying to figure some of that out because there’s a mix in this data set of different things. Uh, so they’re they have a stereo inertial camera with a 120 degree field of view, which is probably wider than what I’m really looking for, but there’s that.
J. Langley: Uh and then the other image set has a smaller uh so the Rossback images which we’ll we’ll show some of those uh are smaller images uh but they are at uh apparently ready for some other data set that they not data set but model that they were training. Uh the SVO images uh are higher quality images. Uh I’m still not quite sure what frame rate they have on those. Uh but the compression was insane. Um on that um the robot and nursery unit uh is 86 hertz and six degree of freedom. Uh they were using a standard plus RTK. Uh their normal GPS under normal circumstances without being in a crop canopy has less than 2% centimeter accuracy, you know. you can get pretty well where you want to go. Uh, and the wheel radius, I can’t do 26 meters to inches in my head, but that’s basically the width by the length. Uh, small enough to fit within a cornrow. So, they’ve got uh four different corn fields they ran um over different lengths, whether they have weeds, whether they have uh weather variability in there.
J. Langley: So things look and that this is something that uh is kind of interesting when you get in the middle of this. Uh it almost reminds me a little bit about being underwater. When you go deep it gets dark, you know, things like that. The higher it gets, the darker it is. Um they didn’t mention in here, but we’ll see it in the video. Um, if you’ve ever been on a on a farm that actually has uh, you know, different kind of uh, uh, center center point fre center point uh, irrigation rigs and things like that. Switchbacks, huh? Switchbacks. Uh, switchbacks and places where you know the roads are going this way. All of a sudden, you have another kind of path coming across because the wheel, you know, and it’s not a lot, but it’s enough to where you could you could see where this thing might get lost occasionally. Um, or hey, this looks a whole lot like a row, you know, but it’s just kind of curvy. Um, things like that.
J. Langley: Uh, they did a couple of other kinds of uh kind of crops, which I’m don’t I’ll probably just play with corn thing to start with. um for that. Uh they had two different formats they captured out of this uh the Ross bag which I learned what that was uh like two days ago because the worst part of this whole thing was I came across this data set right before I had to leave on a trip and I had to put it out of my head because the last thing I want to do is be sitting on a beach thinking about Ross bag images which luckily I’ve gotten much better at that than I used to be. Uh so and this was a pretty good interest a pretty interesting thing. They got all the sensors all of the different kinds of topics that go across the system. Um, so and it and it’s it’s it’s it reminds me of something we did uh years and years ago with a little robot uh for the Deep Racer Challenge where we were trying to build a robot that only ran off a stereo camera and we built we all we had was a simulated environment with this and what they were doing was picking your model up, putting it in a real vehicle around a real track to see how the virtual to live transition affected things.
J. Langley: Um, so in this case, not only do you have the data set coming in an actual video feed if you wanted to do that, but if you wanted the exact same topics that would come across a robotic operating system to your GPU, that is there as well. Um, so you could train on the on the visual stuff or the video, but then if you wanted to see uh because you you got a lot of interesting things on trans, you know, getting things from a video feed into a topic and then stitching that back together and feeding it to a, you know, a GPU, you could introduce some interesting uh things there. So, um, I learned that ROSS is a is a pretty big thing and if you want to install it, that’s going to be a a bit of time. Uh luckily uh there’s this package called Ross bags that gave me everything I needed. Um that was just hey pip install Rossbacks and I’m up and running. Um SVO was a whole another thing. Um the Z SDK is a propri this is one of the issues I may have with it.
J. Langley: Um it’s a proprietary SDK proprietary data formats. Um it has some models built into it for depth peression perception out of video things like that. that it’ll build a depth map out of your image, things like that. But it’s all you are tied directly to using their SDK to do all of it. Um, and I’m not quite sure how I’m sure people use it a lot, but it’s it’s not quite like OpenCV where there’s an open part of it. Um, but it does a lot out of the box. Um, so I had to go figure out how to install it. Um, and do some things with that. Uh but yeah, that it is nuts at how uh compressed this stuff is. So for for instance uh one of the Ross bag pieces I pulled for one round trip down two rows uh was around five gig worth of data. The same stereo thing from SVO uh was around what two or 300 megabytes something. I mean it was So when I initially started this I was like well yeah let’s just do SVO.
J. Langley: That’s really I just wanted the the video and then after I had to hop through eight hoops to get it um I’m now kind of headed back towards Rossback approach but where my thing go there maybe there we go. Uh so it they actually did some work with some calibration tool. I didn’t look too much into that yet. Uh mostly I was trying to figure out how do I get this data out of this data set that I want to use. Uh, but this shows they’re they’ve got two different inertial units and then this camera that apparently works right with Zed. Um, they’ve actually got two different cameras. It it appears I’m not sure what the different pieces are around this the edges of the robot. Uh most that you have like this also have a light a light LAR uh piece available as well with a puck or something that you can use for like uh safety constraints or obstacle avoidance or you know don’t run into the wall, that kind of thing. But this is uh of course I had Claude build my slide for me.
J. Langley: Um this was after having Claude figure out how do I get this thing to work? Uh because initially when I pulled it I I couldn’t get much anything going. Um, so I actually had it go do a bunch for me. Um, so there’s two different again the two different paths. Uh, we did the, uh, the Rossback library um, instead of a whatever that workspace is. Uh, these are gotchas that it recorded that it hit along the way, which was interesting. Um, apparently, uh, whatever version of Ross bags they were using, of course, was 2022. Um, and things have advanced a little, so some of the message structures were different from what it was expecting. Uh, so it had to go uh change a couple of things in the code to read these messages out. Um, it found that the depth encoding wasn’t consistent across each run. I don’t know why they would have done that, but they are different. Um, again, units of measure are different. Um so they found a way to branch based on some other parameter and you know added that to the code.
J. Langley: Um not everyone and I guess these are called bags. Um the file extension is bag. So that’s fun. Um apparently not all of them have GPS in them. So again it’s it kind of reminds me of how uh what happens when I try to capture a data set. I get going um and then I realize, oh, I should capture this as well. But do you throw away the previous data or do you just kind of, you know, you never throw anything away. Um so there’s probably some of that in here. The SVO setup um sucked pretty bad. Um I had to go figure out how to download the Z SDK um and then figure out how to grab the version they used. Um, and then I figured out I couldn’t grab the version they used because the one in 2022 isn’t supported quite as well as as the one you can get now. Um, plus it used CUDA 10 something and I’m on a CUDA I don’t I don’t remember which one I’ve got right now, but it was not that one.
J. Langley: So, I found the version of Zed that matches the version of CUDA I had um and figured out how to get that pulled in. Um, and then figured out that what might be next. Yep. the version of Zed they had only used Python 310. So I had to back up to that. Um that wasn’t fun. Plus they’re not on how do you pronounce that? Is it pi or pipel? I’ve always heard it as pi. Pi. I know it’s not actually that though. That’s the I’ve only ever heard pi. I I heard some pedantic people on a dev conference. They like no it’s I don’t say py. I was like, “What do you say?” That’s great. Okay. So, I know how to make them upset. That is a good word. Oh, it’s pi. That is the official way of saying right. So, pie I’m never saying pi is a separate project. Yeah. Yeah. Um there’s a Yeah.
J. Langley: So, anyway, their installer isn’t even on pi. Um so, their wheel is not. So, I had to install I had to install their stuff in the system level or whatever and then Claude was nice enough to go rip through all the stuff they installed and find a wheel that it could then install in my environment. So, nice. I appreciate that. Um, it’d be better if they actually did that out of the box. At that point, I was actually able to to figure out how to get the script to run. Uh so yes, Ross bags is a much easier way to go with that. Uh also with the SV, oh this was a fun um it try their their SDK when you initialize it um has a some kind of a calibration thing that it wants to do using your GPU. And my GPU on this laptop is not great. Um, luckily it’s greater for 2022, but um, this thing would calibrate and spend like 10 minutes to get to 90.1% complete and then hey, and so some Google searches would would initially tell me, oh, you need to upgrade CUDA.
J. Langley: And I’m like, I am not about to upgrade CUDA when I’m trying to give a presentation in two days. you know, um, every time I wind up trying to do that on WSL2 on Windows, it’s just winds up being more than what I expected it to be. Um, so next up, I probably will, but not today. Um, there’s that. Um, output, you get per frame, PGs, um, and I put them all into a video with FFmpeg. Um, like you do, uh, demo time. So, let’s get out of that. We’ll actually look at some of this code and whatnotss. Uh, and there’s my slide that it made me. There’s my markdown file. Um, this is what you were using when you let Quad fix and things. Yes. Yeah. You weren’t using desktop control. No. Um, so, uh, the Terra sent you a data set I pulled directly off of, uh, GitHub that they’ provided. Um, and they’ve got some download scripts. um for if you here’s a link with all of the SBO files that you can pull.
J. Langley: Um they’ve got them all hosted on Box. So, initially my thought is I may need to rip them out of box and put them into S3 or somewhere where at least I know I have them because um if it’s not a super popular data set, but I’m guessing Box isn’t a free thing that you can just put up there and just, you know, if the world all of a sudden started pulling these, I’m pretty sure um somebody would get curious about why they’re u why their charges went up. Anyway, um you can see it’s got a uh the SVO files. You’ve got like a UTC time stamp type thing. Um and then two rows random. Uh from what I could tell there, that data file was just they picked two rows and they ran up one and back the other. It doesn’t seem to always be the same starting point, which is kind of I guess good and bad. But, uh, three rows, they didn’t do three rows on every date or every time they went out.
J. Langley: So, this is kind of hit or miss. One row seems to be very one row and four rows seem to be the two that they did every time they made a trip out to the field to to capture data. Um, so I would do this and then grab some SVO files. Uh, the other thing you can do is in this cornfield one links and again this when you download it, none of this is commented out, but of course I don’t have space for all this stuff. So um, remember which one I somewhere down in here. I tried to grab August, I think. Yeah, the Ross bags is actually uh captured by you can actually see what date they grabbed. and then they got hours, minutes, and seconds. So, those of you that can’t do UTC to hours, minutes, seconds, and dates in your head, here’s a whole lot easier way to do it. Um, so in this case, I grabbed four rows and then I messed up because I didn’t realize they have uh this links file grabs bags and SVO.
J. Langley: Um, so if you did want to see the exact same thing, but between um maybe this is one row, where’s the other one row? actually one row. Okay. Yeah. So these two um you got one in a back format, the other in the SVO format. So if you wanted to compare um you could do that. And then uh they also provided these two different uh files uh extract data from Rosbag and extract data from uh from SVO. Um, these are the ones where initially I downloaded this. Um, I grabbed a actually, let me see if I still got this. Come on. Oh, come on. Well, there we go. Um, not sure if it captured all of it. No, that was just build the presentation. Okay. So, I may have to go back and uh, let’s see. die. Go away. Okay. Was it dash r to give you the list would have been this guy? Nope.
J. Langley: Oh well. So initially I I pulled the data set. I pulled up cloud code. I said, “Hey, build me a virtual.” It didn’t have any kind of uh requirements file um or anything like that. So, I said, “Hey, I want to use UV for this. Uh build me a pi project um with everything I need to run, you know, these two files.” And that’s where initially it it said, um yes, um I hear you, but um you’re going to be installing Zet because that’s not a thing in Pi. Um, and so I little rabbit trail went off down that. But then it got uh that’s where we wound up working through the SVO file. If I go see what changes it had to make would have been where this is where the calibration thing gets stuck. So you tell it to disable the calibration um and then set depth mode to none. So basically it’ll do the video for you, but it doesn’t try to build the depth map or whatever because I think it’s trying to do that on the fly.
J. Langley: Um, so it figured all of that out and then somewhere down here I had to change the path to where I was running. Um, and then I did that and got some video or some a bunch of images. Uh, let me see if I still got those or if I deleted them because again, every time I would work with something, I would run out of space pretty quick. Uh, the one row and the four rows. Now, these are both from Rossback, so I think I may have deleted the the stuff I got from the SVO. Um, we’ll play around again um and see if I can extract that stuff again. Uh so after I did that um figured out that I had a bunch of images um that then I had to figure out how to get those into some kind of a video format and even that wasn’t great but it gave me something. Um and apparently the guy that did it, his name was Jose. um I’m guessing uh who may not have a name removal script, which is very interest, very good thing to have if you’re going to work for a long time on something.
J. Langley: Um so then I I flipped around and I said, “Okay, well go grab the Ross bag and figure out how to make it work.” And so that’s where it figured out that well, you can either install Ross um or um I’m guessing the dumpsters out back. Okay. um or you can install this easy, you know, Python package that’ll give you most of what you need. I’m okay, go Python package, give you most of what you need. Um so it did and then it figured out how to make updates to this uh extract data uh piece. Again, some of that was what it was talking about in the uh kind of well, it doesn’t always have GPS, so I need to bail out on that if it if it doesn’t have it or um if the unit of measure was different on the depth, you know, kind of thing. Um so at that point, um it would run um and it would give me left images that are a boatload of left images. Um, one of them didn’t give me right images.
J. Langley: Yep, that would be this one. Um, but it also gave me depth, which was interesting. Um, I’m not sure exactly what it is that was computing the depth. You know, I don’t know if that’s coming off of some other sensor that’s got or if it’s actually a model that that it had on the, you know, on their on their GPU. doing something like that. Um maybe it’ll tell me if I read the paper, which I’ve got links uh later. Uh so let me drop down and go. Wow, that’s bright for me. Um I see why people do that do the dark mode thing. Uh so the one row after I put that back together, um here’s what the one row of assuming this works. Come on. And so you can see just general I wouldn’t have thought that oh here’s a building right next to you or here’s you whatever. Um you can kind of see the shadow of a of a guy occasionally. That’s mostly the robot right now.
J. Langley: But uh but you can see how light changes. You can see how leaves get in the way. Um this one doesn’t quite isn’t quite as jittery. there’s a place where something has a cutout and it’s got, you know, if you were trying to do this autonomously, it’s got to figure out at this point, which one of these rows should I keep going down? Um, leaves all over the place. Whoops. Oops. Oops. And again. Oops. Um, come on. You can do it. um different things. So, some things where the grass sometimes you’ll see leaves on the ground, sometimes you’ll see just bare dirt, sometimes you’ll see weeds, you know, things like that. Um and yeah, it keeps on rolling till it gets towards the end of the row. Whoops. Let me get back. You said somebody’s manually driving this thing. Yeah. I’m trying to figure out there’s one place I think it was the one the four row where it comes out and makes the turn.
J. Langley: Um you can actually see the the shadow. U so I mean that was pretty cool. The four rows um more of the same but this time we’re in more towards the middle of the field where I don’t have as much. I say this is just sun coming through. You know what I mean? You’re gonna you’re gonna have a lot of this kind of stuff. Uh you got some places where something came through sideways and took out some stocks and you got to figure out that oh that’s you know um it’s it’s kind of an interesting problem. Uh, the more I have gotten into this and see like the actual video, the more um I’m even wondering how possible some of the things I’m trying to do are. Is there a speed setting on this bad boy? Speed. Speed. Supposedly they’re using stuff like this to deliver like a precise dose of fertilizer or antibiotics to the plants. They’re kind of driving through the If you are over the crop, then you can’t um and generally that’s over the crop before the canopy closes because after the canopy closes, if you go through there, you’re you’re messing up leaves and other kinds of, you know what I mean?
J. Langley: Um even if you’re on a high boy with booms out. Um did we already make the turn? Yep. Okay. Yeah. Um let me go back to normal. But you can see how jittery some of this is. Um, well, there’s a lot of stocks and stuff you got to run over, too, right? And bouncing and different light and um, you know, oh, here’s a leaf, you know, good or bad. I mean, isn’t that what the robot’s going to do itself? It is. It’s just the thing that you’ve got to figure out, you know, something. Um, and again, I haven’t checked out all their their their data isn’t notated quite to the extent of, hey, I’m looking for a cloudy day. You know, there’s not it’s not like, oh, this day was raining or this day was sunny or whatever. You just got to go through all of it and figure out if I’m trying to if I’m trying to do a specific and if it’s manually, you may not have done it during rainy day.
J. Langley: Now, I think they tried different weather. I don’t know when they say weather what that means. You have a better look at the different sensor time and running it at night so there’s not all the light. Yeah. I mean there’s all kinds of um one of the interesting things is um this camera is a little higher than what I would probably put a camera, you know, especially if I’m if I’m trying to get around everything except for Yeah. I mean and like what do you do for that kind of you know? Well, let’s plow right on through it. Um, it’s not a very good driver. Um, help the dogs realize that’s supposed to be a road. I guess. Yeah. Um, do you think music would help? I think it would. Uh, plants have been there have been studies on how plants uh work with music, how they’re affected by different music. When my grandmother traded in her fort, there was a cornstck growing passenger
Lorin Bales: So Jay,
J. Langley: floorboard.
Lorin Bales: what I’m I’m curious about is since you’re I was I was waiting until I could see more data. So I’m thinking like if you go down like four rows, you could even like fully simulate it, right? Like I think there’s an Nvidia Isaac sim that would actually ingest all
J. Langley: Yeah, that’s the next that’s the next task I’m going to give it.
Lorin Bales: that. Oh, sweet.
J. Langley: which is, hey, um, I got all of this video of stuff. Um, is there a way to automatically create a world in Isaac sim that mimics some a piece of this or something like that? And after you get it in there, you can actually change uh and again, this is more on the digital versus you’re still going to run into the digital to live. Is there error when I go from one to the other? Uh, but you get it in the Isaac sim and you could actually say, “Hey, I want a moonlit night on a, you know, um, you know, things like that.” Um,
J. Langley: windy. Um, one of the ones I I actually looked at had a decent amount of wind. So, right now, all of the leaves are moving because you are moving past them from a camera perspective. This one had wind moving leaves as well as the camera moving through leaves, you know, and it was it was pretty interesting. Um, a infrared would be kind of interesting because it would pick up uh again the leaves would probably pop white um on it, but everything else should be quite dark. Um, does it have Does the data set have like GPS data or like where it is? Yes. Does it have Well, it has taken it has GPS data for where the GPS thought it was. So, it’s GPS plus IMU. So, so you were saying that it wasn’t annotated with weather though. But if you have when it was taken and where it was taken, you could figure out what the weather Yes. Yes. I was thinking Yeah. I was thinking about like a cheap little 5 foot light system cut.
J. Langley: Another approach some people have tried and this may actually be the the dumb cheapest way to do things. Uh take your GPS, put on a freaking pole, get it above get it above the crops um is not the most stable thing um in the world, but it’s you know it’s possible. There’s other systems they use for like golf and stuff forth where they’re they’re tracking the ball. Oh yeah. And they put up these antennas around the sides, right? So that they can use a different system than GPS. That way you could penetrate the the canopy maybe a little better and have a higher resolution. Yeah. Let me do a uh the other thing from a training data perspective is you could just 180 flip all of your video and just double your training data, right? because it it doesn’t seem like there’s anything specific to like the sun being in a certain location because that would change throughout the day as well. This is the track um after so it converted from what it does is it’s got one position that that’s a known position.
J. Langley: Um and then it converts everything into meters offset from that known position. So think of it as nearly as like an east north up kind of a kind of a thing. Um, and you could, this is it trying to capture going straight, you know, and yeah, it did kind of run into some plants and have to back up and whatnot, but you can see how off um, some of these are, but it seems like that’s because the GPS when it starts knows where he is and the IMUs are the ones making these lines from there on. No, this is actually a feed from the GPS. So, if you were trying, what this is showing is if you were just trying to use a GPS to navigate, this is how bad it would be underneath the canopy. Um, and then inertial units. Um, you can you can uh crap, what’s the right word? You can initialize them at a known point. Uh, but every you’re basically the error in them compounds the longer you go without resetting to a null point.
J. Langley: So, it’s going to drift. Um, and it may drift back, it may not. Oh, so if you could get the end of every row, you could get a reset, right? Reset here or so. You know, there there may be some other ways to to do it. Um, so from that video, what are you specifically looking for to like are you looking for like weeds? Are you looking for something like what right now using the data to like train? Right now, I am trying to see if I can train a robot to autonomously travel the rows of this field without a user telling it where to go and what to do. you know, can it pop out the other side and know to, you know, hey, when you get to the end of the row, turn around and, you know, run the next row over. Is that even possible? I have a dumb idea. Yeah. Ultrasound sensors on the side of the robot so it can tell how close it is from.
J. Langley: You can do things like that. the you get into weird leaves and other kinds of uh I think as many sensors as I could get tagged in and and combined into some different way might be a way to go. I think I’d start off just uh can you get it just to recognize stalks, right? Ignore everything else. Tell it to stalk as a positive. Everything else doesn’t matter. And then plan a row between anything that identifies as a stalk. And then that’s what one of the thing they were talking about. They called it closed loop. Um which is if I identify a stalk in one frame. Um and I Whoops. Wow. That shows you what Here’s a frame. Um I didn’t mean to watch. At least don’t go. Right. Yes. Well, the thing is if I’ve got this frame and I’ve identified where these stalks are and that these are stalks and then I take the next frame, do I know that the stalks in the next frame were different positions of these as I moved or are they different?
J. Langley: Are they new stalks that have shown up? How do you this talk in a way? Yes. Um, and so that’s one they one of the things they were talking about and the the paper that we’ll I don’t know if we want to cover that paper they used for this because it was pretty interesting which actually before we get going let me flip back over to the slides um and actually finish that um I mean couldn’t you average just say okay here’s the detected stalks and then a between you know the focus You can do that. Um, then you get to the part where Yeah. Well, that’s why I’m thinking like very falling over. Don’t recognize anything. And there are a couple of places in this field where there’s actually gaps where you’ll, you know, something happened in this spot and you come out and it looks like you came out of the field because you don’t have stalks anymore, but you look, you know, five feet ahead and the stalks begin again. And I’ve got this row, I’ve got this row, this row, you know, that are all visible from my spot.
J. Langley: You know, it’s uh it’s interesting. We’ll see. Um yeah there’s that uh we did that. So the concepts that they were so this paper that actually went with the data set they tried a bunch of different models on this data set to to calculate the error in measurement of where it was. So, uh, we’re not going to cover all of this right now, but this is just something loop closure is what the concept is where they’re talking about to see, um, if I see the same thing, do I know it’s the same thing? So, if I see it, but it’s in a new position. And if I’ve got good loop closure, that means I can now detect that I’m x amount of space closer to this item. Therefore, I’ve moved how far, you know, something like that. Uh, let’s see. I’ll get through some of this. So, this is the other kinds of stuff they there’s all kinds of different models that they they pulled. Um, Orb Slam 3 is one.
J. Langley: Vince Fusion. Uh, anybody like common filters? I hear that a lot in radar. Uh, accurate scored. uh they their accuracy was basically they knew where the robot was and they were trying to see where the robot thinks it is and then building some kind of an air model out of that. Um the other thing they’ve got is not only position but which way am I facing and where am I oriented? You know, how far off is that? Um just to clarify, you needed a GPU to analyze the data set, but not for the live function of the robot, right? No, they had one on the robot itself. Okay. um trying to to do a lot of things as well. Um that SVO data that I got actually some of that is part of what was generated by that GPU. Um you know scenario one uh they’ve some of the scenarios that they talked about in their paper um the good one um they had about 1.2 two meters of air, which is a pretty freaking big error if I’m trying to I got I got 40 inches or 20 inches depending on how you plant your rose.
J. Langley: Um, some of these worked okay and I’m still trying to figure out where bag of words fits here, but it’s a thing. Um, see as this was going through that, so their their paper was pretty much ace. So they they built the data set and then in 2022 they found the best models that they could find that do, you know, uh, visual uh, uh, location mapping kind of stuff. Um, and how far have I moved, that type of thing. And then they ran those models on this data set and captured how well those models performed. And so the paper they provided was more of a survey of different models and how they do on this data set. Um so that that may be an interesting thing to cover. Um or we could actually look into hey a lot of time has passed since 2022 and there are actually new things out now. um and then see what it would take to take the new thing and do the same kind of a hey if I if I train the new thing on this what would happen?
J. Langley: Yeah. One other random idea of what about and this might be more expensive or harder but just an idea what if you triangulated signals you have a cheap transmitter put each corner cheap sensor and the robot just had to take how far from it was. you get so much you get the same kind of bounce uh that you would have from an RTK. So an RTK is kind of it’s a separate device you put somewhere in the field that is an exact known position and then you got GPS plus the difference between the RTK. So that’s how they do precision location stuff. But that’s why it’s a hard problem. But of everyone I’ve seen so far has gone into the well, let’s figure out a way to get above the canopy and either drop stuff down each row uh to do with what’s called side dressing in corn where you apply, you know, nitrogen later in life. Um but nobody yet has figured out how to actually, you know, navigate autonomously um up and down roads that I found.
J. Langley: Of course, this could be like Claude holding the data set back from me for three months. It may go, you know, it may wake up, you know, in September and go, “Hey, yeah, that thing you were asking about for a while.” Yeah, here it is. There’s a world model for a robot, right? Even though I’ve been helping you build this, use like a Chihuahua version of one of those little robot dogs from Boston Dynamics or whatever it is. Yeah. build on video, you know, recreate 3D or another physics capable and just train an AI and get it to teach itself how to get, right? Just be like, “Hey, figure out how to follow.” Yeah. Um, self-improvement. Um, that’s the thing. U, so that would be cool until you get down, right? Yes. Anybody watch the What it would do would be look at my email and try to bribe somebody to think outside the box. Oh yeah. If you guys watch the videos of people like trying to play racing games just running like 10,000 variations and find the fastest way like hey just set their model into the corn fields like here’s your race.
J. Langley: Well, I guess you’re talking about the world model. So, you’re thinking about doing like a JEA. I’m thinking about I found one paper for Py Jeppa um where they did I don’t know if you’ve seen it. Uh there it’s it seems complicated. So, I may actually have to go back. Um they’re looking at policy uh and planning things like that. Um, so and again, I just ran into some of this stuff today, so I’m still kind of picking pieces apart and figuring out how much of it is uh fluff to get something published versus actual good stuff. Um, sometimes I’ve run into papers where they spend a lot of time trying to optimize one thing that I don’t even care about, you know. Um so the the PI JEA it used VJEPA it also used the JEA world model kind of separately as a frozen thing. It uses something called Octo which is a policy type thing and it was using that to help so when it’s trying to search for a a valid thing it can get there faster.
J. Langley: Um, so the other thing that’s a little different in this one, um, I probably have a lot more, um, if I was the right way to say this, time doesn’t really matter as much if I’ve got an autonomous thing that’s just going up and down rows. Some of the other things are, oh, we’ve got to plan fast enough to be able to do this like real time to meet some kind of other constraint. you know, if I were working in and around people or if I had a certain amount of time, I needed to plan my next MOO. Um, but in this case, that’s not the driving factor. So, some of the things I’ve been trying to optimize for don’t really matter as much for me. But trying to figure out how much of this is like that or not, but yeah, that’s kind of what I’m thinking maybe next. You should be able to. So, you don’t have to do a lot of post training on JA. You only need a certain amount of video.
J. Langley: My concern with that data set is that it’s kind of a weird data set. What are you trying to train it to do? But can you combine um you know the data set that you do have and then maybe fill it in with some like a diffusion model or something like that create some scenarios that are more like what you’re wanting since you have so much of that data set. kind of going into the cosmos Nvidia stuff where they’re doing data or video generation kind of fill in the gaps in their data set a little bit. Yeah. Um I guess my my only concern like like what you’re trying to does does does the data set that you have have any concept of an action in it as far as like what the robot is doing at any point in time because what I’m I don’t Yeah, I’ve got the data for what the robot um basically the Ross Yeah, you have the Ross. I’ve got all of the topics for what motors were running, what wheels were, you know, I mean, things like that.
J. Langley: Um, what I don’t have is the intent of the driver, right, of trying to go this way or trying to go that way that I know of. I might, that might actually be a signal I’ve That might be in there. I don’t know. There’s this little toy from an for 200 bucks. They they still make them apparently. Uh, it’s the monkey Vector 2.0. Yeah. It’s a semi-autonomous robot. Uh, it’s a toy and you can get telemetry data out of it, right? That that may be something to look up at least for comparison. Yeah, I’ve been looking up uh a bunch of different kind of frames. You know, a wheeled robot isn’t necessarily the best thing um for where I want to run some of these. Um, if you could have your robot be off the ground a bit, you would eliminate quite a few issues with like that data set, right? Because if it’s like three feet off the ground but still under the canopy, then you would just need like sensors around you to detect like stocks because presumably you very rarely would have a stock that has fallen down and it’s still at least like three feet in the air or some some notional.
J. Langley: I don’t know what the number would be, right? Problem with that is the time the field’s different every every week. Well, it’s uh it’s different every week. It’s different between, you know, who knows the last time, you know, one of these sprinkler heads came around. Um it’s probably it’s probably different when the leaves are wet. I don’t know. Um, I should be able to get that actual information out of the data set as well. have two rows. They go, they find the end of a row, drop a line down the line. You just roll the line. Yeah. And then it gets to the end and it goes down one row after pretty much. Sometimes Yeah, that’s another thing I’ve been playing around with. Um there is a field on my way into town that is now cotton I believe on the right um coming in 72 and I’m you can watch it. You got the rows um the rows are going this way and they get towards the edge of the field.
J. Langley: They turn and they follow the edge. Yep. You know, so I mean that’s the thing. Um and then so it’s doing that. You got other rows that just come to that one and stop. Yep. You know where and when. So, it’s uh I’m still on the fence as far as whether it’s even possible or not. But, have you looked into like like the like the autonomous like lawnmowers just to see like what sensors they have, how those like or if there’s any like crossover there. Mostly what I run into is you got to have a good GPS. Okay. Um that’s where most of that is is operating. Oh, I sent you a link to sell robotics. Okay. Okay. a combination of pallet and nice kind of see what he did there which could work like Intel came right they have a little GPU for them yes well there’s an AI GPU it’s made just for AI that is not made for graphics.
J. Langley: Okay. Like for for IoT stuff. Well, with the Raspberry Pi, you can Yeah. It’s a new new plate or whatever they call a new hat. Okay. Interesting. I think that’s it. Uh let me see if we got anything online comments. Inertial measurement unit. Yes, thank you. Oh, 51 minutes ago. Oh, that’s great. Lorin has a couple of those. Yeah. On her car, telescope. Yeah. So, interesting stuff. Um, one other potial idea, uh, just kind of spitball here. You could break this up into two steps. You could have one step that you send a drone out on top to view like the canopy from the top down to effectively like map out, right? Right. And then that you would have significantly stronger GPS. Then you have like a smart way of then using that data to pass to the robot because now you’ve effectively scanned the field. So instead of having the robot or the autonomous vehicle in the canopy doing all of it, you can essentially have like the answers to the test ahead of time.
J. Langley: Right. So maybe triangulate a little bit too as you go. What if you not getting all the information, but if you can get sparse information, you might be able to that would be interesting. Basically, it’s a duel. Yeah. Plus, you would know where your robot is. Yes. Because there’s a drone right now a little bit. It may actually fix the time constraint issue if I’m not multiple drones then so you can have the battery time. Oh maybe um you know I’ve I’ve had I’ve had Gemini working through logistics of how many robots and how fast would it take to cover how many acres and it it gets pretty interesting. smart. Um, and then sweet spot as far as how many versus how much they cost and then how long you can run on a battery and if you’re carrying things and you know there’s a lot but yeah also hybrid yeah motor driven by power is generally long right you could have them field premapped just by using the tractor to put down the seed yeah you could but you know what the plants are going to do or how they’re going to grow or things.
J. Langley: You’ll you’ll notice some fields you’ll have um smaller plants in some parts because they didn’t grow as well, right? And then other spots they’re, you know, twice as high and then figuring out, but you don’t need to know that. You need to know where that row is. If the row is really short and in your way, right, you still know where to go between maybe. Um, as long as you I mean the problem is you can tell it to drive straight. Mhm. But if some of the wheels slip or something happens all of a sudden I’m not exactly but you have GPS because it’s so low you know you’re slipping. If it’s if it’s low Yeah. Oh, when it gets high I don’t. Right. Yeah. And just play around doing things. Yeah. Definitely when you start driving a track you what you always have a tracking wheel instead of monitoring the wheels that are actually driving it when the pulls behind that it measures all it does it’s not driving right it’s just just measuring it’s just a tractor interesting huh that’s why saying use the tractor to cheat to do the mapping right to start with just so you know hey the road’s gonna curve It’s going to go around this way because the tractor did.
J. Langley: Yeah. Because if I can figure out how to do it, there’s about eight or nine different applications that would be useful that you can’t currently do. So, the most fun so far being put a freaking laser on the bottom of this thing and go zap some weeds. Oh, they say you’d be uh doing tilling. go down. Yeah, that’d be very important to stay off the plants. Yeah, the weed killing. Um, and I I’ve got some good data sets for weeds. I’ve got a good data set for insects and pests and things like that because uh most drones that are doing it are looking from above the canopy. A lot of your pests aren’t on the top. A lot of them are up underneath leaves um doing things. Um that’s kind of interesting. Uh been thinking about uh Hudson Alpha has some projects going for bio stimulants. Those have to be applied on the stem of the plant after it’s up. So finding something that has the precision actually see a stem.
J. Langley: And you know, something like that would be would be, you know, there’s some things that are possible um if you figure this out that aren’t possible at all today. Um without a person doing the work. So, I’m getting close to being out of battery. Um I do have cord. I just don’t want to use it. Any any questions or thoughts online for those still on?
Lorin Bales: That was great, Jay. Thank you. It’s very interesting.
J. Langley: Yeah, there’s a projects thing now that I can drop things into. Oh, there’s a whole channel for that. That’s awesome. Um, all right. Well, that we will. Do I need CUDA to play with that data set or will you can it’ll work. Um, you you need CUDA to run anything that’s got the SVO files. Okay. the Rossback stuff that’s pure Python. Okay, cool. So you can pull it do all the things. Um huh. The data set that you want to download links to that. Yeah, they are in they come in the repo. Uh so you download the repo, it’s got the shell files with all the wget stuff to go get the link. Yeah. So I’ll just in case Yeah. Um, so what I’m looking at next is actually pulling all of this data um and getting it somewhere that I know I have access um because these Ross bag files I mean some of these are like 10 gig per um yeah and I don’t have room for it. So all right gonna let’s see stop presenting first and then we’ll kill the meeting off. So thanks for coming.