Transcription provided by Huntsville AI Transcribe
This transcription was horrible, so we’re still checking to see what went wrong. To stay transparent though – here’s what we got 🙂
I got to come in with something and they’re not laying new cover. Luckily, they got ahead of it a little bit, but. uh we did one with I try to remember the there was a competition we got a little ways into that was the sound thing trying to take the the music and then decompose into the I don’t know if it was an AI crowd or it was one of those things um and then another one we looked at was a City learn I think something like that, but their data set was crap. It was either the gym or the data set. Both, okay.
So some of them provide like an SDK or AI crowd always provides like your starter kit and that’s where you start from and that’s how you tell you what gym to run and how the evaluation works and how to submit your stuff and it’s really cool. It’s really easy. But when there’s, you look at the data and it’s obvious that, well, this is not a thing that somebody’s thought through.
And then the gym doesn’t run or the whatever, you know, there’s no, like I, there’s not the new year. Yeah, my preference is the open-ended ones. Like I did the open data science conference act a couple weeks ago and that was do whatever you want.
That’s cool. Yeah. That’s cool. That might be something. So there’s a lot of those either online or in person. The thing that on the music one we finally got to the point where it was obvious that there were other people in that competition that had like academic level horsepower. servers behind them and there was nothing we were gonna do.
I mean, they’re dropping like eight sufficient today of new models trained and we’re at like one every other day just because I’m, you know, borrowing Ben’s server there. But so some of that we probably don’t want to hop in, you know, some of that you look at and go, oh, you got to have a cluster somewhere to even enter.
Yeah, that was the resource thing.
I find the best thing to do is look for the places that are all about startups because startups don’t have all those resources typically. And so if they’re really expecting you to pitch something that’s built in a day or two as opposed to somebody who’s got an academic think tank, that’s right.
Some of the other things like on AI crowd there are some There are some competitions that are like always open There’s one that they run for the Spotify.
How do you do this?
How do you do the song recommendation? There’s actually a benchmark and so they’re basically been trying to see who can inch that benchmark higher and higher and higher There’s nothing behind it as far as the competition as far as prize money or anything But it is a if you’ve ever wanted to submit something to a competition or something like it, just get that under your belt the first time. It’s a pretty decent place. No risk. You don’t have to tell anybody if you were bad at it or whatnot. There’s that. It was a little depressing to go look at some of the original stuff we were doing in 2018. One of the first thing that I was working on was NLP based. trying to figure out uh you might like the first thing I ever did was Beijing uh to do a probabilistic model of uh initially it was in grad school to figure out if you gave me a question I could tell you which news group to post that on to more in line with the topics of that news group to get you a problem better probability to get an answer um So took that flip that over. I think about a leaf actually I’ll show you this real quick just to give you an idea of what it is We were doing and some of you have been around for a while. It’s heard me talk about this way too much. Oh You’re doing clustering of the news groups. Sorry.
I wasn’t at that point What I was the so the problem statement is this was actually like 2017 Based on the eclipse foundation and the amount of source code they have So I added time somewhere 75 something million lines of code and at the time somewhere around I was looking at a number of committers less times 10 but so anyway um so they got past 1500 committers for 70 something million lines of code howling new world Are you supposed to maintain that amount of code with some people? Which is a problem we still have on nearly every program I’m on. So the thought was, if we can get into the, they were using bugzilla, so we scraped a bunch of bugzilla data, and the concept was, if you go sign up and say, okay, assign this bug to me, I’m gonna fix this. We wanted to recommend the next three that you might. I want to look at because they’re highly related to the thing you’re working.
So it was a lot of, at the time, it was Word2Vec had dropped, Glove had dropped, Doc2Vec was the thing we wound up playing with and then doing a similarity between, you know, and then looking at, well, I could probably just host all of this stuff into a chat in LLM and just ask it now.
Oh, I’m like, crap. That says Codex on the bugs. Yeah, I know.
which might be something interesting to do. Some of, especially if you focused on some of these open source communities, I still have some contacts back at Eclipse Foundation. If we wanted to try to get something together and host it, you know, get them to run it in their CACD somewhere, you know, to do that.
The other thing from their perspective, they do a lot of If you ever work with any of the foundations like Lenox Foundation, Eclipse Foundation, Apache Foundation, one thing they have that’s helpful is a freaking team of lawyers that can check license stuff. And so they’re really, they care a lot about if you’re going to commit something that it doesn’t have things in it. that have other kinds of licenses associated with them or that are blacked up. Actually, I made a lot of money off of some of it. I actually got a nasty grant. I was contributing to a Lynx Foundation product for a while. And I got a nasty grant because I had some things that flagged on their enterprise level sort of things. I have this person, I’m like, I’m just a dude. Yeah, I got the Linux cyber on me. Like, OK. Like, from Linux.
But just thinking through that could be, that might be an interesting thing.
One of the things I’ve talked to Josh about this one before and that sometimes I find something fun and I will fork the GitHub repo into my own thing. And then the repo I forked just keeps moving on. It’d be really cool if I had an agent that would check occasionally and go see, hey, is this move, do I need to pull this and can it do it for me?
especially if it’s not trivial blah blah blah even if it even if it’s uh what do you want it to do that okay so i’d like for it to make a merch request for me that’s ready for me to review okay so you’re in the loop though yes It’s just a fun thing. It would be nice to get rid of the Dependabot alerts from GitHub. It’s all like these projects that I’ve had from like five years ago and it still has a security vulnerability. I’m like, well, I’m never touching that coding.
Hang on, hang on, hang on.
Stand by. So, Dependabot alerts, if you were to go look at… presentations.
2018 has all kinds of pipelines and notebook code and Python code and other kinds of stuff I get like daily.
Yeah. Oh, that’s cool.
Turn off the notifications. I will after I figure out how to do it. They wrapped to a specific folder that I never told you about. Yeah, so.
No, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no I’m guessing if you built a project like that, by the time you started getting an attraction, all of the other tools are going to have it already available.
I’m actually a little curious why some of them don’t.
Or like a keyword, free-bow-up-the-date sort of thing. Yeah. Integration, probably. I mean, to do it’s easy, but to get it integrated with all of your leaps and stuff, you know, you probably have to do it in a certain… What about… having a massive number of bugs and bugzillas somewhere and being able to group them together to say, hey, here’s a collection that are likely in the same code section.
You know what I mean?
Something like that.
Or if you’re working somewhere that has been referenced in a bug to actually bring it to you, you know, if you’re already, it’s more of the SonarCube approach of, hey, if you’re in the code already, you might as well fix these things while you’re here. We got a lot of good stuff out of that with teams that are just, you show it to them and they’re already in there.
Sure.
I mean, it’s no sweat on my back to change a Sprint F to an Int Sprint F. Yeah, go ahead.
And it’s not extra review time later because we’re already going to review the file.
You know, stuff like that. I don’t know if that’s, I would have expected Atlassian or somebody to actually already have that nailed. See like a deep app sort of thing. I think, I mean, it’s a lot of people that try that it gets really hard.
I think when you have lots of languages and lots of systems and frameworks is like, how do you successfully, you know, chunk the code base and do work, you know, what level do you do those annotations on it, right?
It’s going to capture enough context and size that you’re going to be able to track the bugs because sometimes bugs are wide, you know, and not very coupled.
It’s kind of a hard problem. Yeah. Okay.
That might be why.
The other thing that Bugzilla drove me nuts on is it had a kind of a way that when you reply to a comment, it would take everything previous to that and do the, you know, whatever indent and put it right below your comment.
And anybody replied to that, it would take a copy of that indent and put this thing up and copy of that and put this thing up. Oh my gosh.
And some of these you can imagine the What do you want to call it?
Techno religious arguments. The developers tend to have long running on what’s the best way to do, you know, the best one I remember is the proper live return at 80 characters or 120. I had two people on a program that just were like going after it as to which one was the right one.
And all I had to do was the size of their monitor.
that was that was something the other one that I had dropped in just just for reference if I can find it this is something we had done I don’t even three years ago so but further back to that I don’t even know if this is a problem anymore, if they’ve already found ways to do this.
This was an implicit recommendation. So in other words, you don’t know anything about the person. So if somebody walks up to me and they’ve got four items from my store, how do I recommend the fifth one?
Just based on what they brought. I don’t know their history.
I don’t know how many times I’ve been there. All I have is the thing.
And the bottom line is, and this is the McDonald’s approach.
If you can get better than just recommending the most often bought product, if you’re golden, that’s why they do that.
It’s really hard to beat that. Because people normally buy the most bought product, just in general. So there’s been a lot of papers, and I kept coming across this when I was looking at the multimodal embedding sort of stuff. It’s a concept of a semantic ID that they are starting to try to play with to kind of take a skew. and find a way to embed things about that item, whatever it is, you know, a song, a pack of twizzlers, whatever it is, and then have some way that they can index all those things together. So that might be an interesting thing to play with, is like to be able to do some of that multimodal stuff, and find a way to do that recommendation with, you know, something that can be a product or something else.
But it’s something that It’s kind of, I think it’s hard if you’re trying to do it for real, for real, but to kind of play around with it and do some fun nifty things, you can do it for products, you can do it for code.
The concept for this one is all of the people that went online with their stores and whatnot, you get a lot of stuff from Shopify, because Shopify knows what I’ve ordered before. It’s got a lot of interesting things. If I’m a store owner in a physical retail place, I don’t have any of that until I pay. And at that point, I’m almost too late in the user experience kind of thing. The other thought was if you had people working at your store that weren’t as knowledgeable about your entire product set, especially I got a cashier that’s minimal wage or a little plus. I don’t expect them to be an expert on all of our products. So just some helper thing to say, hey, people would buy these three things, normally buy this other thing, recommend this or something. I learned a ton in this about cold start problems, and when I get a new product that’s never been bought before, there’s no data for it.
So what the heck?
See, you got some randomness and some fun stuff. Anyway, I did find several different data sets that I’ve linked in, and some of them actually have a decent amount.
And by decent, I mean in the millions.
That was one of the other things that might be interesting to take another look and see if there are additional data sets that are there now that weren’t there three years ago.
Because that was fun. Also learned that grocery stores don’t need this at all because if I bought bread, it doesn’t mean any really anything. That was the other. People buy just absurd things based on what they might be out of or what they’re there for. There’s actually a, I don’t remember if I’ve got it linked up here, but it’s an interesting data set if you were just looking at, I think I threw it out because I could, you can’t do anything with it.
It’s nearly random.
I didn’t even realize that until the next time I’m in the grocery store and I’m buying like bubblegum, eggs, and like a lemon. Nothing could predict what that was going to be. So from that that’s just stuff we’ve done in the past Just throwing that out there and then one other thing and I’ll shut up We were on a call Yesterday one day was it yesterday with Larry. Oh, yeah. Okay.
Yes, but the call that we were we’re doing stuff for the AI symposium coming up in January and Larry blows now the city’s Chief Innovation Officer. And previous same committee member is me on some of the answer stuff. So we’ve gotten to know each other at a decent amount. I’ve had discussions with them about city data.
Even before they went, you know, when they did the garbage truck thing and then everybody was like, I don’t know if you’ve seen the city council meeting. It was quite interesting. Comments from the people. Yes.
We already had the discussion about, well, who owns the data?
If you have garbage trucks rolling around, collecting images of houses, and then doing whatever is that owned by the city. If it is owned by the city, that’s actually valuable. I haven’t been able to make much progress there, but it is something I want to continue pushing. Normally, our city likes to offer things like tax subsidies or ease of you know, helping with permits to get businesses to move here and stuff. I’ve seen other places offer like for like AI kind of companies offer compute time on servers, things like that.
I don’t know that I’ve seen a city offer a data set as a luring way to get people to come as long as you’re here. You can and here’s our data and it’s like something like that.
It’s valid validated and all this kind of stuff, but it’s a city asset.
So if you get approved to come in and work through this kind of thing, you’re not only you’re helping the city with something, but you can train your model on actual real data.
Of course, synthetics all the thing now, but that’s just thinking through some of that.
And then they went and did the city council. There’s a few cities that have open data.
okay systems yeah like i know boston chicago they’re all bigger though i don’t know if smaller ones do okay there’s some small ones that do but it is mostly big cities if i head okay um that’s the other fun thing well austin doesn’t california wants the california city boston’s an ai city you know and you’ve got that i would be an ai city Oh, there’s a lot of that going on. So yeah, manipulation might help. I don’t know. Marketing. Marketing. Some people call it leadership.
I don’t know.
It’s not as hard to tell.
That was something. But anyway, Larry was at his call the other day. And this is kind of in the way back machine. Before co-working nights started, There was one group called HackConsul that was meeting at the AL.com building downtown. This was around 2015, time frame, 16. And then there was another group that started up called Code for Unsul. It was a derivative of Code for America, a nonprofit. You go basically, the thing is you get a bunch of techy kind of people and you want to do community-based stuff. They give you an umbrella kind of a thing along with several projects that have already been done in other cities. And so we actually made one of these, went and talked to the city about getting access to some of their data. They had just started up the trolley system, and one thought was we could put some sensors on some trolleys and put it on a map so people would know where the trolley is. I mean, I walk around downtown, I’d like to know to go this way or that way. And it turned into a really interesting, I don’t know that they had a good understanding of what data they have.
And they definitely had a fear of releasing anything that might be sensitive. But they didn’t have a way to, you know what I mean? It’s like they were missing that whole section that they may have fixed by now.
So that could be something, another thing to think about. Because what he mentioned on the call was one of the people that they work with, it’s an emergency management cognitive person. They currently, access all kinds of open data sets, or for like flood prediction, or I’m not sure what all they do. And then they’re trying to do some kind of a task with that.
That might be an interesting, if you have a person that’s actually doing something, and if you can find a way to help do it better or whatever, I don’t know, it could be a good replete thing. It’d be kind of fun. Invite that person to come by on a Tuesday night. That’s what we might wind up doing.
Or, I mean, that’s what Larry offered. Hey, we can get somebody in and go interview the folks and see what they’re doing and whatnot. You know, the city is looking for ways to be smarter. I actually believe them when they’re trying to do that.
They try all kinds of things. Some of them are great. Some of them were kind of okay, really.
But they’re trying. Well, I think that Larry will be a big help with that because I know he’s still, I think he’s 30 days out for me or he’s only been in his role for like 60 days.
So I know he’s still working through like understanding all them, but he’s in like a different meeting habit. And he’s meeting with basically every department head across the city as well to try and figure out like, I think data standardization seems to be the biggest thing initially. I’m just like, I’ve got lots of tools. I’ve got lots of different data stuff.
How did they start to try and manage some of that. But then his vision, it sounds like, is a lot more how do we start leveraging that data. So I think he’ll be definitely the best person if we get him and then he connects with someone.
He’s the one that got us in to talk to the city planners about where AI has been useful in other places and what the, you know, and we did some fun stuff where it’s useful and then where you better watch out for it.
You know, policy needs to be right, you know. I don’t want a self-driving car parking in front of my house just because that happens to be the best place that it thought it could park. Yes, it’s legal. No, it’s not normal. You know, we don’t want to park it in the lawn. And then the other part about the, I mean, I don’t know if it was Austin or San Francisco, the car kept parking in front of this guy’s house. And so he did what normal folks do and took some traffic cars, but one in front, one behind, and now the car’s sitting there. It can’t go anywhere. So somebody comes out, you know, and so that problem I think fixed itself. Slightly more petty version of that drone with fiber optic cable. And then I know from the stuff y’all have been working, I know Lauren, you had talked a little bit about, I’m not sure what, but it was a data and medical or a, I’m going from memory. Yes, that’s okay. So we have, at CULSA, we have data scientists, and we actually have equipment.
And some of them are biologists, right?
Jason Batch didn’t let that group. And he just took anybody, anybody who was really interested and had a circle of friendliness that you guys have, right? That’s what it’s all about culture. So we have several people interested in just like healthcare data science that come inside. kind of change it up from defense because we’re all doing defense. So the project we’re looking at right now, we kind of like let it be open and then we’ve kind of like still been, so it’s still technically open to other ideas, but it’s a proto-omics, so proteins from the genes.
And then you take that data basically You want to think of it like a graph network, like graph database, but you have controls, healthy people, and then you have people who might have cancer.
And you know this, it’s supervised. And then if there’s a graph there on that work, they use separate.
Apparently you and me just did a hackathon like last week.
Yeah. Because a bunch of random people I know from another program were all like, we’re coming to Alabama. Cool.
I had no idea. But UAB, we’ll have some stuff. I don’t think it was specifically proteomics, but it was, I’m going to mess it up. It’s spatial, something else ending in omics, but it’s similar idea. Very similar idea.
There’s several of those, right?
Genomics in general right now is a big thing to add to the end of just anything for having AI search through things. if you put a genetic in the front. For a genetic AI in Alzheimer’s disease, like today, I can’t find like information about it, like what data is available.
But my guess is it’s a lot of imaging data. And they’re looking for ways to speed up similar probably to what you guys are seeing. Is there something with LLMs, a genetic AI, any of that that can speed up identifying problems or innovation areas to help move research forward faster. That would be cool. Try to throw it across. The main goal really, from my perspective, is one, to do something interesting. Actually, three things.
One, do something interesting. Two, do something useful. And three, do it in an open way where people can learn. Because even if I don’t know anything about omics or whatever, I can build a graph database, but I can do something I know and then learn something I don’t. So that’s just kind of the mindset. So there was an additional paper. There’s the GoNet paper, and I can link this or something, but there was an additional paper that was like, well, we just went out and got the sequences and got the data we needed. Here’s what the distribution looks like. We’re done.
And in the conclusion, yeah. And you can tell that that’s such a significant amount of effort just to get data in that world, right?
So in the conclusion, it says, well, we need a classifier.
Yeah, you’ve got to publish the data set first before you can do anything else for any of the medical stuff.
One of the most cited Alzheimer’s disease papers is the, hey, we put together the national standard. It’s called NAPNACC.
And it’s just like, here’s how we update it.
Here’s the protocol for everything.
And that is one of the most cited papers, because you have to have something to reference in the future.
And I won’t rant about it, but academic publishing is also a grift.
So you have to put some stuff out there. But yeah. Does alpha-fold or alpha-genome, is that played into, which all are doing it all yet?
I did look at that.
There’s a point where you could do that in the database, right?
and you could come off from like this gene node and do that, yes, if possible.
This is from an interest area point.
If we could do something that has diffusion in it, I think that’d be super interesting.
No other reason than I think it’s interesting.
Yeah, it does. Great. Yeah, I was just curious because I don’t know biology at all. But Alpha Fold and Alpha Genome have been just like super fascinating to like learn a bit about. So I was just curious. And I looked at that. Where are they overlapping?
They’re at license. So of course, I’m going to go, I’m going to look at the data. I’m going to look at both the licenses. It’s my usually my first step. We can’t build the, we can use it as a classifier.
But you have to be very careful not to build the same classifier, the Alpha Fold classifier. Okay.
Interesting.
One of the things that life keeps that I really like about the A.I. side of all of the things is all of the papers that we want to see are there.
You go to archive and you look at it, you read it, you go all the way back to touring and you read his paper and you go further than that and you go to Asma. You read his stuff and it’s all out there. If I want to do medical stuff, all of a sudden I’m in the Journal of Nature or whatever and wait a minute, I can’t get there from here. And my library access finally got cut off.
They finally realized I graduated. Oh, no. It’s been almost three years, but like two months ago.
And I was like, why can’t I get to this anymore?
It’s like, you have to be on campus with your alumni thing. Well, great. I’ll live in this city anymore.
Take one class. You get full access to everything. Or you can sail the sea. For 500 bucks, you can take a class. What’s it called now? It’s still like uptown or something like that. Uh, Sy Hub is what I use.
Yeah, that’s what I use. The one you’re not supposed to know about?
Yeah. You just copy the DOI number in there.
Oh, okay.
Yeah, I don’t know about any of that.
The nice thing is if you ever… If you ever stumble across using SciHub for some reason and you need an actual citation, you can just take the DOI, go to Google, scholar, and paste it in, and then it will generate the citation for you.
That way you get the full citation.
And then you can do it for real stuff.
Yeah. My grandmother used to tell me a story. Yeah, there was something I just, we’ll cover that after a bit. See, so there’s that. That is pretty interesting. How much of that is, is that wrapped in any cool stuff, IP, you know, proprietary stuff, is that a personal thing or a, okay. That’ll be good, because I haven’t been sued yet, man. I’d rather, I’d like to, I haven’t been threatened by the other guy, but that’s, that was, yeah. And then the other thing I was looking at, Ben had dropped some notes to me about some projects. You have, uh, worked in the past and are probably, I don’t know if they’re, I don’t want to say mothball. Have you played with those lately or? Oh, see, the… What I like about it is generally it’s not an LLL. Yeah.
I did forget to put one LLL and one on there. I had a few that have been gathered, some of them not as much. The LLM one was the just scraping archive and seeing what they have, setting up some automated run job, container stuff to run and scrape that and re-dump it to a database and keep track of it, doing the whole fancy rag approach. Other than that, the non-LLM stuff was like working on rewriting. some libraries for adversarial machine learning. A lot of them are really old in that, like if you’re familiar with art, it’s written to support TensorFlow, PyTorch, or DumpI. Nobody in their right mind is using two of those options for any of these models anymore.
And it’s a legacy thing, so good luck trying to get it to do certain things or scale well.
So I’ve got some rewrites of it to make it. Actually, some of the attacks you generate can be scaled across multiple GPUs, multiple nodes, and you can generate significantly faster. We’ll find around with that.
That is on my GitHub.
I’ve got some projects for automatic deployments and building of projects, but that doesn’t really matter as much.
The, oh, Pokemon RL.
It doesn’t love Pokemon RL.
That one’s about a year old though now.
I haven’t come back to it in a bit because I can never get it to work.
Basically, just battling. of doing random battles for different generations and trying to learn and never does. And no matter what I do, self-fattling, randomization, annealing, no matter what I do to build it up slowly, just never, never, never picks up.
If you limit it to a single generation, does it do any better?
It is limited to a single generation. The problem is I refuse to let it use pre-constructed teams.
And the only people that have been able to get results use pre-constructed teams. And I refuse to do that.
I wanted to be given a random team and do the random battles and perform well.
And to learn what to use in each situation, to look at what it has on the team, what the opponent has. And the information comes out over time.
It’s masked.
And I wanted to do well in that environment.
And without pre-constructed teams, I haven’t seen anybody have good results on.
So maybe I’ll get back to that at some point.
Is it possible?
Probably. But it’s interesting.
Yes.
something a lot of other institutions would be useful for. It’s long horizon planning is the problem. It’s hard. Only 32 steps usually on average.
I’m into to be able to look at all potential Pokemon that are out there.
Yes, I don’t want to model it. I want to do model free.
Okay.
So I don’t let it search the space.
I’ve seen some people get some success searching the space is simulating it out. I refused, I wanted to look at what it currently has and what the current information was available and all the previous ones. Is that the vibe? It’s just like, ah, dude, I’m all about this diglet. Yeah, exactly. I don’t know why.
Yeah, I don’t want to feel this. But if you’ve put an agent that had a shared experience with some other agents that had tried it, is that more like back to what you were talking about that you didn’t want to do? So that just learns some stuff. That’s the way of search. I just learned some stuff about agents that have memory.
The only memory I wanted to code into it was previous state.
So I’m considering it like a complete Markov, I don’t remember Markov diagram, whatever it’s called, where the complete state is observable at all times.
So that was the approach I was going with.
I mean, as I was saying, the average typesets are only like 32 to 100.
I mean, it can stack up 32 to 100 time steps to be fine.
But I’ve dealt with how to optimize the network to take a sort of convolutional layer approach.
where it looks at things relatively, and so I have like subnets for like moves, subnets for set of moves, subnets for Pokemon, subnets for items, and let it kind of take the convolution approach where the locality of information is important so we can learn better.
Nah, none of the badgers never learned from it.
I eventually just flatten it all out and throw it at it and it still doesn’t do anything. And the main reasons why I got frustrated the last time is like I spent more time like, the libraries to make it work. and the tools to make it work had a few issues and like I had to fix memory leaks and their stuff and I couldn’t run it for more than like three or four days without it like running out of memory due to leaks somewhere. I got most of those patch but couldn’t find the other ones and the thing with RL is sometimes it’ll work sometimes it won’t you’ll never have an idea why and sometimes you let it run for like four days and you’ll see absolutely no changes and then on the fifth day it’ll all of a sudden become amazing and I have a suspicion that I just needed to let it run more because I mean if Compute isn’t solving a problem, you’re obviously not using an alpha. So I just couldn’t let it run long enough without it crashing out.
So I was like, I don’t know. I was too lazy to code it up resuming and restarting. So it’s like you have a hope. I have a hope. This thing will do it. I’ll go back to it. It’s never happened. I still get emails about all those people. There’s a hope for all of the people. In summary, do you keep thinking this way? Yeah, I just just state pass. I get that separately. Yeah, for the same kind of thing. Totally get back to it at some point. Totally. Yeah. What else I have?
A few others I can’t think of. One thing I started playing around with and dropped is still a it’s probably only interesting to me and probably only useful to me and may not learn anything.
Um, is back to the, uh, I want to go back to Glacier National Park, but to get a room for like four days, you got to be like on there and, and people, the, so the problem is there is, uh, you only have to pay for the first night up front and you can cancel up to 24 hours in advance, get all of that back. So a lot of people think I might be able to go to Montana for a month. Let me go ahead and reserve that I’ll just cancel later if I need to know no harm about so there’s always rooms coming available through the year But I’m looking for You know a string of like three or four heights in the same room blah blah blah and their website is pretty horrific. There is no API so finding some way to use a Basically a browser driven thing and just to search whatever and if you find it at least Lock it and then whatever send me an email to log in and yeah, because I don’t trust it enough to Give it my credit card number. Um, I don’t know what else that would be useful for I’m sure there’s other things that may already do that but Yeah, so what what interesting aspect of all this stuff is like your last comment You know trust enough to give you credit card number and I have an example of you know, not trusting a GPT-5 So I was playing around with it the other day And I asked it to build out a list of references. So I had a list of references, which is a list of article titles for AI papers. And I had the embedded hyperlink.
So, you know, the hyperlink is hidden.
And I passed it in and I asked it to make the hyperlink explicit and also add in the author name.
I did not give GPT-5 instructions on how I was supposed to do that.
And I didn’t even ask it, you know, tell me how you’re doing this. So you’re looking up on the internet or anything. But anyways, it came up with a list of references and a list of authors. So how many people think it got it entirely right?
How many people think it got it completely wrong? How many people think it was in the middle? I think it got lucky once you guys.
It was about 75% right on the authors and about 75% right on the links. Now, Not entirely, I didn’t do enough investigation of term to see if it could actually read the embedded links or if it was just generating the links all along.
But it’s interesting, you know, and obviously you have to check. So I checked and I did the obvious thing and I clicked through. Right. And I also checked all the authors because, okay, if the links are 25% off, you know, maybe 25% of the authors are off, which is in fact the case.
But it got right for 75% of them. Yeah. We could build a model that all the hyperlinks look real, but actually, Rick rolled the entire… Oh, they all put it actually.
All the hyperlinks.
Super easy to do that.
That’s great. I like that.
They probably already knows the link.
Probably does. All the hyperlinks worked and went to AI papers. They just didn’t go to the right AI papers. Wait, I’m going to get you there.
I don’t know where we’re going.
Eventually.
Yes, that’s one thing that’s interesting though on that note on hallucinations in the GBT-5 system card. They were talking about how they were measuring like hallucinations and whether 03 or GBT-5 was better. And they have a line in there that they said that they found that using 03 as the hallucination checker was more accurate than humans doing it at this point, which I thought was incredibly fascinating. They didn’t really expand on it much after that, but apparently like when humans are reviewing, especially like more complex output from LLMs, they can be very easily fooled in thinking that it’s like incorrect or correct. And so they’ve actually found that 03 is stronger than their human raiders for the same things at the time.
That tells us they don’t hire careful human raiders. But the GBD5 system card was pretty interesting. Yeah. The one thing that, or the two things that caught to me from like a benchmark perspective, They had one benchmark called the open AI proof benchmark and basically it was a set of issues that the open AI teams that had during development that cost at least a day of like downtime. And then they gave it to. GPT-5 to try and solve.
And it was only able to solve, I think, one out of the 50. So it was like 2%.
But the other one I thought was cool was MLE Bench, so machine learning engineer bench. But they were taking like Kaggle competitions, would give it to GPT-5. I believe it was like the highest pro thinking one.
And then if it was able to get a bronze medal in that Kaggle competition, it would have succeeded. And I think it got 9%.
I’m on that metric, which even just like getting one bronze medal is pretty impressive. But that one was pretty cool. Let’s go through that system card.
One of the things I don’t, this was a separate podcast I was listening to.
They were talking about some, it’s not Moore’s law, but it was something similar to that.
Or what they do is they take a task a human can do and then they figure out what’s the longest task a human does. If you give it to an AI, it has a 50-50 shot of getting it right. and that the amount of time that that is doubles about every seven months as far as new models coming out.
And I think the state of the art was somewhere around two and a half hours. Yeah, GC5 was right at two and a half hours. That was a pretty interesting, I think they found it was like 20 by 2050, it’ll be like two million years, something like that. Just because of how it doubles. Of course, if they’re projecting and it’s a perfect, you know, lines, that I mean, yeah, it’s good so far.
They’re pretty reasonable to think it’ll go for a while.
You can tell me for a minute.
Yeah, there’s a there’s a there’s a good figure and I don’t know where it is in the system part, but they show that they show that all the way going back to GPT2 and they showed like all the line of is that in that? Yeah. Okay.
I want to say it’s in the system part. It’s from a company called METR. They’re like a outside consultant. Wait, METR?
Yeah.
Okay.
Yeah.
M-I-T-R-E. No, it’s not. I thought that as well when I like better. We talked about them during the Claw 3.7. They do all the things of like, are AI is going to become able to self-replicate essentially?
When are we hitting the singularity?
So you can do AI research.
They’re getting cited by all of these companies like, hey, we hit this on ME bench, but it’s actually a safety company, which is the fun thing. Yeah, they’re in the same place. Well, last week in AI or this week in AI, that podcast was talking about it last week or whatever. Followed up by there’s apparently been a boatload of money being dropped into all these companies trying to reach that, you know, so whoever can get there first holds the world. As long as robots follow the rules. Well, that’s one of the things they talk about in the, this is, I think this was in the, the cloud system car that looks as well.
About just like the AI self-improvement of like, hey, if it gets to the point where it’s strong enough that it is able to essentially train itself and continue to get better, then you have that just like scale up exponentially.
It’s one of their like three scary categories that they’re tracking.
I was using it as backing info on my kind of the stuff I did at the AI symposium about it’s AI is moving really, really fast.
It’s hard to keep up.
And the whole thing about, well, if a model doubles that thing every seven months, by gosh, yeah, good luck. So from a project standpoint, we have any other, I’m gonna try to write down all of the ideas and luckily I’m recording so we’ve actually, I can go back and not have to remember everything. So I actually have a comment. Yeah, I don’t know if anyone. But Berkeley does a free multi-user agentic AI course.
Okay.
And at the end of it, every semester, they always have a hackathon. If anyone’s interested in that course, there is a hackathon at the end of the course, even though they haven’t announced it till now. That is actually something we, I don’t know that I’ve ever thought of a cohort from our group going through something together. I mean, I need to know all the time. So, that might actually be a fun thing. Yeah, I’m going to take the class if anybody else wants to. Yeah. If you get info on that, I’ll throw it out on the… Yeah, I’ll send you the link. Okay. Do you know what the class is about?
It’s a gender gay act. Basically, it’s a series of Bay Area guest speakers come in and talk about agentiKI and what they’re doing at Meta, OpenAI and all the rest. It’s almost a survey of the different things and approaches they use. They updated every semester, so last semester a lot of it was on like… To what extent can you actually do math with this stuff? To what extent can you do programming? To what extent can you do logic, you know, you know, the seven guys, red and green hats and all that kind of stuff, you know, and then talking about an agent terms, the other agent stuff beyond that.
But that was like three of the big topics that covered last semester. You can also watch all the previous semesters because they’re all up there on the internet. But then this semester will be the latest agent and AI research. Cool.
I was trying to remember the other, wow, something, Rachel Thomas, I’m trying to remember the guy’s name. Shoot. They had some forces, it was, I can’t remember the name of it, it’s where we initially got the, if you look up Rachel Thomas somewhere, there’s actually a really good Matrix Math class associated with it, but they approached it in much more of a do something useful first.
Then learn how that worked and then go a step down and learn do that and then learn how that worked And so you start off by doing something that’s interesting and then you kind of so they don’t jump right into Well, here’s here’s how Adam optimization works.
So, you know, it was like I I can’t really I think but cool on or something like now Was it fast AI? That’s it.
Jeremy Howard.
Not Rachel.
Was it Rachel Thomas?
She popped up. She’s like something tied to fast AI. But I think Jeremy Howard’s the one who runs.
Yeah, he’s been working for a while. And then we did a couple of things where we actually went through. They had like eight lessons of really approachable AI stuff all built into Jupyter notebook type things that you could actually, you didn’t have to have a lot of things. to go do it.
Weirdest thing he actually, his convolutional network theme walking through and how to do that, he built it in Excel using macros and steps through and it’s like, oh, that’s what it’s, you know, it forces you to slow down enough to go, oh, that kind of makes sense. You can do a lot of neat things in Excel. Yeah. That used to be my question between do I need a deep network or do I need just a machine learning classifier thing?
Can you open the file in Excel?
Yeah. If so, regression is probably your friend.
A couple other potential resources.
I don’t have any specific product or projects from here, but things just attract email contest.com. I don’t know if you can see some of those.
Yeah.
Um, so they, they essentially are an aggregator for Kaggle’s and the, all of these like okay data ones.
So it’s just like a single place to go to.
Um, and then challenge.gov.
Um, those are all generally like federal, some type two, some type of government. So like really they have one that’s like an explainable AI for Medicare fraud. But those come out semi-frequently, but I try to check them pretty often. Yeah, but other two places to eventually look for other projects going forward too. Okay, DevPost is probably one of the biggest hackathon sites.
So they get a lot of wide open hackathons.
Okay, done. That’d be pretty cool. We are at seven o’clock. I know we got started a little late. I’m going to skip the last part that I actually took the whole transcription service that we initially did on Fargate and then pulled off of that and went to Lambda because it was super cheaper. I’ve got revamped to run and run Cloud IO as an endpoint. The startup time is like in like seven seconds or something if I haven’t get it yet, whereas Lambda and Fargate were minutes.
So this thing basically was ripping through an hour’s worth of audio to text in about the same time it took to start up because I’m now on GPUs and I can like go wide with like, oh my gosh, this is so yeah.
So last I was running it on a 16 gig GPU and it was around seven to eight cents worth of money to transcribe a little over an hour worth of it. video.
So it was kind of neat.
See I’m gonna basically ditch a lot of the AWS stuff I’ve done because they’re costing more money than what you would expect and it’s all on the fringe stuff like oh you need an internet gateway or oh you wanted a load balancer or and by the way you can’t run without those. I’ll tell you when you get started and you know It’s, it’s a best infrastructure as a service to the nth degree. It was fun to learn. Um, I learned it. Yeah.
Now I would definitely go climb that or whatever and say, can’t you?
Cause I did the whole thing with basically I told client, Hey, I have this thing. It’s already in a container. Can you put it in a container that works with robot and who get up and also. build the tests out for me. And it did. And I deployed it. And there’s a rest-in point that you can just, if I published it, anybody could just go ahead and it would transcribe their stuff for me. I think that’s it. I will probably do something on the Discord trying to come up with topics that we want to talk about coming up because I’m a little dry. at the moment and see what kind of scheduled people want to handle what folks are interested in. I do have the one that I’ve got kind of a time you would want if you were talking about covering your approach to hackathons, not necessarily like, here’s a hackathon that we, you know, something like that. Yeah, although actually I think my approach to hackathons would be pretty short. We should have so much time talking about hackathons. Okay, you probably like I can probably do it in five minutes right now Let’s queue that up and we’ll probably do a one of the things we haven’t done in a while was like lightning talks. Yeah, we’ll do it as part of a lightning talk. You know five minutes go up there, you know and then I’ve got Tony It comes occasionally, does a lot of AR, VR stuff. They’ve actually got an IEEE spec out for, I can’t remember the name of it, but it’s actually encoding location as part of like a URL kind of a thing. And looking at how that applies to AI as well as robotic.
I mean, it should be somewhat of a game changer for some of that stuff, I think.
So that was a thing. and maybe get some cool AR or VR stuff. That’s another interesting thing that I don’t know anything about. Well, the other thing you could do, if you want to get more speakers, is spam the people who spoke at the conference last January.
We could do that.
Most of them are in town. Yeah. Some of them, I don’t want to say she owes me, but yeah, she didn’t throw my name out to something else. I was like, whoa, hold on. Just because the follow on to that was a cool thing that she had seen where they, uh, where they’ve actually done like a biological version of a neural net. So you basically growing a neural net and like cells and stuff because they’re much more energy efficient. Yeah. With computers. Yeah. I don’t remember. I don’t really think about that. That’s fine. It would be cool if we were able to bring in some people who, where it’s not as much AI focused, but it’s like AI in some new domain. Like I think those would be some interesting speakers. I know we’ve got a lot kind of around Huntsville that would be pretty interesting to hear from. Yeah. Yeah. If you got thoughts on who?
Yeah.
I’ll try and go in there. Hey, we got a group that people like to come listen to people talk about us. Yeah. You tend to know some things about Aesta. I’ll try to think of some people.
Let’s do it. All right. And I did get a cold call from the Pryon company that we stopped by at SMD symposium.
So I’ve only got one that’s like trying to get my phone number off of the things, actually put the real one on to see who would do it.
So the others aren’t as good marketing. But yeah, I guess that’s it for tonight. Thanks for stopping by.