Transcription provided by Huntsville AI Transcribe
We’ll jump back and forth between a couple of things. So the paper, link to the paper. Of course, the other thing, if you’re new with us, we have a GitHub account at HSV-AI.
If you look in presentations, you’ll find stuff going back to 2018.
So this thing we’re covering tonight is already posted. So if you lose track or you want to go back to it, you can find it there.
Link to the paper, I really wish they would have named it something different because the only thing they cover is co-pilot. But then they call that generative AI and then mention, well, there’s some things that people don’t use it for. And I’m like, well, I know it’s co-pilot. It’s not mid-journey. It’s not all these other things that are also useful. So anyway.
But other than that, I do appreciate the work they put into this, because this is a topic that a lot of companies that do a lot of AI stuff are avoiding quite heavily at the moment. They still kind of didn’t go right into here’s the jobs that could be done by AI. They actually just kind of mentioned the overlap between what AI can do and what people are trying to do, you know, things like that. I pulled a couple of snippets out of the intro, which kind of set the stage. So given the rapid adoption of generative AI and its potential to impact a wide range of tasks, understanding the effects, we could take a step toward the goal of analyzing work activities people do, how successfully and broadly those activities are done. and combine that with data on what occupations do those activities. We’ll kind of break that down a little bit moving forward.
And then at the end of their intro, it says they characterize the types of work performed most successfully.
And then one of the things they do is drop in to… Like wage information and say okay, so these are the jobs that are impacted by AI Where does that land on like the wage scale or higher paying jobs impacted more than lower paying jobs or you know, is there anything to get from that?
they also roll in the Education side of okay, how if you are masters level education Is it more likely to impact you in your kinds of jobs versus, you know, maybe GED or high school or, you know, something like that? And that’s, that was a pretty interesting thing. Some of it winds up to be, well, not really. But this is the main point of the paper is their AI applicability score.
So they go through Kind of how they set this up what inputs are used where it came from You know that kind of thing So that was that’s pretty much the interesting thing And of course the data they get out of it will hit that too But a lot of it you could guess and It’s some people have there was actually part of their paper goes through a comparison to some other paper a while back that actually tried to predict where AI would impact jobs. We’re not really going to cover that. It was kind of interesting. The paper overlaps quite heavily for their predictions, which may be why they only picked one paper to compare with. There’s a lot of papers that have predictions on where AI is, and they just happen to pick the one that happens to match up.
Go figure.
So we’re gonna, it doesn’t, I didn’t think it added a whole lot to what they were trying to do. And I almost think it attracts a little bit from the main point.
The other thing, if anybody’s reading ahead, they use the term coverage and they use the term scope.
And in a table down at the very end, they actually have a column for coverage and a column for scope.
And I don’t really know the difference between the two.
And I’ve read this thing like three.
here four times.
I still don’t quite get that part.
So if anything jumps out at you, let me know because it’s bugging me. Some background info. It only covers Copilot, which I mean, it’s nice of them.
Oh, let me see.
Yeah, they could have definitely picked a paper that was, you know, opposite of what they were thinking. But then they’d have to explain why the other paper was wrong, though. So it only covers co-pilot. It is specific to US users. So they didn’t use international data at all, even though other countries use co-pilot. The reason for that, and we’ll drop down in a minute, well actually right now, two of the data sets they pull into this.
One of them is the standard occupational classifications from the Bureau of Labor and Statistics. And then that’s also where they get data for the number of people in each occupation. And then that’s where their wage data is coming from.
So if somebody’s a doctor, they just got some kind of index into that.
We’ll hit some of this pretty quick.
I think I already had some of these open.
Let me close this.
If you’ve ever taken a. Well, actually, that’s the aptitude test that’s coming next. So Bureau Labor Statistics, they’ve got this. Gosh, what was it called?
S O C wrote it down. So I’d remember standard occupational classification. So being statistical folks, they have broken down a hierarchy of. Every job they can think of in the United States and put numbers associated with them.
And then put them into categories and groupings, buckets, all that kind of fun stuff. And then you’ve got a major group, a minor group, detailed occupation, description, things like that. I mean, it’s a pretty interesting, wide-ranging thing.
So that’s kind of where this is coming up.
And then the other one, which is the actual definitions, well, I didn’t mean to click in the same place. Let me go back and move that over. So this is actually where you’re, it’s a lot longer, but your actual description of an occupation that’s much more low level. Not quite, not quite at the level you would put on a job posting yet, but still. pretty low level about, you know, what this job actually is.
So that is one dataset they used.
The other is this, I’m not sure if you’re supposed to call it one net or on it, or they put a star in the middle of it. So this thing is also, I believe, sponsored by government somehow. This is a basic taxonomy of occupations and actually we’ll let me jump to the paper where I have page four.
The paper actually does a better job of explaining what that is than the websites I could find on it. So what you have is an occupation which is Economist. That occupation is broken down into the tasks that an economist does.
So specific, a very specific item.
And I don’t think I have my notebook.
In this OSTAR net database, tasks, there’s like 18,000 of them listed total.
One of the things they note in the paper is that tasks are fairly They’re really, really kind of discreet things and they tend to overlap more than what you would find useful.
I’m also trying to keep the chat up and going. Oh yeah. So there’s a lot of tasks more than what you would find useful for this paper.
And then they they map tasks to I can’t remember what D something work activities I Think they actually line this up in here And I wish I had brought my notebook upstairs Detailed work activities I guess I could just show this part I’m also trying to figure out how to minimize this There we go. So they’ve got detailed work activities, and then each DWA belongs to an intermediate work activity, which in turn goes to a generalized work activity. And then all of that kind of, so you’ve got generalized work activities on one side, occupations on the other side. And then what they found is the IWAs don’t have quite as much overlap and are pretty good at matching up with the type of thing they’re doing in a chat with Copilot.
Let me see if there was anything else.
Yeah, so they if you look at kind of the number I’m doing this from memory so the numbers are probably not exact They’ve got like 29 occupations when you get down into the IWA or intermediate work activities there were I think two two to three hundred of these So and of course you can have one IWA can map to multiple occupations if you start walking backward The other thing that I think they mentioned in here, the breakdown of occupations into tasks. There’s actually some kind of waiting. Actually, I think I wrote it down in here. Oh, yeah.
The tasks that are associated with an occupation, they have a relevance and importance metrics that are used in the paper, which was.
It’s pretty good.
You could you could imagine that because as a software developer one of the tasks I may do is You know check email, but that is not the core thing that makes me a software developer You know lots of people check email So it kind of gets you into the more important parts of the task Or more important tasks for the occupation and I believe they flowed that all the way through to come up with some kind of a waiting for IWA’s back to occupation. And as far as I know, the coverage thing that they talk about is how many of these tasks associated with an occupation are actually covered by, let’s say I’m in co-pilot asking it to Let’s say I’ve got a CSV file and it’s got one column that’s a bunch of dates all smashed together and what I want it to do is break that down into a bunch of different places.
If I’m in a chat with Copilot for that, there’s actually an LLM they used in the paper to try to come up with a number for how many of the tasks associated with my occupation are actually covered by that particular Copilot interaction.
So that’s as far as I know that’s where coverage comes in and then they have this other thing called scope which I’m still not quite sure and We’ll get there in a minute So far that’s some background if oh also the education requirements That they have were pulled out of this data set as well So from the BLS data They’ve got how many people are in each occupation and the wages from the OSTAR net data.
They get a breakdown from occupation to tasks over to IWA’s and then a matching up of of education requirements. They go through in the paper a little bit and talk about the difference between the O net.
You know occupations and then the standard because these don’t match exactly And so they actually laid out the mapping they did or the mapping they used I think there’s one that’s actually provided Between the two occupant, you know the between the two sets of occupations Any questions on that background data so far let me go check comments if I can figure out how to what button click So can I take a stab at trying to define the scope and the coverage? Yes, yes. I think I got it.
So I think so coverage is to say I’ve got six tasks and the LLM could do half of them.
That is able to do something on half of them.
You know, whatever percentage of that is, it would have 50% coverage and scope.
is like, so for one of those tasks, it’s like I have an Excel spreadsheet and my job is to classify four of the columns.
The LLM can classify three of those columns consistently and I trust it, but there’s one that I’m not, I always have to do that one.
It has 75% scope of that task.
Okay.
And that seems to be kind of how they’re doing it.
All right. Lloyd till we get to the end and I will try I think that sunk in a little bit.
I’m still a little bit fuzzy, but let me I will try to repeat or explain back to you what you just said and we’ll see if I got it or not Because I think I’m close Any other questions so far All right, we will hop into user goals versus AI actions This one, again, the paper explains it better than I can. Let me minimize that again and then the chat thing. So what they’re looking at is some kind of like I’m asking AI to do something, but that’s not actually my goal. So my goal might be to… In other words, I may be trying to build a presentation that I can show a Huntsville AI group.
So I’m trying to build a presentation.
What I may ask Jim and I like this morning is, hey, here’s an outline, provide a summary of X versus, you know, something.
So the AI action is summarize data or provide data.
But my goal is make a presentation, which are two different things.
but I use the action in order to meet my goal. And so they go through that a good bit.
And a lot of their scoring is kind of tied in with this user goals versus AI actions.
The things that I’m mostly interested in are the parts where the goal is the actual action.
So, when we’re talking about, they kind of walk into the difference between augmentation and automation.
So, if it’s an action that I use to meet my goal, then AI is augmenting my occupation.
As in, it can’t do what I do, but it helps me do it a lot faster, a lot better, cheaper, whatever.
The other side is, if the goal and the action really align, then the thing that I’m doing is actually could be done by AI itself. You know, if my job was to summarize data out of a paper into some other thing, and AI is actually doing exactly that, well, then there’s a highly likelihood of just pure automation.
And so at the point of they don’t need me to do this, you know, why do you need me to go ask AI to do the thing?
So that’s that was kind of interesting. Um one of the ones they used uh somewhere in here uh customer services uh asymmetric resolving technical support and height wasn’t it there was one uh that’s definitely oh one of them is definitely uh if like to to show the such a wide range or a different, where the goal is actually way different than what the AI is doing, is kind of what we walked through with Replet the other day, where Andrew was putting something in about, well, I want to come up with an exercise plan. So the user wants to exercise better. The AI can’t actually exercise. So the task from the AI is actually coming up with a plan or something, but it’s actually the person, there’s no way to AI is going to do that. So that was, and they spent a lot of time between actions and goals and whether it’s one thing or another. So that was pretty cool. So then the other thing I wanted to cover, so they’ve got three different ways.
So, and I wish I had done a better job of drawing this out.
So they talked about that. Let me drop down to this real quick. We’ll circle back. In general, as we’re working, they cover a lot of stuff. Intermediate. So they throw a lot of data in and towards the bottom is where they actually get into how they did some of the pieces.
Now, predictions, I might actually drop down into the appendix part.
All right, this is where they actually talked about joining up, you know, the Onet with the BLS aggregating occupation.
We got that. There we go.
So.
One of the things they did was develop a way to take a conversation with co-pilot and they had to classify it into, okay, what kind of, what is the goal and what is the action that the AI is taking? Otherwise, they don’t really have a way to know that. I found it a little odd that they’re not using a Microsoft model to do that, but I guess that. It kind of all goes together, doesn’t it? So they go through working through to figure out kind of which IWA, this particular or set of IWA’s, this particular conversation covers. And on this one, I believe they actually had a set. with actually a human going through and, you know, grading how it was doing.
And then they go in through some, there’s some math in here to work through that.
I liked that they dropped their prop in the paper that they used to actually do the classification.
So this was pretty interesting. Hold on, let me check chat again. Oh yeah, they, they don’t, I don’t think they checked to see if these people were working or not. For instance, I, getting ready for this, I use my work-based connection to ChatGPT to do a presentation on a paper review.
Doing paper reviews is not part of my occupation.
So if they were later trying to check ChatGPT, they would think that I’m trying to, you know.
ChatGPT can replace AI researchers.
I guess so.
They would look at, well, one of the other things they would see later on would fail.
So that is a pretty big hole.
Hold on, let me, because I’m actually trying to make a list of the things that I thought were either weird or not quite there.
Kind of like using the term generative AI when they’re only talking about co-pilot. So assumes using it for occupation.
and not something I’m fairly certain this week somebody used co-pilot to help them figure out who to play in their fantasy football league. And I’m pretty sure that was not related to their occupation. So, but I do like that they left the prompts about how they did the classification because I’ve got thoughts towards the end on some of this. And they, they went pretty far. I mean, this, I guess it’s a decent, decent length prompt.
I’m not, I’m not a master prompter person.
So if any of you all have a ton of experience, you know, looking at this.
So this one was the trying to classify the user prompt.
The other one was trying to classify what the bot is doing. There was a, so one model is trying to figure out, so they’re using an LLM approach when they have a conversation to figure out whether or not it, which IWA it aligned with. to give them, I don’t know if they’re pulling scope and coverage and all of that out of that same model or if they got something else, I’ll probably figure out when I keep scrolling.
They had a different one looking at completion, meaning they were trying to figure out, how do I know if the AI bot in this case actually did the thing the user was asking it to do?
They’ve got some data implicit because Copilot has a thumbs up thing that, yeah, you can tell it’s doing a good job.
But they also, in addition to that, actually looked at the conversation and then threw it to another model to try to figure out if it actually did the work or not.
And so here was the prompt for that one.
There was another piece, I think, further down. Oh, they give you some info on the models they used. I think that was it for that one.
So somewhere up here, they go through, so they’re getting scores for which IWA it was, how much of the tasks were actually covered by the conversation, whether it was successful or not, and then using that to build out their actual, let me see where we got that from, using that to build out their actual applicability score.
So that’s back up to, now we’re back up to here.
So that’s basically, and I still wish I had a diagram of where the different data points are coming from, where the actual models are, and how they pull that together.
They spend a little bit of time talking about their threshold.
They used for, for whether to include something or not. Pretty much trying to keep things off of zero or off of one.
And then they start talking a little bit.
There’s actually their algorithm they use for that.
So if you really love math, there you go.
It’s got math terms.
And this kind of shows one of the things they were trying to stay away from, they weren’t trying to say, this job is totally overlapping with AI stuff. They’re looking more of a relative approach. So from one tack, one occupation versus another, where is the biggest impact?
You could probably draw further conclusions.
you know, from the data set or from the output of this to make some other, you know, estimations, but that’s kind of what they were going with.
This is really interesting, this chart, because they’re talking about the threshold of what they use to consider covered.
And so they chose 0.05%, which is just arbitrary.
I know.
It seems like.
It seems that their arbitrary value proves that they have interesting data, though, because they’re showing a linear curve, where if you just bump that threshold three to the left, then 0% of jobs are covered or are in danger.
And if you pop it four to the right, then all the jobs are covered.
So that’s kind of interesting.
It gives you some idea of the spread.
And that’s also, I feel like they tried really hard to not be alarmist. They’re putting a lot of effort in here to make sure somebody doesn’t pick this up and say, Microsoft says everybody’s job is gonna be done by AI, specifically co-pilot. That was fairly interesting. The results.
So some of them are pretty much obvious.
Oh, let me see.
Oh, yeah.
So the things that AI is good for is getting information.
Okay.
Communicating with people outside of an organization.
And then the, before I get on the colors here, the orange is a user.
goal and the blue is the AI action.
So when people were just trying to get information, there’s a lot of coverage when it was part of, you know, the AI action to get the information was slightly lower.
The actual, and this is, this is one of the interesting things. The, let me make sure I’ve got the tick marks right.
Yes.
So blending the OSTAR net and the BLS information.
So they were able to take getting information, which is kind of one of the IWAs, figure out how much coverage it is, but then also map that backward using the BLS dataset and their SOCs to figure out how many occupations include that and also how many people work in those occupations. So that’s where this bar comes from. And you’ll see it move up and down or left and right based on the number of people that do this task as part of their occupation, which I thought was pretty interesting.
Working directly with the public. That’s where the tasks actually came in.
higher, the AI task was actually higher than the user goal there.
So I can imagine people are trying to do this task as part of some other actual goal. Assisting and caring for others. This one was one they actually call out somewhere else. So if you are a caregiver at a hospital, you may use a co-pilot to help you learn some new techniques or do something, you know, or compiler report or, you know, hey, what is the, what are the regulations related to, you know, a patient with this age, something like that. But that AI can’t actually go do the thing that you’re wanting to do. So this is one place where they’re definitely separated. Decisions and solving problems.
You get down here where it’s really not useful and this is one where it was pretty clear that just using this for generative AI somewhere in here you’ve got operating vehicles.
um, mechanized devices or equipment. Um, I’m pretty sure if you walked over to Mazda Toyota, this line would not be zero, you know, but that’s, they’re not using copilot to do that. They’re using some other things.
So, um, I do think it’s, it’s good for what they’re trying to do with it, but I think it would be, well, never mind. We’ll skip to the discussion part on that one. I’ve got, you know, some thoughts on that. Um, Further down, they actually get into kind of some of the differences between the user goals and the AI actions. So like one of the main ones for user goals was, hey, I need kind of like what I did when I started the paper.
find all the references in this paper, you know, find all the documents referenced in this paper, give me a link, a list of links that I can click through to read more, you know.
But then the actual tasking, I’m trying to think of which one of these tasks would have been, that may have been assist others to access resources, because that’s kind of what I’ve asked it. gather info from various sources. That would have been one of it, one of them. So that was kind of interesting, but I’m still, it’s kind of neat that they broke the action and the goal separate, you know, from each other. I’m not quite sure what to do with it moving on other than just knowing that, well, if the if the AI is not doing the thing, but it’s helping you, maybe that’s the main thing I need to take away from it.
But intermediate work activities, they cover the ones that are more often, and I guess this is kind of the point, assisted versus performed by.
So the way I would roll this out, And some of this could just be due to trust in these models so far.
So right now we use AI to go help us find goods and services, but we don’t actually let it buy them, buy them for us. Or it can advise us on financial transactions, but it’s still us sending the check, you know? Some of these you could see actually moving in a different direction at some point, but even there’s several on here. like prepare foods or beverages. I look up recipes all the time on how to do things. It’s getting really good at taking a shot, you know, an image of a bunch of stuff and figuring out what ingredients those are and then going and finding recipes that you could make with a set of ingredients, but it can’t actually cook the food.
So that was pretty interesting. And then most often performed training others. a lot of training. We got coach, train, train, train, advise, teach, teach. I think there’s a definite, definite way to get, thing it’s doing there.
And I’m wondering some of that, if you got thoughts on it.
Some of the ways these things are overlapping might be very specific to the way that copilot is put into products and intended to be used.
You know, I’m not, I don’t know that.
I don’t use it a ton. But could you see like one kind of a model kind of leading people towards or away from specific, you know, actions? Like if they had this in Codex instead, would it be a different set of things?
Talking to Codex about teaching me things, generally. Right.
Again, we’ll hit a question at the very end. So they get they also spend a bit of time on and this is part of something we we hit already on trying to figure out Based on the thing that happened Did it actually do the thing and was the user happy with it? We’re running into very similar things Some of the pieces of work I’m doing with some you know code development assistance agents things like that where really the I don’t have a good way to know how much of the code coming out of the organization is actually partly generated or partly from that. It’s under the user account who uses the assistance to create the code. But then when the code goes in, well, there’s no way to backtrack that and say how much of that was actually used. I know some of there are, I think you could look and see from the AI part itself.
There’s some things as far as, you know, rejected tasks or canceled conversations, if you will. But that mostly tells me how useful the AI is.
It doesn’t actually give me Like real metrics towards the end where I could say hey this code was 28% you know assisted or whatever That might be an interesting thing later So on the satisfaction and completion and this was another one where it was pretty evident that just focusing on co-pilot kind of kind of leaves you lacking sometimes. So apparently it’s really good at research, health care issues, doing some interpretation, assisting in purchase of goods and services, you know, things like that, things that it’s bad at. don’t use it to make visual designs or displays. And it’s kind of interesting. They’re like, well, apparently co-pilot isn’t good of that.
And I think everybody that’s used co-pilot to try to do that went, well, yeah. We might like it better if you could figure out how to do that. Yeah, Josh, what does an OTEL, what does that mean? It’s like open telemetry. Oh, okay. Got it.
Yeah, there’s one of the tools we use that actually claims to be able to identify AI generated code.
That’s bullshit. I know. There’s a price with it.
If I can trace it, like it came out of this system, you can do it, but by checking it is nonsense.
It is not cheap. Anyway, I made that claim I didn’t I might have The analogy I used was the thing that you get warned about for when you get a virus on your computer And it says hey, I just detected a virus pay me money and I’ll remove the virus and it’s like you really trust the thing that put it there to start with That’s how everyone those tools looks to me That’s AI writing let us help you make it better right um With AI, you know So positive feedback let’s keep rolling I Think it was page 12 that actually is the chart that everybody’s wanting to look for And this is where the coverage and scope were in the same Completions fairly fairly clear And I believe this is rated on the over ordered by the overall score because the scope goes up and down some completion goes up and down some coverage up and down some so Interpreters and translators Historians This one it seems a little weird because passenger attendance seems to be a physical thing But maybe there’s some overlap that I’m missing Cells reps, I would much rather go ask an AI about something than talk to a person if I’m trying to figure out the difference between two models of televisions, you know, I could see that writers and authors definitely customer service This is where bought city. I believe is generally dropping in Tool programmers ticket agents Educators I didn’t know that farm and home management educator was a thing though Apparently there’s only 8,000 of them in the country Hosts and hostesses you get down towards the bottom of things One or if there’s an actual yeah, keep going the bottom 40 um Phlebotomists the people that draw blood. Yeah, I could see that and bombers, not great at dishwashing. The best one at the bottom was the dredge operator. Of course, there’s only 940 in the country, but if you are a dredge operator, your job is fairly safe, I think, until your job is safe from co-pilot. And this was kind of an interesting one. I don’t remember the name of this particular kind of chart.
But you’ve got kind of the thing that it’s doing and then the actual occupation over here.
And some of these they kind of ordered in weird ways.
But then again, the size of the bar is related to how many people actually do this job.
for like interpreters and translators.
The task was, where was that?
Ah, I missed it.
Somewhere in here.
Oh, interpret language.
So provide information to customers, goes into passenger, okay, I guess that’s where that is, customer service, telephone operators, ticket agents, you know, things like that. Let’s see if there’s anything other than here.
Oh, and here was the back to the BLS, you know, SOC grouping.
Their groups of occupations, how much is, you know, the same scores applied to those.
And this is kind of where you get into computer and mathematical being quite highly covered.
And then just below sales.
And then at the bottom, health care support. Actually, I guess this is still the… I wonder if that’s all the major groups. I don’t remember how many there were.
I think there’s like 20 something. So maybe that’s it.
And then the other thing they did, they took the list and then they figured out, oh, okay, this is the one where they went and found the paper that did the predictions and said, well, hey, they were pretty good at predicting.
And I’m like, well, they predicted that customer service reps would be pretty well. Anyway. We’ll skip the prediction part dropping down into Where they looked at it as far as wages go On this line basically I’m looking to see For applicability score and then the wages is this does it go up a lot as you get higher wages and it’s it’s not flat, but it’s not It’s not like alarmingly Oh my gosh, there’s no need to get a higher education, you know. And then the, as far as the, no, actually this was wages. I’m sorry, I jumped ahead. As far as the education goes, if the line would, you know, I mean, if this bar was somewhere way higher for this, you know, instead of being kind of in the middle of your spread here.
But apparently masters or higher has a little bit lower applicability score than a bachelor’s As far as copilot goes Let me see they do a discussion part let me jump back make sure I didn’t move anything Yep discussion Well before we hit discussion any Any other comments? On that so far Riven chart. Okay, that’s it All right, one of the things that I was thinking about is that If you could tighten up this framework a little bit, I really like the way they they found the Ostar net and the BLS and Along with the wage data the education requirements the weighted tasks, you know things like that that seems to be a very a very pretty solid starting point.
And then there, the way they identified kind of the scope, the coverage, the, you know, the way that the kind of the framework they put in, I think is pretty useful.
And what I would really like to see is that same framework. Now let’s, let’s pick up about every AI thing we can find, run it through that framework.
and then do a compilation or some way to tie them all together. You know, if all of a sudden I’ve got, well, let’s throw mid journey in there.
Let’s let’s throw self driving cars, Waymo, you know, forget generative, just go AI overall.
Of course, I don’t know who’s got the money to write that paper, but that would be pretty interesting for me.
And then the other thing they copy on the.
what they talk about a bit is that right now they are, the data they’re looking at pretty much tells them, well, they’re calling the upstream, you know, measurements, things like that.
They could tell you how much overlap there is and whatnot, but what businesses actually decide to do with that downstream is not, really, you can’t predict it that well.
One of the things that they mentioned, let’s see if I can find it. There it is.
So one of the things they mentioned, and I didn’t even think of this, bank tellers and ATMs. So apparently when ATMs came out everybody was thinking that you know Well, we don’t need bank tellers anymore where bank tellers are gonna go away What actually happened was that the bank tellers job changed because they’re not responsible for just counting out money for people that that’s all they want so you wound up seeing more actual You know banks open up or branches because they they could use less personnel because they, you know, but in turn, you wind up with more more branches. Overall, you wound up with actually more tellers, which I didn’t, you know, didn’t quite think of that. That was that was pretty interesting. Yeah, I agree. I thought it was generative overall, but then got into it and it’s only, you know, they could have titled it much, much better.
But yeah, um They just kind of you know, I think I’ve been fine with all this stuff like the box and whiskers and their scatter bots and all like what are you talking about?
You know, it’s they don’t have the data for all of these things like what you’re saying about You know the the fun one day or whatever Mazda plant. It’s just you know, they’re trying to make these assertions It’s just don’t don’t make the assertion. You know, you didn’t have anything to say don’t say it right Um, it’s like, hey, we looked and we expected something and we did. I think what they’re saying on this one was they expected to see some kind of, uh, you know, something with wages and they didn’t, they didn’t see it. Um, but that was, I think that was what they were, but then again, you could have just said it. I don’t know. Um, uh, but so far, any other questions, um, comments, cries of heresy?
No, this this was interesting. I have some of the same problems with the paper but so many of these that say generative AI broad stroke and you look at it, you know, it’s like and we only tested GPT-3 from two and a half years ago. So it didn’t surprise me too much. Right. Of course, if you followed one of the other topics I’ve been working on, I don’t think it’s in here.
I keep seeing job postings for AI engineers and courses for AI engineering and things like that. So far what I found is how to write a prompt. But interesting thing, I don’t know that I would qualify that as engineering any more than the tire engineers store where you can get tires. Anyway. That’s just me on a sub box.
Well, cool.
The interesting thing for me was just the way they put the framework together and tying those two data sets was probably the most interesting thing I got out of the whole paper. A lot of their actual outcomes are things that you would probably expect having used it. But it’s also I think it’s also something that you could use The the question I’ve got is this if I’m trying to and because this is something I do a lot in Let’s say January of next year. I’m doing another couple of talks for learning quest and I’m gonna have mostly retired folks in a room shotgunning questions at me about AI and can I trust it is it gonna take you know, what what all blah blah blah and It seems like having at least some kind of, okay, we know for this, if you’d have to present it very specifically and say, okay, for this kind of job here, this company did a study based on their product and found that here’s kind of how that worked with their stuff. So I don’t know, it may be an interesting like tidbit, but I don’t know that I would make any kind of assertions off of it.
But it’s interesting.
Yeah, it might lead to more confusion, but.
Everything is going to lead to more confusion. The. Yeah, anyway, I need to fancy confusion.
I just wanted to throw in there. I think I think it’s interesting, you know, I’m in the. finance industry.
I’ve been in finance and banking for 27 years or so, but in the financial advisory space, the advisors, not just in our firm, but in general, always are chasing bright shiny objects, but everyone, of course, now is chasing AI and trying to integrate it into their practice.
And of course, they want to use search engines. or not search engines, sorry, the different AI engines. And it’s been my goal in life to try to slow that thing down knowing that the horse is way out of the barn. And it’s just a matter of trying to keep it, not on the rails, but at least somewhere in realm of where the tracks are at. And working with our legal and compliance, trying to help them understand. And of course, They think they understand but they don’t and really it falls down to they they don’t understand the difference between co-pilot and chat gbt and you know me talking about getting co-pilot subscriptions versus You know advisors using you know public models and the difference between the data and whatever But I think it’s interesting as they start, you know, I wish that this was a little more relevant outside of co-pilot, but yeah It’s really interesting if we start seeing the use cases and You know right now the the big thing we’re using it for is is note-taking It’s been actually very good for that. But we’re using AI models that have been built specifically for AI note-taking for advisors. But as we continue to go on, what other kind of use cases are we seeing? Again, this is so far different than what most people in Huntsville are talking about, since it’s mostly defense or all defense. But the investment team wants to use this to build proposals. And we’re talking about proposals that are either for institutional investing or what have you where they wanted to point at a folder full of PowerPoints and other data and have it build a proposal for them. And I think everyone on this meeting right now knows that that’s a wonderful dream.
But how could it ever possibly actually happen accurately with the amount of hallucination we’re seeing?
But I’m definitely interested to see how this continues to develop, and especially as you guys are doing these white papers, because it gives me more to think about and it actually gives me more to talk about internally, especially the way you boil these down.
It’s giving me better bites that I can take away and use for internal discussions and maybe drive it in a more meaningful direction from a business use case. Yeah. Oh, the other thing I like about it is it’s actually The name at the top of the paper is Microsoft. That carries at least some of them. You could say, hey, Microsoft Research came out with this, and they’re like, oh, okay, so we can at least some of the parts, even though there are flaws with it and all, at least the way they did it is, I’m pretty sure that what they say here is actually what they did, which is good. Another thought I had while you were talking I don’t know where the thought came from but it was kind of interesting to me Not all my thoughts are but let’s see I Think it would be pretty interesting to take this actually Maybe maybe this is the right place to look where you’ve got the score and everything I wouldn’t mind seeing because each one of these occupations has, especially when you get down into brokerage clerks or mathematicians, technical writers, some of these actual occupations have some kind of regulations or guidelines.
About how their job has to be done and responsibility and whatnot. I’d be interested to see the overlap between What how many of these actual occupations have some kind of guideline like overall I don’t care if it’s just a Coming from a Shoot what do you call a group of businesses all in one kind of an area consortium not a consortium Shoot Congratulations. A guild or a association of builders or something, you know, I’m pretty sure there’s, I’m pretty sure there’s some kind of a trade organization that interpreters may belong to. You know, I can, in our case, you know, we’re not, we’re regulated by the SEC and by Petra, right? You know, in banking, you know, maybe a state banking regulator or maybe the, the OCC.
So, you know, we all, You know, in banking, we’re under FDIC. You can almost consider that our almost your trade organization.
That’s where your insurance is coming from.
But, you know, with with on the SEC side, we’re, you know, I’ve been regulated for almost my entire career, but We’re still hoping for regulations on this and they usually the way these things work and I don’t know how it works in the defense world necessarily But is that you hope you get it right because they will surely come in and tell you how you did it wrong without providing you concrete guidance on how you should have done it in the first place Yeah, what we get is they come in and tells we can’t do that and it’s already a year out. You know, sorry we did that like It’s been a minute man I’d be interested to see a heat map of occupations versus which one of these have… Right now we look at the scores almost like AI penetration into these occupations. You could do something else as far as what kind of risk is there if the AI goes wacky that somebody’s going to get hurt or fired or… You know what I mean?
That’d be an interesting thing for me.
Yeah, actually, that would be very interesting. Obviously, I know from my perspective, imagine we give you bad financial advice or we’re using it for client prep, asking, you know, what kind of questions would Jay Langley ask based on whatever your concerns are? And then we give you some advice that you were not asking for and it was unsolicited and now you’re like, well, these guys are idiots and I need to look for your advisor. Yeah. Yeah, don’t do that. Yeah. I mean, we have different layers of risk we look at that are different than yours, but it’s reputation risk, transactional risk, credit risk all the way down.
And when you start talking about some of these, you’re really heavily into the reputation risk. Oh, yeah. We’ll suffer by improper use of AI or just an AI that’s not built to or used.
being used for what, or you’re using it for what you shouldn’t be using it for. Right. Yeah, to tie into something both of you said, you know, you mentioned finance and wanting to jump on the bandwagon and Jay is sort of like you’re saying with how do you define AI engineer, you know, with When so many people say AI, it refers to these, you know, the co-pilot, the chat GPT. Okay, how do we use that? And I’m not knocking it in any way. Most of my usage is based off of flawed GPT, all that. But just like this paper, you know, another thing that’s interesting is this paper focuses on generative AI, even if it just focused on all generative AI. And so it’s still not looking at the broader AI ML picture.
You know, there’s so many use cases and tools that can be built with AIML that don’t fall under the scope that have one effect.
Yeah, I think it was a couple of maybe a month or so ago.
It was a law firm that’s local that actually made some bad press because one of the things they had submitted had referred to either cases that didn’t exist or had gotten, you know, something like that. And I don’t, I don’t know if they got fined or what the heck, but that, that made the news. Um, and if there’s anybody that’s supposed to know the rules, you know, um, so we’re all kind of trying to figure all this stuff out together. Cause even the legal side, uh, or the legal profession is still struggling with, you know, how to use it and how to. you know, how to regulate and things like that. So, I mean, it’s, it’s a very interesting, and it’s, I don’t have any, I don’t have any good advice on that one. I can just point at things occasionally and go, well, that’s broke.
And then find something that might work.
So, all right, well that, let me go ahead and stop recording. All right.