Transcription provided by Huntsville AI Transcribe
For those that haven’t been here before, this is Huntsville AI. We run a every-other-week meetup. Usually we’re in person over at Hudson Alpha, and we also do a virtual zoom call at the same time.
The main thing is making sure, I mean, basically what we’re trying to do is advance the knowledge and application of AI in ways that make it available to everyone and improve our quality of life. So that’s why we do what we do. You can also get to all of our stuff.
Let me find my GitHub.
If you go hit this presentation under GitHub, I will have the stuff we’re talking about tonight uploaded by the time we’re done.
But this is pretty much everything we’ve talked about going back to 2018. We generally try to make sure everything is up and available for anybody that wants to play along. So with that, this week we have Jack Francis talking to us about deep research.
And let me flip over, stop sharing, and I will make Jack the host after I admit two more people.
Alright, let’s see. And this is really, really annoying because I have to scroll through.
There we go, more.
And make host. Alright, so Jack take it away. Alright, sounds good.
Okay, so let me, I’ll go ahead and share my screen. Since we are virtual, let me jump back up here. I’m just gonna go with this mode because I’m going to be bouncing between this and some of the, some websites. So we’ll be bouncing kind of back and forth between the slides. But I wanted to kind of start off here.
So the topic for this week is deep research. So you may have heard recently there’s been quite a few different companies that have brought out a deep research-like capability. So today I wanted to just kind of overview those. We’ll do a little bit of a technical dive on on kind of how they generally work.
We’ll go through a few of the different offerings from kind of some of the major companies out there. Do a quick comparison. You’ll see there’s these things generate tons and tons of text.
So we’re not gonna get to do kind of a full read the entire output text, but as Jay mentioned, those will all actually be available in the GitHub afterwards and have kind of comparisons for prompts. Then we’ll dive into kind of just some practical tips to wrap up with kind of using deep research. So to start off, kind of what is deep research? It is an agentic personal research assistant. And really the goal is it would analyze hundreds of sources and generate a comprehensive research report for you. So you can kind of think of it like if you know, if I asked you to go do market research on some topic, likely what you would do is you would start with going to Google or equivalent, searching for a few sources, identifying information from that, then based on those initial sources that you found, go look for additional sources or kind of augment your plan. And you would continue iterating on that over time until eventually you come back with this like full research report that you could show off to answer the question. And so that’s kind of what deep research is at a high level. We’ll get into a little bit more specifics a bit later. But just to give you a timeline of kind of how recent a lot of these tools have released a deep research like capability. Google’s Gemini Deep Research was the first one and it came out only two months ago. OpenAI Deep Research released two weeks ago, and that was kind of the impetus for this kind of this presentation. But even since then over the last two weeks, Perplexity released a deep research capability a few days ago, and then Grok has now a deep search capability with their Grok 3 model that came out on Monday night. So with that, where I think is a great place to start is I’ll start with an example, kind of see what it is, how it works.
So with that I’m going to jump over to this initial Gemini window so we can kind of see how it works.
So the prompt that I’m going to start with from a deep research perspective is Research AI ML groups in the Huntsville area. Tell me about each of them and provide a comparison between the groups. So, oops, and this is actually, okay, so this is one thing I’ve done a few times.
I forgot to switch over to the pro with deep research mode. So let me switch back. Let me jump to that really quick. Apologize there. And paste that in. And so each of these deep research tools have a little bit of a different way that you kind of interact with them initially.
So the way that Gemini does this, I’m gonna go ahead and start the research and then we’ll talk through that research plan that it provided. But the way that it will work is you provide a prompt. It’s then going to provide a research plan for you.
Before you start the research you can go over here and edit the plan. So for example, if I wanted to look through these and let’s say I don’t care about the leadership team in the edit plan response, I could tell back to Gemini and say, hey, actually I don’t care about the leader or I don’t care about 2D. Please remove that from my research plan. And so if you do that, it will then come back with a new research plan updated based on that kind of that response and do that. So once you have clicked on start research, it’s going to start looking.
It’s gonna say you can leave the chat which has been a nice feature because it does take a few minutes to go. But on the right it’ll say hey, it’s starting your starting research. It’s gonna search the web and browse content from relevant websites. So within just a few seconds here, we should start to see some websites coming through. This is a little bit variable depending on the Depending on the specific prompt that will come in.
So we’ll give it just a second.
But it will eventually pop up and say yeah, here’s a good example.
It’s gonna say researching 16 websites. And so basically what this is doing is deep research is reaching out using Google across a variety of different sources. So you can see here’s our Huntsville AI group. It picks up on both of those.
It picks up on the other Huntsville AI unfortunately. It picks up on some companies that have you know, have big AI presence. So I’ve grown this prompt a few times.
Arcarithm and SAIC it tends to pick up on.
It picks up on something from the Chamber of Commerce and then it picks up on some UAH kind of AI research activities as well.
So it’s doing a fairly good job. You can see now it’s already up to 148 websites.
This number will vary and as I go through and look some of these are going to be related. Some of them may not be but we’ll kind of we’ll come back to this here in a bit to see what kind of the output would be. Now the other one I wanted to quickly show before we jump back to the slides. This is an open-source recreation of deep research by the Hugging Face team.
So this Gradio app that they have is significantly less capable than kind of their full version and I’ll talk about some of the benchmarks for that a bit later.
But the one thing that I did want to show that I think is really helpful from this view is it gives you a little bit more information about what’s actually happening under the hood from a deep research perspective.
So we have the same prompt here research AI and ML groups in the Huntsville area and tell me about each of them and provide a comparison. So it’s first step from open deep research. It’s gonna say, hey, let’s begin with a web search to find online directories, search engines, local organizations about that and then here it’s providing the query that it’s providing. You can actually see in like the Python interpreter and then the execution logs what it’s actually pulling out.
So it’s doing a web search just like we would do from Google.
It’s getting, you know, these various, you know, top results that it’s found and then based on that now kind of its next step is it’s identified some so I’m gonna try and visit this first one that I found. So it’s gonna again use the the Python tool to actually visit that page and it’s gonna give you information from that page in text. And then it will be using that now here from like a details perspective to, you know, craft, you know, what is the focus, activities, contact, description, and location about that specific one. So it’s gone through for that Huntsville AI group. Now the next one that it finds that it thinks is relevant is UAH.
Then it’s going to find HSV.AI, which is our group and provide some information on that as well.
And it continues this kind of like step-by-step process where it’s taking essentially at every step it’s using one of the tools that has available to it to make a, you know, recommendation or kind of understand, you know, what it, what should be the next thing to do so they can continue getting information.
So if I continue kind of scrolling through this you can see it will, you know, continue to do the same thing. This one ends up, and this is kind of with it being a much limited version of kind of their full one, at times you will see some errors here. But at the end you’ll get this kind of like final answer as an output.
So it’s going to give some information on this Huntsville AI group, and give some information on our group, and then it’s going to give some information on the ARC at UAH from there. And so here are some comparisons, here are some additional considerations, and then some conclusions.
And so jumping back to the slides, kind of the high level on how deep research works.
And again for a lot of the other approaches they are closed, so we don’t have specifics on how they work.
This is taken from that like open source recreation of it, but the general kind of skeleton is more or less how I would assume that most of these work.
The way it would be is you have some user prompt that comes in.
The deep research framework has a set of available tools that it has access to, so things like a web search. This one has page up and page down, a finder tool to essentially mimic like a control F or command F feature, a download tool, a visualizer, a Python executor, a text-based web interface, and then this like kind of like final answer. Which you know, we want to have kind of some some control over what actually goes out to the user, and so that’s why a final answer of some sort is actually needed.
But from the kind of the skeleton, the first step is going to be to create a plan given the prompt that I have been provided.
So with Gemiini we saw that that was immediate, right? It came back with the research plan and wanted us to kind of ride along with it to update and edit that. For this one, and for some of the other tools we’ll see, you will just put the prompt in and then it will generate the plan on its own.
Once it has the plan, generally you have some allocated way, or you have some allocation of steps that you can take.
So you can think of this as like for every step I’m using a tool of some sort, and so how many times can I use different tools?
And that would be your kind of your max steps.
For the open source implementation, they just have this kind of max steps as like a general thing. But what I imagine from kind of looking at the outputs from some of the other tools is that this max steps is broken up based on what stage it’s in.
So if it’s in a research stage, you know, it may have so many steps of like searching the internet, then it may have another set of steps for comparing specific or comparing and summarizing the information, and then some final steps for actually outputting that to the user.
But kind of the high level would be is for each step I’m gonna be able to take some action and that’s gonna be leveraging some tool that I have access to based on the memory. So this is, you know, all of the the messages and information I have so far, that’s gonna return back with, you know, some results from that call and which tool was used. I’m gonna have some planning interval that I’ll update my plan for.
So you can imagine this, right, if I’m doing like market research, for example, I might do four or five initial kind of Google query or Google Google searches to to find something out and then after that fourth or fifth one I now have a better idea of what I should be searching for, what I should be looking for. So it needs a it needs a step to be able to update the plan so that it can continue kind of going forward and it may make adjustments based on, you know, what information it’s found already.
All of that together we would then update our memory and then we’d have kind of a check to say hey if the tool that was used was, you know, get the final answer, we’d have some step for validating our answer and return back either the the last the last message that was provided or, you know, some, you know, summarization or, you know, modification of that kind of last message.
And if we get through all of those steps, however many steps we have in our in our kind of our batch or allowable range and we don’t get it, we never hit that final answer, then at the very end, you know, with whatever information you have currently just make sure that you get that final answer and return that back to the user. So that’s kind of at a high level.
That seems to be how these models generally work.
And so with that wanted to, well I’ll pause there for just a second, see if there are any questions or thoughts on that before kind of the next part is jumping into some of the benchmarks and then kind of continuing on from there.
I think it looks pretty cool so far. Okay.
Anybody in the lobby? I have I’ve gotten notifications to admit people and I’ve been admitting them as I see them at the top. So I think we are good there. Let me see.
I’ve got chat pulled up on the other side. So you make sure and yeah, I’m not seeing I’m not seeing any any questions there. So yeah for just I’ll stop kind of periodically throughout just to make sure but if you have any questions feel free to put them in the chat and I’ll be I’ll be monitoring that as well on the on my screen over here. Okay, so we have kind of an idea for at least at a high level and granted I know there’s not a ton of technical details and that’s because a lot of these models are still closed source and are more or less like the framework of how they are working, they’re still closed source. But that does give kind of a general overview of how these models are working. So now the next step would be is okay.
So for these deep research capabilities, what are some of the benchmarks that are being used? How good are these models? And what you’ll see and we’ll talk through kind of some of the more direct comparisons a bit later, it’s gonna be a bit hard to do a kind of direct comparison because of how how much output there is.
And so we have a few benchmarks that are I would say moderately helpful.
They’re at least directionally correct.
One of them being the general AI assistants or GAIA.
This benchmark has three different levels of questions.
Level one is basically go look at this website and get me information from that website. Level two is you know you need to either interpret maybe an image in this example, or you would basically have like three to four steps of kind of continuation that you need to do.
So like for this one you need to you know process the image.
Then you need to also look up what are the US federal standards for butterfat content?
And then you need to combine those together to provide an answer.
And then level three is a very difficult.
So this one is if you’ve had a chance to read it, we’re looking at a specific picture of the day from NASA’s astronomy picture of the day.
The two astronauts that are in that picture look at the one find the one that’s smaller than the other one.
Then for that smaller astronaut look at all of the other astronauts that are in his group.
Out of all of those in the group figure out which one spent the least or spent the least time in space, but exclude any that did not go to space at all.
So if you think about like even if you’re trying to solve that right, I likely would need like notes to take along and continue to do that.
Like a model would have to have a significantly or a very good understanding of like the context of the question and make sure that it doesn’t get derailed along the way.
And so these these level three questions are still pretty difficult.
Even OpenAI deep research which got 67% on this benchmark overall still struggled at about only about 40% on these level three ones. The open source deep research here, this capability, so to kind of give a kind of to set this to set it straight for kind of what we saw earlier in the Gradio app, the Gradio app it uses GPT-40 as its back end and it can only do 10 steps. They’re full version.
It seems to use or it used 01 and then it could take many more steps though I haven’t been able to figure out exactly how many steps and so that’s why you’re gonna see this is going to be significantly better because it has a much better model on the back end. So throughout the presentation, I’ve included kind of the open source. You’ll see kind of for the comparisons using the Gradio app.
Know that those are kind of the the lower end and it could get better but I did not get a chance to set it all up locally for myself.
But the kind of to give you a perspective on GAIA GPT-40 just kind of out of the box gets about 7% on this benchmark.
The other benchmark that OpenAI gave results on from a deep research capability is called Humanities Last Exam. This is one that’s been fairly popular over the past few weeks.
The idea is that it’s a bunch of very specific graduate level questions generally coming from professors in very niche scientific areas or just kind of just niche areas in general. And then we’re asking, you know, large language models to answer these questions. So these are questions where you know, they may appear once in training data.
So it’s really something where like being able to search the internet be able to you know contextualize information could make these significantly better. So here you can see a couple of the questions. You know, the one on the right is asking about paired tendons and a bone for hummingbirds. I have literally no idea. I wouldn’t even know where to start outside of just googling that question.
But on the left side, you can see a lot of kind of the traditional models in Gemini Thinking here to be clear. This is not the Gemini Deep Research. This is just their kind of their advanced or their flash thinking model. But OpenAI with the deep research capability was able to get 26.6% accuracy, which is almost double anything else that has been tested on Humanities last exam. Again, this one came out a couple weeks ago. So really only have info on the OpenAI Deep Research.
Haven’t seen anything from Crocker Perplexity yet. And then this came out after Gemini Deep Research. So hoping they will go back and update that but I haven’t seen anything yet. So those are some kind of overall benchmarks on how we’re comparing these different models.
But honestly, because these models are generating large amounts of text, a lot of it comes down to kind of personal like vibes essentially on like how you feel about the model and then some different comparison between the tools on like why you may or may not use them.
So we’ll jump into a kind of a more direct comparison between the tools.
We’ll go through this a little bit quickly just to because a lot of this is just gonna be information that you can read and I don’t want to sit here and read all of these out to you. But we’ll start with Gemini Deep Research. We saw the workflow earlier from a pricing perspective. This is included in the $20 per month Gemini Advanced tier. There aren’t any kind of like overarching rate limits, but you can only do six concurrent research tasks at the same time. Three to five minute time frame.
You can get text in browser. You can also open it in Google Docs natively and I’ll show you an example of that in a little bit.
And on average, you’re looking at like four to six pages of output.
A few quirks and bugs that you know have listed down here.
But overall, it’s been a good model.
This is the one that I have the most experience with.
And then this over this past couple weeks, I worked with David and he has access to OpenAI Deep Research. And so a lot of the kind of the the points from this section for the OpenAI Deep Research side are more his kind of thoughts. But I’ll talk through them and then I’ll give him a chance to add anything there. But here kind of the workflow for OpenAI Deep Research is a little bit different. Instead of providing a prompt and it providing a research plan, you’re gonna provide a prompt and then Deep Research will ask you for follow-up questions. And I have an example that I’ll show a little bit later of kind of that full output where you can see all of the follow-up questions that it’s asking. But here is just an image of what it looks like when you’re waiting for it to generate that full response.
So you’ll have a window here, you know showing what it’s currently looking at.
And on the right side you can see its activity and what sources that it’s using as well.
From a kind of other metrics perspective or other things perspective, currently from a pricing perspective, it’s only included in the $200 per month OpenAI Pro tier.
It does rate limit, you can do a hundred deep research queries per month. Generally, it takes about five to thirty minutes for this for it to generate your research request and it’s around eight to forty pages of output.
Have a few quirks and bugs kind of listed there, but then David was gonna pause there really quick see if there’s anything else you wanted to add kind of over the last few days of playing around with it or anything else that you thought would be relevant for the group. No, I still need to get more time comparing it.
Honestly, the last few days have been spent in Grok. So no, this is this is a good overview.
If anybody has any specific questions about it, I’ll be happy to answer.
Okay, awesome, sounds good. So after that the next two have were added earlier today because Grok came out on Monday night and then Perplexi came out pretty pretty recently before, so I don’t have a ton of info on them. But just to kind of give you a high level for the Grok deep search, the prompt is you’re providing a prompt as the workflow, your pricing is generally it’s going to be in the $40 per month, the X premium tier, generally less than five minutes and output format is text and browser.
We’ll see we’ll go through kind of one example, but I don’t have kind of a general sense on you know what what the expected links will be when using deep search and then kind of from a quirks and bugs it’s still pretty pretty early on to have any kind of specifics there. And then Perplexi, a lot of the same things here.
You’re gonna provide a prompt as the workflow. One notable thing here there’s actually a free and a $20 per month variant.
If you are logged into Perplexi but you’re on the free tier you do get five queries per day that you can use.
If you’re on the paid tier you get 500 queries per month is kind of your limits there.
Time to complete two to four minutes kind of same thoughts there from a similar to Grok on you know text and browser but it’s still pretty early on from a Perplexi deep research and I haven’t gotten to play around with it too much yet.
And then finally the open source implementation that we showed a little bit earlier.
You know similar kind of things here providing a prompt like we showed earlier.
There’s not kind of that interactive piece like with OpenAI or with Gemiini, but the pricing it is free and is open source.
So you can go and play around with it connect it to bigger models to do better and that’s really the main kind of part that I would mention here is that it’s just it’s it seems to be pretty dependent on the base model you use.
You know, they got much better results on kind of the GAIA benchmark when using 01 is that back-end model, but with the Gradio version using GPT 4.0 and limited number of steps you can see it wasn’t it wasn’t great.
So that would be something that you probably have to play around with and tune a little bit for it to be for you to get like really good performance locally.
And so, you know, I did not get to go through all that over the past couple weeks but as a heads up on kind of where it is and and how you can use it. And so that leads us into kind of a more direct comparison of the tools.
So what we’re gonna do is and I’ve got a couple things. Let me pull, sorry, let me pull the view window to the side so it’s not covering over stuff. So what I’ve done, so the outputs for these tools and you can see I mean just looking at some of like the number of words for some of these they’re extremely long. So manually reviewing them like during this meeting is just not, it’s just not feasible.
And so what I’ve done is in the, sorry, this keeps bouncing, my window keeps bouncing around. What I’ve done is in the presentations repo that Jay mentioned earlier, there are ten prompts listed here and for each of those prompts they were just passed into kind of an unedited from that prompt, passed into OpenAI Deep Research, passed into Gemiini Deep Research, and passed into the open source kind of that Gradio app that you saw.
All of the outputs have been unedited. They’re just copied and pasted or provided a link directly to where you can access them. And so you may be wondering there are some weird titles here. You know, how did I come about picking these?
So David and I worked together on the history of AI one and that’s the one we’ll do a little bit more kind of direct comparison on the next few slides.
The other nine, there are a few Reddit threads out there where people are offering to, they had extra Deep Research queries available for their for their month. And so because of that they were able to, they were asking people to give them prompts and then they would answer the prompt or pass those prompts to Deep Research and then provide the input or the, sorry, they provide the output back on Reddit as a chat GPT shared link. So I went through, I found nine of those, that’s prompts two through ten. I went and copied the prompt exactly, gave that to Gemiini, and gave that to the open source model.
And so all of those are available in, will be available in the GitHub.
And I’ll show you very quickly, let me pull this over really quick, to show you what that looks like.
So within the repository for each prompt, there’s a prompt.md that gives the full prompt out and then the response here will give you the open AI Gemiini. And then for this AI history one, we do have Grok. That’s the only one we have a Grok example for.
And then the open source one, which again is generally going to be pretty limited.
But you will be able to go through and access all of these so that you can kind of for yourself see, you know, which one was best. I’ve got some initial results, but a lot of this is really going to be dependent on, you know, which one you find the most useful, you know, and so forth. So all of that will be available.
It’s all in the GitHub.
And so to kind of give you a peek of that, we’re gonna go through the AI history one in a little bit more detail.
So let me jump into here.
And let me, actually, let me check chat really quick to make sure there is no no questions or anything.
Yeah, Nick asked if you’re able to bring in your own data. So it depends on the model on whether or not you can bring things in. Right now, I believe that OpenAI Deep Research, you’re able to bring stuff in, but I believe it’s only as an image, if that’s correct, David? I think with Gemini you can do a little bit.
You can, you can connect to a Google Drive now with OpenAI, but at the same time, a lot of the higher models really just accept images through the chat GPT interface.
The OCR is really good.
So you can kind of work around it by converting a converting a PDF or a document to a large image that contains multiple pages and feeding that in. The OCR is good enough in that you can work around by doing that. But yeah, Grok, Grok, I’ve been able to just drag and drop or click upload and it’s chewed through anything I’ve put in so far. Awesome, yeah, thanks for that. And I have, I’ve actually in like my playing around with Gemini, it’s been mainly more internet focused, so I’ve not brought anything in, but we can pull that up once we get towards the end and see whether or not that has the same thing as well. So and then yeah, I think that’s kind of the the main questions there. So next what we’ll do, we’ll jump into a comparison for specifically for the history of AI. So this is a extremely long prompt and we’ll talk, it seems to be for most deep research kind of capabilities, the more structured and more information you provide ahead of time, kind of the better your prompt will be or the better your output will be.
So David and I worked together. We had like kind of an initial idea for what we wanted to cover from a history of AI. And then David put that into O3 Mini High and asked it to generate a research plan and then we, you know, slightly modified that after from there.
So for this one, we’re gonna look at a literature and milestone identification, look at the scope data points.
There’s some specific milestones here, look at some chronological structuring, so timeline format, linkage, a technical deep dive, tracing the model lineage, some comprehensive kind of here’s all the different topics that we wanted to cover. So early theoretical foundations, neural net research, symbolic AI, machine learning paradigms and statistical methods, advent of deep learning, and then the development and impact of transformer models and large-scale language models. And then asking for, you know, detailed references for all mentioned papers, some broader impact evolution of ideas, and then some output format specifics along with like what the what the final deliverable, man, what a word, what the final deliverable will be. And so here’s kind of the high-level summary of each of these various ones and I’ve got them all pulled up so we’ll dive into each of them as well. So for the first, for kind of the start with the bottom just because the open-source one again, that was the Gradio app. So again, it’s significantly less powerful kind of in current form and so because that I was not expecting it to do extremely well. But again, you know, the more the power there from the open-source side is that you’d be able to connect your own model and be able to, you know, scale that up.
So we’re mainly going to focus on OpenAI, Gemini, and Grok.
So OpenAI for this AI history prompt generated almost essentially 32 pages of output.
It covered 48 different AI topics in the timeline and it cited 27 different sources.
From a prompt adherence perspective, I would give it an A+. It consistently followed the provided format for all the topics even for like the general trends it provided specific papers and projects along with the overall impact.
So let me jump over to here.
Oops.
The bar of zoom is killing me right now. Okay, one second.
Hopefully we will… I need this to go up. Okay. Um, sorry, let me pull this down just a bit and I believe it is this one here.
Yeah, okay. So this is, and I’ll scroll fast because it is very, very long.
This is all the single output from OpenAI’s deep research.
So what you can see and what I wanted to share really quick.
So here’s the the full prompt that we pasted in and to be clear in the GitHub, the link to this is provided there.
So you’ll have access to go click in through prompt one, click on the chat GPT share link, and you’ll get exactly the same page.
So we pasted in the initial prompt and then asked some questions.
Five questions on depth links, timeframe, focus areas, preferred format, citation style, and then we answer those questions and then everything after this is one single output from OpenAI deep research. So I’ll kind of slowly scroll through this kind of at the start and then we’ll jump to the to the bottom to show some things but some of the important points, you know, it followed kind of what we were asking for from an abstract and key contributions, a technical analysis, a model lineage.
You can see if you hover over this it’ll show you the the source and one thing that’s actually unique I think that OpenAI’s deep research does that the other ones don’t is if I click on this specific source I mean, it’ll actually highlight the the portion of the other document where it is pulling that information from. So if I jump back here from I believe it was this this one here you’re able to see like hey, it’s it’s not just pulling from this general page, but on this page it’s pulling from this specific spot.
A couple of things that are notable and obviously showing OpenAI first you’ll see kind of the contrast as we look at the other ones. It’s generally very good about it still has kind of the the list structure because we asked for that but within each topic it’s really good about synthesizing that information down.
So it’s more of a you know, this is more of how you would expect a research paper to be written.
It’s gonna be written in paragraphs where like each paragraph has a different kind of focus for that. It’s less like a kind of what you typically would get out of an AI model or one of these large language models where you have like all of these bulleted in text and we’ll see some examples of that going forward. But overall it does a phenomenal job.
It goes into significant detail for a lot of these. I’ll actually jump down.
One that I’m gonna show we’ll do a more direct comparison on a little bit later will be GPT-3 just because it’s a it’s a good kind of step here.
But what you can see is like it’s pulling from you know, the language models or few shot learners.
It gives us a direct archive link to that paper.
It’s telling us it’s you know, it’s 175 billion parameters.
It’s talking about you know here about competitors at the time talking about the lineage. Like it does a it does a very good job in my opinion of a what I would call like deep research. So it’s not just giving you that high level of what is the what is this thing? But it’s going kind of that next step down of like, how does this fit into kind of the overall environment that it was in at the time? How are these related?
And it does a really good job of like formatting it in a way that at least to me seems more like a research paper compared to what you’ll see from from the next two coming up.
So with that and let me double check check in chat over here.
So then the next one.
Okay, wait, okay.
Sorry got a couple of questions in chat. So let me see.
Okay, yeah, the first one.
Oh, go ahead David.
I was just gonna say on the being able to vet whether it’s accurate or not. It is it is a definite shift in the time frame.
So I found with deep research, you know, you’ll you’ll spend a little while formulating the prompt, go away for half an hour, come back and then you definitely have a lot more work in one sitting to validate.
It’s different than we’ve kind of gotten used to with faster individual AI responses where you’ll validate for a couple minutes, do another prompt, validate a couple minutes. You know, I’ve spent three or four hours validating one deep research response. So Have you intended to find that since it’s finding resources that it’s a little more trustworthy or do you still have to be very particular about coming through it? I’d say be very particular. If nothing else it gives an excellent starting point and is still worth it.
This is just total napkin math, but in general I’m feeling like 75 to 80 percent of it is accurate, which might not sound like much.
There’s a lot of errors in there, but I found that, you know, out of 40 pages of response I’ll still get 30 usable pages of content and information that I can go back to you.
Cool, thanks. Yeah, and Nick to your question about how is it finding sources? How does it determine its sources? That’s a good question.
I looked into that a bit.
Because all of these are still closed source, I’m not sure what like tricks and things they’re doing behind the scenes. At like a very high level it’s, you know, I’m gonna go Google specific topics or, you know, look up specific topics on a, you know, in some rag of a common crawl so I can find what things are most relevant. But it’s unclear on how they’re doing that right now. A lot of that is hidden in the like specific thinking tokens and a lot of these are not showing that right now.
So I don’t have a specific answer to give you, but that’s a yeah. Yeah, I mean definitely potentially could be as well. It could be it could be a graph-based approach. So with that, so we look through OpenAI.
It’s definitely, I would say, far and away the best right now from kind of from a true like deep research of, you know, getting more than kind of that high level.
Next one we’ll look at is Gemiini.
So it’s response was about 10 pages.
I’m still a significant amount of information. It covered 35 AI topics in its timeline and it had 41 sources cited in its final report.
So prompt adherence, I gave it a B. It followed the format for some topics, but for some of them it missed like its tech contributions, its impact, or lineage.
It generally was a higher level overview.
It did not go into as much technical depth and kind of interestingly it never cited archive, which I thought was kind of interesting, especially because you know, we were explicitly asking for that. So I’ll jump over to the to the to the Gemiini one. And so as you can see this one and actually I’ll show an example.
So here for any kind of prompt, this is from our AIML one, you can just click this open in Docs button once it’s finished.
And so that’s why I’m showing it in this kind of Google Docs form since you have that direct move to Google Docs. And one of the nice things is that it will it will copy over all of your references.
So it’ll give you a superscript here and at the very end it will give you a work cited.
But what you can see for these, we are covering a bunch of things, but it’s generally pretty limited information.
Generally, it’s authors and then the abstract slash key contributions.
As I scroll down a few of these it will have more information.
It’ll have you know, the more full-on technical contribution, impact assessment, and subsequent developments.
But generally it’s staying at a higher level. It’s not going as deep as OpenAI’s model.
But what it is good about is that it is it is capturing a lot of kind of the the expected AI parts of history that I would expect it to capture.
And it’s doing a good job, even though it is limited on each one.
It is capturing kind of all of the main ones that that I would have expected to see as I kind of slowly scroll through this.
And this has really been kind of different from OpenAI’s deep research.
I’ve used Gemiini a lot and that’s one of the things that’s been helpful for me is it’s more of a high-level here’s the general things and then I’m gonna go do kind of further deep research on that. But overall it’s done a fairly good job of kind of you know, what are the the main topics that it’s covering? One thing interestingly for Gemiini, and I’ve seen this in a few things, it tends to have less information on just like immediately past information.
Or like your very recent information and we’ll see that’s kind of much different than Grok which we’ll look at next.
But here’s kind of basically this is all that it gave on kind of the large language model side.
So significantly less than kind of the OpenAI side.
But overall it’s capturing kind of from an AI timeline slash history.
It’s hitting on all of the major events I would have expected.
But it’s doing it at a much higher level compared to OpenAI deep research which went into much more detail. They have a lot more information kind of around the the specific topics themselves. And then that will take us into Grok, which is actually pretty interesting.
Because when David had run this the first time we passed in the full prompt or the first time he passed in the full prompt, sorry, it generated roughly three pages of output.
For that it generated 15 topics in the timeline and had six sources cited. But what you’ll see, and I’ll pull this up when we pull up the Grok example, at the very end of it it basically was like this is a condensed summary that I’ve provided, but if you’d like me to give a more full explanation I can do that. And so David continued to say, hey, yeah, give me the full expanded one and then continue to do that four more times. And that ultimately gave a full output of 20 pages around almost 10,000 words. And I’ll show you that in just a second.
But for that full output it had 43 topics in its timeline, but it did not have any sources explicitly cited.
There was lots of info on like where the paper was published, but it wasn’t a direct link to that paper like we had asked for.
I would say kind of prompted here and kind of doing a combination of the two, it followed the format better than GemIIni. But it provided fewer relevant results from kind of that initial prompt.
For the kind of for the longer one where it provided kind of those 43 topics when we continued to, where David had continued to prompt it, it did much better.
But there was also a kind of surprisingly, maybe this is just a, you know, again this is a sample size of one.
So it’s always dangerous to extrapolate from there, but a lot of it seemed to be more focused on more recent things lately.
And so let me pull that up really quick and pull up the the Grok one.
And so what you can see here, I’m again kind of the initial prompt that was provided, you know, similar to the others. And as I scroll through, I’m gonna scroll through this one a little bit faster.
This is the first output that Grok provided. So it goes through and it talks through, you know, very high level. It’s not capturing a ton of different things, but it’s giving a little bit more information on each one it provides than Jim and I did.
But once I get through kind of this first conclusion, you can see here, you know, this condensed report captures the essence of AI’s technical history. Full version would expand each entry and so forth and so forth.
So David said, you know, research and expand all starting from the beginning.
And so now what it’s going to do is it’s going to go into much more detail for each of the things that it already talked about. So here, theoretical foundations, you know, significantly more information as I scroll through this, you can see.
And what will end up happening is roughly every, I believe it’s every 20 years-ish, it’s gonna say, hey, this section was 1940s and 50s.
Subsequent decades will follow the same depth. Here are some things I would talk about.
Would you like me to proceed with the 1960s next or focus on specific era? So David said, please proceed.
And so now this next kind of response talks about the 1960s and 1970s. We get to the end there and now it says, okay, now the next part will be the 1980s and 1990s and please proceed. One thing that was interesting as I went through and did this, and this is just kind of a rough calculation, the number of words in each of these sections, including the first one, is around 2000.
So I’m curious if potentially Grok is limiting how much output it can have for one step of deep research.
And that’s why it ended up doing it in this format.
Unclear, again, sample size of one, but something kind of for consideration there.
But the second output was definitely much more in detail. Gave a lot more specific technical details on the on the specific topic that it was talking about. So let me… Yeah, go ahead, David. Yeah, something I found since I sent this was I did more experiments on my own to get actual citations, especially from archive.org. Grok currently has a tendency to hallucinate the archive.org links. It understands the standard naming convention, you know, the date and the progressive numbering, and it’s basically tried to guess what the correct link is.
It always links to a paper on archive.org that’s close, but it’s usually a few numbers off.
So I am being cautious with links from Grok to archive.
I haven’t had that trouble with other sites yet.
Hmm. Yeah, that is that is interesting. Yeah, I had noticed there weren’t many citations. But yeah, I would rather have no citations than broken citations. That is a… Now it did give the correct paper name and info about the paper.
So I was just able to do a quick search for the paper name and it completely matched up the info it said about it. I just couldn’t put the direct link inside Grok. Yeah. Do you recall roughly how long it would take before you had to prompt it to continue? Like how often do you have to return to Grok to be like, yes, please keep going? So it was it was way different. I did time it. So the first response and for most deep search questions it’s been about two minutes and that’s to complete response time, like from sending the prompt to complete response. And then for each of those follow-ups on this one, it took right out a minute and a half for it to generate.
So it was a more involved process, kind of more similar to the models that we’re used to using now than the others where it’s just send it and I can’t remember if you mentioned or not, but most of them have an alert you can set up so you don’t have to check back. It’ll just alert your device when it’s done. Yeah. And then the other question, Sushil, and good to see you. So the models underneath, they do seem to be tuned.
I know OpenAI has mentioned that for OpenAI Deep Research, it is 03, but it does seem to be fine-tuned for Deep Research specifically.
I believe that Geminis is the same way.
Unclear on Grok and Perplexity on whether or not they’re fine-tuned. I would assume that they are at least a little bit towards Deep Research. But that is another piece that’s like becomes a bit more difficult to recreate some of these because there is some tuning going on before they’re releasing kind of the full Deep Research capability. Yeah, so with that a couple things and we’ve kind of touched on some of these but wanted to kind of give a some specific examples. We’ll walk through kind of the output for what the model said about GPT-3 for OpenAI, Gemini, and then Grok, kind of first response Grok and then the deeper response Grok. So the one on the left, OpenAI Deep Research, you can see it is significantly more involved.
I’ve taken out the the the links or the kind of the citations to language models or a few shot learners that we saw earlier just because there’s some weird formatting that comes with that. But you can see, I mean it goes into significantly more detail. You know, it talks about the the paragraph before this was on GPT-2, but it talks about, you know, competitors that it had. It talks about the lineage significantly, you know, GPT-1 coming, you know, coming after BERT and Transformer, which were also mentioned previously in the OpenAI Deep Research output. It talks about few shot prompting, talks about, you know, leading the way for chain of thought prompting, prompt engineering, prompt tuning, and then that, you know, led to chat GPT, GPT-4, and so forth.
So, you know, a significant amount of information and like David said, you know, there is the the validation verification on the other side.
But in general, as I’ve read through read through the various prompts, and I’m not an expert in some of the other fields, but overall it seems to do a to do a fairly good job there. On the Gemini side, this is actually, as we showed earlier, this is pretty much all that it gave after 2020. And from kind of a large language model and beyond, you know, it talks about GPT-3, BERT, Lambda, and PALM-2 as being, you know, models that were there, but it doesn’t give a lot of information beyond, you know, just like the very high level.
You know, the impact is more talking about LLMs in general.
It does a little bit more for chat GPT, but not, we’re not really a ton from a, you know, what I would consider what I’d want from a deep research perspective.
But it is hitting on kind of the major things that I would have kind of expected it to see.
Weirdly though, neither of these models or neither of these approaches end up citing anything from 2024, whereas Grok is actually willing to cite itself, which actually I meant to show because I thought was pretty cool.
If you go all the way to the bottom of the Grok response, it actually has Grok-3 on the AI history timeline, saying that it’s speculative as of February 18th, which I thought was pretty good that they were able to, however, they, you know, fine-tuned it, it was able to already have information about itself, which I thought was pretty good.
But then if we go over to the Grok examples, the initial response for GPT-3, you can see more specific technical information than what Jim and I provided, and kind of a high-level overview.
But then for the deeper response, it goes into much more detail.
You can still see kind of on both of the Grok side, unlike OpenAI’s deep research, which synthesizes information, in my opinion, a bit better of kind of generating it into that paragraph format. It still is very AI output-like, where it’s lots of lists and bullets, and so that’s where it is getting a lot of good information.
But then if you were going to go and, you know, move this into some sort of research paper or something after, there would still be some level of work kind of on your side to go through and do that. So with that, kind of the last piece I wanted to touch on is just kind of some practical tips on things that, you know, I’ve experienced working with Jim and I, some things I’ve seen on Reddit and Twitter, talking with David on kind of his experience. Two of the big ones from a practical tip perspective, like I mentioned earlier, these approaches tend to do much better when you have highly structured, very detailed initial prompts. So recommendation here, you know, whatever you’re wanting to, you know, do deep research on, take that prompt, put it into O3 Mini or GPT-4 or, you know, 2.0 Flash Thinking or whichever, you know, general model you have access to, and just ask it to make a more detailed, clear, verbose research plan. You know, figure out what sections, what format you want your output to be, and if you have, you know, specific sources that you’d want it to look at, provide those when you can.
And then the second one is, one thing I fell into kind of early on is, because, I mean, Jim and I was still generating, you know, eight to ten pages, it’s not at the level of OpenAI deep research, but that’s still a lot of information to go through.
I initially was asking it to generate an executive summary and then do kind of the full deep research portion after that. But what I found and is that it’s actually better to just let deep research generate kind of whatever it’s going to generate, and then if you need to summarize that, just use a different model to summarize that after.
That way you can take, you know, all of that output from deep research, summarize it down into one pages or two pages, and David’s mentioned, you know, Claude, in his opinion, out of all the models he’s tested, does kind of the best job of breaking down that info, but I’ve used O1, I’ve used 2.0 Flash for this as well, and have gotten, you know, good summaries as well, but kind of some practical thoughts there.
And then, you know, where would you actually use deep research?
The two main use cases I see currently are in market research and in like technical topic deep dives. In the open AI demo for deep research, they also talked about this like personalized reports on making, you know, purchasing decisions, like you’re looking to see what kind of car should I buy, you know, where you would be going out and doing essentially kind of a mini market research, but more tailored to you. I think it’s okay there, but really the first two are kind of the more useful ones in my opinion.
I think the kind of the mental model to think about is like if you’re doing some problem where like you would do a little bit of googling and then that googling would inform your you know, the next things you would look up a research and then that process continues until eventually you’re crafting a research report.
That’s kind of the framework.
That’s kind of the mental model I recommend for like when deep research would be a good fit is kind of that format.
And then to kind of wrap up, you know, what is next from a deep research perspective?
You know, where do we where do you see it going?
Where would it, you know, continue to improve over time?
The big thing is I mean vision it will get much better as we have better browser manipulation tools. So things like Anthropix, Computer Use Agent or OpenAI’s Operator because you know currently these deep research models are researching, you know, they’re researching across the internet and whether that is, you know, a common crawl saved into some vector embedded database or if it’s and I imagine that’s part of it, but there’s also kind of that live search as well. And for that, there’s lots of places where I would imagine that, you know, being able to hit access paywalls with, you know, a user’s information or being able to, you know, understand how to operate some page to get to a subpage that’s more relevant for, you know, for the for the agent.
As it’s able to kind of better navigate the browser, I would expect that the deep research output is going to increase because it’s more likely to find high quality information relevant to the prompt that you passed in.
I also would expect as like these reasoning models improve that we would see the outputs of deep research continue to improve as well. Kind of like we talked about last time, you know, for these reasoning models, it’s not actually reasoning. It’s, you know, reinforcement learning to get from, you know, point A to a correct answer.
But as these models have better tools and better access, you know, they’ll be able to identify better ways of getting from that initial prompt to output.
And I would expect so as these reasoning models improve, they’re going to be leveraging, you know, prompts like a deep research.
They’re going to be leveraging these browser manipulation tools. And so as those continue to get better, I would expect deep research to get better because it’s more, it’s generally better at getting from prompt to output in kind of the same kind of way. And then on the kind of where some of the the tiers and stuff are going, OpenAI, they’re planning to bring their deep research capability to the plus and the free tiers in coming months. It sounds like plus plus members will get 10 prompts or 10 deep researches per month. Free tier would get two per month. I’m expecting, it looks like, so Grok or I guess XAI, you know, that conglomerate, they recently upped their premium pricing to that $40 per month to account for deep research. Gemini, currently using the 1.5 Pro model, I would expect that to move to the 2.0 Pro model here fairly soon to kind of compete with some of the other ones. But kind of longer term, really would expect that you would see different levels of intelligence for deep research based on pricing tiers. So with deep research, because a lot of it is, you know, it’s the number, it’s tools that you’re using along with the compute on the back end. There’s lots of opportunities for them for, you know, for these larger companies to kind of have different levels of intelligence and tie that to pricing tiers. So maybe at a lower level of intelligence, you don’t get access to all the tools and you only get access to so many steps of kind of movement for your agent. Whereas if you’re paying more money, you get more access, more compute, more tools. And so you could end up in a situation where like we have pricing tiers that are tied to potentially, you know, how long did the agent spend on completing your request?
Or how many tools did you use?
Or what level of compute did you use? So I would not be surprised to see that kind of paradigm over the next few months as deep research kind of continues to get better. So that’s all I’ve got for the slides. So I’ll go through the prompts, or the prompts.
Sorry, I left it on my mind. Go through the chat really quick. Let’s see. Okay, for those who have tried deep research tools or seen any prompt framers, I’ve not seen, so Jacqueline to your question, I’ve not seen anything yet outside of the kind of having a more specific prompt. It still feels pretty early just because like the Gemini deep research when it came out, there wasn’t as much buzz around it. And so I think there are a few people who used it, and it was, you know, used some, but really it wasn’t until like OpenAI’s deep research that they announced a few weeks ago that I’ve really seen it blown up on Twitter and Reddit for a few different like, you know, more AI machine learning focused subreddits there. So it’s still very, very much like early days, and because these are generating so much text, it’s also hard to kind of go from prompt to output in the same way that we did kind of early on for large language models.
So, yeah, and then Josh, I think we’ll start seeing React or something like that to start interleaving tool culture. That’s a good question.
I would expect so.
I think that the kind of, at least how the the Hugging Face, like their open source implementation, the way that they handled that kind of approach is they have a, they call it a facts list, at least that’s what it gets called quite a few times throughout code. So basically like as they are doing their outreach to, you know, some site or something to get information, they’re always updating that facts list internally. And so that kind of becomes their like mini response. And then once you hit that like final response where it’s like, hey, I’ve got enough information, it’s going through its facts list plus like any other summarized information and generating that final response for you.
So I think to a degree that’s kind of already happening, but I’m not sure at like what level.
I imagine we could definitely get much more like involved on the tool call piece, but it seems like the general framework right now is like, do all of the tool calling up front, generate your kind of either your facts list or your internal summary, and then provide that back to the user instead of generate a couple paragraphs of the user sees and then do another tool call based on that and continue iterating in that way.
So that seems to be the kind of how I’ve seen it so far.
But yeah.
I think that is all I’ve got. So I’ll open it up to see if there’s any other questions outside of the chat. Hey Jack, when this fails, do you have an understanding of where it fails typically or where things are going wrong? Is it in the plan logic or the plan creation?
Is it in you know, the selection of the sources?
Like where do we see this, you know, our guess on the converse of that is where will there be improvements or where do you see improvements potentially?
Yeah, that’s a good question.
So I do have one example and let me my zoom bar keeps moving and I believe it’s this one. Yeah, this one.
And so this is a I will say this is a more Gemini specific thing where it broke.
And so I think this is gonna vary tool to tool. But for this one it was one of like the the default prompts that I had put in and I was I was hitting it a few times because I’ve seen this about 5% of the time with Gemini and wanted to be able to show it as an example in case someone asked. So what happened here is it did everything kind of successfully from a research different websites. This is for a and sorry, I’ll show you the prompt real quick. This is an AI powered marketing campaign to benchmark for 2025 planning. I edited the research plan to focus it on defense industries just to be able to kind of show that that capability as well. But what happened it seems like is it started doing the research for this for this prompt. It was continuing on and then eventually somewhere it basically just responded with like the next thinking action that it was gonna do. So like this is what I would imagine that internally like Gemini is supposed to be supposed to see but then it’s supposed to get here’s all of the snippets from the websites now, you know make that next make that next or make the response for this and it just didn’t. So it ended up like failing at some point, you know along the way of the kind of the internal steps from getting from the tool calls in the kind of research side to then kind of moving forward. So there’s that’s kind of the main that’s the really the only example I can give you of like where I’ve seen them like break so far.
But where I would see them to get better would be you know the like accuracy of information that we’ve talked about a little bit and also as we get like more browser manipulation tools as we get you know more ways of you giving them direct access to information where they can they have a little bit more flexibility to go find the information. I would expect they would get better because they’re gonna have kind of higher quality information coming in. So this is really the only failure mode I’ve seen so far.
There may be others but I’ve not I’ve not come across any yet.
I noticed your prompt was a lot more thorough than a prompt I’ve used for deep research. Have you found that giving it more information is useful? Do you use an LLM to help you generate that prompt?
Is there anything you can do to be like think longer because it seems like you’re part of your results here is that the longer thinking ones are better?
Yeah, so I’ve not found anything that’s like I think longer. I will say one thing that was interesting to me.
So when I when I when I started doing that I do use like an LLM to generate a more like robust prompt and then I’ll do some slight editing on that.
One thing that was interesting though, I did expect because of that very very point I expected that as the prompt length increased here I would expect to see like the output length from like OpenAI or from Gemini increase and there really isn’t much of a relationship there.
Now as far as like, you know, granted like this could just be right the the prompt could be longer but it could be looking for a more specific thing and that’ll be kind of some you know you’ll have to kind of go through and look at it to kind of verify for yourself because we don’t have time to go through all of them, but I’ve not found something that’s like a make sure that you think longer or give me even more examples here other than like if I prompted it after it finished generating I’ve not found I’ve not found any sort of like prompting or anything to like force it to be, you know Give me closer to ten pages of text compared to three pages of text or something like that Interesting. Yeah Thanks to you showing that I just tried flash thinking and gave it my normal short prompt let it write a long prompt and you know deep research is going through like a hundred websites. I’ve never seen it do that before so I’m like, oh look at that Yeah, and that’s one thing that’s also it feels very random to me So like this history of AI one it searched for 125 and the one that we saw Earlier the one that I had shown earlier. I think this one it was at 148 When we said so this one’s 177 exact same prompt this one I think this may have been the one that I’m pointing to so this one was only 38 So I’m seeing like very different numbers of websites that it’s searching, but I end up getting Relatively similar content.
So it’s like for some of them It feels like it searches a lot more, but it doesn’t end up using them as much So I’ve not found any really rhyme or reason for why that is the case Doesn’t like the information it got and it goes back and does a second search if you know If it didn’t find what it wanted or something, but that’s interesting. I had not really been trying that thorough over prompt So thanks. I’ll have to give that a shot. Yeah, absolutely Yeah, hopefully hopefully that helps that I have seen the other thing that that really helps with in my opinion is One of the things I’m doing for work is I’m doing a like a market research thing where I need or I mean I guess I don’t have to have it But it’s nice to have all of my outputs in the same like structured format because I’m ending up dumping these into obsidian Just as like markdown files. And so with the longer structured prompt, I’m getting very consistently I’m getting the same headings throughout so I can almost I can’t 100% rely on that But it is much more consistent in like the formatted output as well Which has been which has been helpful for comparing across different things and as I’m using as I’m using the deep research tools You Awesome, um, let me This See if there’s anything else Yep Yeah, I don’t think any other questions and I don’t have anything else So I guess we can go ahead and wrap it up there so J if you want to go ahead and take hosting back and wrap up there And had to remember to unmute my headset, um Yeah, my my personality at work is I need at least two mute mute buttons between what’s in my head and What I’m about to say So yeah, really really appreciate the work put into that The fun part for me was walking through what you had done with the history of AI If you look in the same folder that your presentation was dropped into in 20 in the 2025 directory you will find a What did I call this? Introduction to AI for learning quest which is the history and fundamentals of AI That took me probably 10 hours of personal time to put together In an hour and a half to blow through I’d be interested to see I’m gonna check that against what because you were walking through it and I actually I started off with You know early 40s and 50s, you know And then just you know, it was it was just about spot-on how I had broken things together at the same time, but yeah Well, and that’s one things like with the that would be another good one to compare against then and that’s why I wanted to give All of the like prompts and unedited outputs everyone can go through and look but that’ll be another good one to compare against is Like, you know the the AI deep research ones versus what you had come up with is a more, you know human specific marker On the topic to be able I was sitting there with perplexity going through I had the general outline I know what I’m looking for and I was pushing that in and getting it to go get me more more You know more things I could click through and grab images and you know to build slides, but that was pretty cool So finishing out I dropped some thoughts in if I can find them If my chat will quit scrolling all over the dang place So I think the challenge data for the Hudson Alpha Tech challenge, I think drops on Friday I’m not exactly sure.
I know they’re dropping it before the challenge because it’s only a one-day event this year. So After some of us take a look through that we you may see some Traffic on our discord trying to put together some teams to kind of go after that.
I think some of it is Kind of a vision model based type thing with I’m not sure what kind of scans are looking at but that might be something Looking I’m looking for topic ideas for the first week of March We’ve got a besides conference for those that want to attend. It’s more cyber related that I think is Yeah, just jumped my Yes thing is about as scattered as my thoughts are as far as what this comment section is and then Let’s see Looking at I haven’t nailed down a date with Josh yet, but I’m thinking the 12th That would be the spring break week for those that deal with spring breakage in in Huntsville area Then also thinking about an ex-social Hopefully it’s warm enough You know the third week of March It’s possible to get out of that get out and do do something Of course, it’s by do something is probably stovehouse with me having You know tacos and a beer In my hand, so there’s that And I think that’s it