Ash Tilawat: All right.
Ash Tilawat: Welcome, everybody, to Night School. Obviously, at the beginning of Night School, we always give our fellow learners a couple of minutes before we get started, but again, Night School is an opportunity for you to upskill in AI and do it for completely free every Wednesdays. Please mark your calendars.
Ash Tilawat: As we get started, let me know in the chat where everybody's logging in from. Last time we had a New Zealand, we had somebody from California, we had somebody from Europe, and obviously the amazing Austin, Texas always shows up.
Ash Tilawat: We are starting night school this week from Gauntlet HQ here in Austin, Texas. I just saw Madrid, Spain, that's probably the farthest.
Ash Tilawat: Austin, Houston, Turkey. We have Istanbul, Turkey, Plano, Texas, right around the corner. NYC, Philly?
Ash Tilawat: Excited to have all of you guys joining us. We had over 400 people sign up for night school. We're gonna give them a few minutes to get situated.
Ash Tilawat: If you've never been to Night School before, Night School is a free session that we run every single week, where we try to upskill individuals, specifically engineers and AI. We'll cover technical topics, we'll cover interviewing, we'll cover what is the latest and greatest tool or methodology that you should be using as an engineer.
Ash Tilawat: It is completely free. Please bring a friend next week. Oh, Bangalore, India is by far the farthest. Welcome, welcome.
Ash Tilawat: Today, we'll be covering Agentic RAG. Now, for those who don't know what that is, we'll start from the beginning, we'll start about… start building the foundations for this topic, and then over time, we'll get to the Agentic RAG specifically.
Ash Tilawat: Couple of things to call out.
Ash Tilawat: I have created an open source repository here.
Ash Tilawat: It is the RAG cookbook, so gauntlet-aidp forward slash rag cookbook. I'll be putting the link here inside of the channel. At Gauntlet, we believe that you need to apply what you learn, so this is a breakdown.
Ash Tilawat: of 5 different RAG methodologies, with evals inside that you can use after this class, so you guys can keep building and actually try out some of the things I'm going to be teaching you.
Ash Tilawat: Alright, we're about to hit that 100 number. Again, we're gonna give it one more minute, and we'll get started in just a second.
Ash Tilawat: Before we get started…
Ash Tilawat: I wanted to introduce a few people here inside of the Zoom. I have Sonny, Aaron, Derek, and Tom, which are members of my team here at Gauntlet AI, and there's gonna be so many questions in the chat, and there's absolutely no way that I'm able to answer all of those questions out loud.
Ash Tilawat: But these individuals will be there to answer your questions and give you some valuable feedback on any question you ask in the chat. So thank you, Sunny, Aaron, Derek, and Tom.
Ash Tilawat: Now, before we get started, if you're an engineer, let me know in the chat. If you're a PM, maybe you're a UX designer, what is everyone's professional background?
Ash Tilawat: Sales. Love it.
Ash Tilawat: Sales? Okay. Alright, Tom, this tells me we gotta add some marketing classes.
Ash Tilawat: I'm open to it.
Ash Tilawat: IT, software engineer, love it.
Ash Tilawat: Software engineer, data scientist, petroleum engineer, amazing.
Ash Tilawat: Pro Vibecoder, I love it.
Ash Tilawat: Marketing, Salesforce, Amazing.
Ash Tilawat: Now, one of the amazing things about AI is, as we're learning each of these topics, my goal is to start from the beginning. I'm not trying to make this hard to understand right away. The goal would be build the foundations. As we're building the foundations, you guys can get deeper and deeper and deeper into the topics.
Ash Tilawat: Okay, this is my queue to get started. What's gonna happen is we're gonna have questions in the chat that Derek, Tom.
Ash Tilawat: Aaron and Sonny are gonna be answering for you. I'm going to have about a 40 to 45 minute lecture. I'm going to stop in the middle for questions, and then we're going to keep moving forward with some Q&A at the end. Again, if you have not opened up the open source repository I provided, please check it out. Otherwise, we'll get started on Agentic rack.
Ash Tilawat: Okay, welcome to Night School. Today's session is Agentic Rag. Agentic RAG is a process by which we have an agent between the entire retrieval process.
Ash Tilawat: So there's something called a context window. A context window is the amount of tokens that you can actually input into a large language model that is then sent to the actual neural network, and then you get an answer back based on those tokens.
Ash Tilawat: What's actually happening behind the scene is statistical analysis on all the tokens that are being used, and what tokens should be predicted next. Meaning, a large language model is just trying to predict the answer to your questions using statistical analysis.
Ash Tilawat: This context window is all of that preemptive context that you can send in to that neural network or large language model to get an answer back.
Ash Tilawat: To give you an example of this, I can say, my name is Ash, tell me what's your name. And it could just give me a single name Mac. Or I can say, my name is Ash, I'm the CTO of Gauntlet AI, and I can say everything about Gauntlet for the AI to give me a different response.
Ash Tilawat: More context, more details, a better response on the other side. That's why we've created something called RAG.
Ash Tilawat: RAG stands for Retrieval Augmented Generation, which means that we have a question that is coming into the system.
Ash Tilawat: And we're gonna go out, and we're gonna find any relevant context associated with that information, we're gonna add that to our context window, and then we're gonna add that back to the LLM. So if I were to break this down really simply for you, what's happening is, let's say we ask a question like, what's the weather like in Paris?
Ash Tilawat: We're gonna take that, and we're gonna change that into numerals. We're gonna change it to a numeric language that the neural network actually understands. We're then gonna compare it to existing database of numerals that we already have. These are all mapped in the same language.
Ash Tilawat: They're embedded so that we can compare how similar they are to each other. We're gonna do a vector search on top of what that is, and we're gonna grab everything similar and get it back. Once we get it back, what we're doing is we're trying to ensure that these match what we're trying to actually query for.
Ash Tilawat: This is then added to the context window of the LLM as a prompt, and then sent over to help define and see what's happening.
Ash Tilawat: Now, if I were to break this down step by step from that example, what's the weather like in Paris?
Ash Tilawat: What we're doing is we're taking that term, what's the weather like in Paris, we're embedding that term, and then we're comparing those numerical values with the embeddings in our vector database. Now, we're gonna go into all these terms in just a second, but I want to give you a high-level view of what's actually happening.
Ash Tilawat: you have.
Ash Tilawat: a question coming in, you're translating that question into a language that the AI understands. You're then comparing it with a knowledge base where you have a library of information already in the same language. You're finding the most similar information, grabbing that information back, and then adding it to a prompt.
Ash Tilawat: Then you're sending that entire prompt back to the LLM to get an answer.
Ash Tilawat: The goal here is, as humans, when we answer a question, we go into our memories, we go into books, we go into research papers, we go talk to somebody else. We retrieve information before we make a decision.
Ash Tilawat: Similarly, what we're trying to do here is, if there's an initial question that we want to answer, we want to go out and retrieve information first. Once we have that root information, we want to keep moving forward and try to solve the answer.
Ash Tilawat: Now, for everybody in the room that is already familiar with RAG, there's a bunch of downsides to just using RAG out of the box. Meaning, if I just have a knowledge base that I'm routing everything through, then it's really hard to decide if the right information is actually coming into my contacts window.
Ash Tilawat: What happens is you have this knowledge base of information that is embedded, and you have to go out and grab all this information and bring it back.
Ash Tilawat: As a result, most of that information is unusable. There's a lot of noise and very little signal. And so it's up to you to decide, hey, how can I make this better? What can I do to ensure that the information coming back actually is more useful?
Ash Tilawat: There's an argument in the market right now that as context windows get bigger, meaning the amount of tokens I can input gets bigger and bigger, bigger, and bigger, do we actually need to go out and retrieve more and more information?
Ash Tilawat: As context windows get bigger, that doesn't mean you can just put all of the information in the world into the context window. You need to be able to delineate and organize what information you're actually putting in, and you need to be able to say, hey, this is the most important information, this is the most critical information, only use this to make a decision.
Ash Tilawat: Just because you have all the information in the world doesn't mean that the output will be better with an LLM. The goal should be to control the information going into the LLM and make it the best possible.
Ash Tilawat: So, rag patterns are sort of…
Ash Tilawat: Increasing in complexity across this different
Ash Tilawat: Table. So we start with naiveRag, meaning we go into our knowledge base and we grab all the information that we have.
Ash Tilawat: Then there's something called metadata filter drag, meaning we can filter that knowledge base based on a title, or a topic, or maybe a different date, or when something was created. And once we filter that, then we're able to just grab based on that information.
Ash Tilawat: Then there's something called hybrid, or, search, which is, like, we do keyword search, then we grab the information, then we try to find what the relevant vectors are.
Ash Tilawat: Or we do multi-hop, which is, hey, I'm gonna do multiple queries on top of RAG, and I'm gonna go out and find all the different information.
Ash Tilawat: And then finally, what we're going to cover today is called Agentic rag, where we put an agent in the middle to decide which pattern is actually the best pattern to use, and as a result, it's a tool the agent uses to then go out and retrieve information.
Ash Tilawat: Now, as we go down this ladder, what we realize, there's more and more complexity into what we're building.
Ash Tilawat: As this complexity increases, the goal is that we have evals in place to make sure that our system is performing at the level it should.
Ash Tilawat: Okay.
Ash Tilawat: Now…
Ash Tilawat: There… let's say you have a 2 million token context window, meaning you can put 2 million phrases, characters, or words into a single context window.
Ash Tilawat: Why is it that we even need RAG at all?
Ash Tilawat: Right? Modern models, like, if you look at some of the Gemini models, their context window is massive. So, if we take all of the information we have, it could be 17 documents, it could be 80 PDFs, it could be a ton of SQL information that is coming back, then all of that is just put into a context window, and then the answer is put out.
Ash Tilawat: What this results in is that every single time we ask that model the same question, we have to put that level of context into the context window.
Ash Tilawat: This gives us no systematic way to decide whether or not that retrieval process that is happening is actually working or not. Because every single time, all we're doing is saying, hey, here's all the information we have, go figure something out.
Ash Tilawat: as we get different queries that are not the same query over and over again, it becomes sort of tedious to gather all the information and ensure that all that information is present when we're actually sending that question to an LLM.
Ash Tilawat: What happens over time is because there's no filtering, there's a lot of noise in that information, and as a result, with all that noise, it becomes hard for the LLM to give you the right answer. And that's why people think that the LLM is maybe hallucinating, or maybe it's not understanding something, or it can't achieve what I'm trying to make it do.
Ash Tilawat: Because what's actually happening in the background is it doesn't have enough information to make a call, or it has too much information that it's getting lost in the sauce.
Ash Tilawat: So, what I'm trying to showcase now is the…
Ash Tilawat: you want to promote your LLM from just being, hey, give me all the information, and I'll give you the answer, to being, hey, let me go find all the information myself, then give all that information back to myself, and then give you the answer.
Ash Tilawat: That is a huge distinction, because what we're saying is.
Ash Tilawat: Let me figure out what context is actually relevant, and what context is actually unnecessary.
Ash Tilawat: And so what we're doing now with Agentic rag is we have a query that comes in, and the LM decides, hey, this means I need to go do some sort of SQL query. Oh, this means I need to go look at this document. Oh, this means I need to go do this database. Oh, this means I need to find some other piece of information online.
Ash Tilawat: The LLM gets a decided plan.
Ash Tilawat: Now, for those in the room that have used, maybe, Cloud Code, Cursor, we know that there exists a plan mode, where you can have a step-by-step, hey, do this first and do that, a sequence of tasks that it can do.
Ash Tilawat: When it comes to Gen Grag, what it's doing is it's creating that plan.
Ash Tilawat: It's that Ash is asking me the specific question. This is my plan to get the information required for that question.
Ash Tilawat: Then I'm gonna organize that information and find signal within that information, then send it to be…
Ash Tilawat: That changes the paradigm, and this is what sort of unlocks a lot of things in…
Ash Tilawat: When we think about work today, when we think about companies today, they have so many fragmented sources of data.
Ash Tilawat: Right? You have information in Slack, you have information in Gong, you have information in HubSpot, you have information in some SQL database.
Ash Tilawat: What Agentic RAG lets you enable is it says, hey, I have all these data sources, I'm gonna figure out, of all the data sources I have, the understanding of that data source I have.
Ash Tilawat: What is the plan for me to find the right information to get the right answer? I'm gonna go get that information.
Ash Tilawat: And then I'm going to organize that information to get the right answer. This is exactly what Claude Code, Manis, Cursor are doing in the background. What they're doing is, they're doing agentic rag and figuring out, hey, maybe I should do keyword search across my codebase. Hey, maybe I should go search a document online.
Ash Tilawat: Hey, maybe I should go look at our HubSpot instance and see what's happening. Hey, maybe I should go do some SQL queries and see what information comes out of that.
Ash Tilawat: But when you transition the LLM from just being an answer engine to the actual retriever, that is what Agentic rag is.
Ash Tilawat: Now, Agentic rag drives the entire workflow, meaning at the end of the retrieval process, when it's gone into these different sources, if it feels that it doesn't have enough information, it can keep going.
Ash Tilawat: Which is a much smarter approach than just saying, hey, take all of the information in the world and throw it into a context window.
Ash Tilawat: The approach here now is, hey, first do a single retrieval and go find all the information you think is necessary. Once you've found that information, do an assessment based on this prompt, based on these criteria, based on this acceptance criteria, if this is correct or not. If you need to go out and do another search, you can do that as well.
Ash Tilawat: As a result, what this lets happen is more detailed.
Ash Tilawat: Faster responses that are actually grounded in real documents, real data within a company.
Ash Tilawat: Now, I have an example for you on this slide, which is a support agent.
Ash Tilawat: A support agent, let's say, gets a question, can I get a refund?
Ash Tilawat: In normal RAG, what would happen is, it would just say, of course, let me route you to our support staff.
Ash Tilawat: But with Agentic Rag, what we can do instead is we have a connection to the database that houses all the orders. We have a connection to the app experience so we can show UI elements on top that show maybe the photo, or where the order was dropped off, or the location where the order was going to.
Ash Tilawat: Also, we have the refund policy, we have the 30-day cashback policy. All of this information inside of company that we can go have the agent retrieve.
Ash Tilawat: So the agent receives this question as a support agent. Can I get a refund? It's able to understand who's asking that question, maybe why they're asking that question, and maybe what that question entails. Decide, hey, these are the 3 or 4 data sources I need to go grab.
Ash Tilawat: Grab that information inside of those data sources, and then, as a result, give an answer back.
Ash Tilawat: Compared to just give a cookie-cutter answer back to everybody asking that question, and then eventually route it to support staff.
Ash Tilawat: The future of AI is agentic, meaning
Ash Tilawat: the AI needs the ability to do the connection, to grab the data, to make the plan, to do the sequence of task. And once it has those connections, those integrations, then it can start doing work for you.
Ash Tilawat: Okay, what the agent actually can decide in a Gentic rag.
Ash Tilawat: So…
Ash Tilawat: inside of Agentic Rag, we can actually say, hey, I have a question coming into my chatbot that I actually have all the information for. I don't need to go retrieve any data at all.
Ash Tilawat: So, for example, if we have something like
Ash Tilawat: a math problem, or a random question that is coming into your chatbot. Or, for example, it's like a support chatbot, and it's asking about very simple information that is already on the UI.
Ash Tilawat: As a result, we don't need to go out and retrieve and grab this information, because the agent already has everything it needs in its training data to give an answer.
Ash Tilawat: So, even deciding whether to retrieve is a decision the agent can make. What is the refund window for a B2B product? Hey, I need to go grab that from HubSpot, I'm gonna grab that information, I'm gonna answer your question. But what is 15% of $4,000? I don't need to go anywhere, I can answer that question directly.
Ash Tilawat: Oftentimes, what you find is, there was a story for, like, this chatbot that an airline had, where the people were trying to ask it to change its prompt, and maybe, like, give it a refund,
Ash Tilawat: and basically convince it that it deserved a discount so that they can get a discount automatically on top. And so what's happening there is people are asking questions that are outside the purview of the prompts that were originally designed for that chatbot, and so people can, you know, maybe get discounts, or maybe convince it to do something for them.
Ash Tilawat: And so we create something called a fail-safe there.
Ash Tilawat: Where it's saying that the agent can decide, hey, this is a dangerous question. Hey, maybe there's SQL injection going on. Hey, maybe this is a question I shouldn't answer. Hey, maybe somebody's trying to trick me. Hey, maybe this is a discount I shouldn't offer. So, deciding whether to retrieve or not is actually something that the agent can say, hey, this is a good idea or a bad idea.
Ash Tilawat: The next decision that you can make in Agentic rag is, what sources should I retrieve from?
Ash Tilawat: A company uses 7, 8, 9, 10 pieces of software, all with data stored inside. With Agentic Rag, you can decide, hey, do I need to even go to this software? Hey, can I, do this without this other piece of software? You can have a sequence of tasks back and forth as a result, so that you can make
Ash Tilawat: The perfect plan to grab the perfect amount of data so that you don't have to waste time grabbing data from all sources.
Ash Tilawat: So, it could be entity type, it could be metadata filtering, it could be routing based on different query types. The goal here is, as… because this is a more smarter data analyst, it understands that sometimes I need some sources, sometimes I need all the sources, or sometimes I just need a few sources.
Ash Tilawat: And the agent can decide when to stop. Now, this is something that happens in ChatGPT all the time.
Ash Tilawat: Where we are just, like, putting a ton of information into our little chat window right there. And over time, that information compounds again and again, again and again, and then eventually ChatGPT starts summarizing all that information.
Ash Tilawat: Because there's just so many tokens being used, so much information flooding the context window, so then the output is really hard to judge.
Ash Tilawat: In this scenario, having an agent in the middle of retrieval and the initial query lets us decide how much information do we actually need.
Ash Tilawat: What that sort of does for us is, without us being involved as humans, we can go out and get that information and come back.
Ash Tilawat: We can run it again and get more information and come back. We can eval it, we can score it, we can make sure it's the right information. And that is what changes the game from this just being a chatbot that we're answering to something that is actually doing work for us.
Ash Tilawat: Okay, so under the hood, what is happening? Agentic rag just means that we've created an agent, and an agent has tools, and those tools let it do research.
Ash Tilawat: An agent is just a loop that is running and calling different functions for you to actually accomplish tasks. In this scenario, if we have an agent inside of our RAG pipeline, what we've done is we've given it a tool for vector search.
Ash Tilawat: for metadata or keyword search, for structured data like SQL, or an external API connection.
Ash Tilawat: These tools correspond to the different data areas that we have. It could be SQL, it could be Slack, it could be HubSpot, it could be documents. And what we're trying to do is, it knows to call those tools in a specific order, based on the description of those tools, to actually get the information back.
Ash Tilawat: So, Agentic rag, at the end of the day, is just an agent placed that is available to receive queries. Those queries are then sequenced into a set of tasks that decide between 4 or 5 tools what information to go out and retrieve. Then it's then sent to an LLM to then decipher an answer.
Ash Tilawat: The goal here is we've mapped every single data source as a tool, we've mapped the important API calls as a tool, we've mapped all the metadata and the structured data as a tool, so we can then use this inside of our process.
Ash Tilawat: Now, when I think of an agent, what is the difference between an agent
Ash Tilawat: and a workflow, or an automation, is that the agent is doing the thinking and the doing. It's making a decision, and then it's taking an action.
Ash Tilawat: So the… when you put an agent in the middle of your retrieval pipeline, what you're basically doing is you're saying, now the LLM is going to decide what's next, what's… what tools should be called, what should be retrieved, and whether or not we have enough information.
Ash Tilawat: A really nice way to understand an agent is called the react loop. Reason, act, and observe.
Ash Tilawat: Reason is, hey, I need to get all the information necessary to answer this query.
Ash Tilawat: Act means I'm gonna call 1, 2, 3, 4, 5 functions to get all the information required for that query. Observe means I've called 2 or 3 of these functions. Do I have enough information to proceed?
Ash Tilawat: Am I following my path? Is my sequence correct? And then I can loop back around to reasoning to say, what step should I take next?
Ash Tilawat: So when people explain agents, oftentimes it becomes confusing. Like, what is specifically an agent, what is specifically a workflow, what is maybe just a chat interaction? An agent has to think, and it has to execute. An agent has to reason, and it has to act. An agent has to observe, and it has to react.
Ash Tilawat: When you think of an agent, think of doing and thinking, and what we're doing with Agentic Rag is saying, I now have an agent that I've created that understands what I'm trying to get at, what information is required to answer that question, and then actually executes on top of that information.
Ash Tilawat: Okay.
Ash Tilawat: Another thing that happens is, when we interact with AI,
Ash Tilawat: Our questions are really, really vague.
Ash Tilawat: Our questions might be, why is that the case? Fix this error. I don't understand. These aren't inherent questions, and they might have different connotations, or different contexts, depending on what's happening previously in the conversation.
Ash Tilawat: So what Agentic rag lets us do, it takes a hard question and actually breaks it up into a series of questions. So for example, if I ask the question, how's the weather, there's a couple of inherent decompensation steps inside of that question. Where am I located?
Ash Tilawat: What date are… what time is it where I'm located, and how do I access the actual weather?
Ash Tilawat: When you have an agent in place, it's able to decompose that vague and hard-to-understand question into a series of steps, a series of different questions that it may be able to ask to then get to the answer.
Ash Tilawat: So what we're trying to do with Agentic RAG is also decompose tough, vague, and hard-to-pinpoint questions into a series of steps that can actually be answered.
Ash Tilawat: The best example of this would be, let's say we're using a coding agent, and the coding agent comes across an error, or maybe a failure in execution.
Ash Tilawat: It's our inherent instinct as developers to maybe just say, fix the error, find what's happening.
Ash Tilawat: Make sure that this is actually running.
Ash Tilawat: What this results in the agent responding in is basically saying, hey, go out and go check to see the logs. Go out and go check to see the information. Go out and go check to see the code. As a result, the agent is able to decompose the problem so that it can take the problem, get the information required to solve the problem, run that in a sequence of steps, and then go out and solve the problem.
Ash Tilawat: So, Agentic rag lets you enable that on top of a pipeline.
Ash Tilawat: There's a concept of memory here that I think is important, and I want to talk through really quickly.
Ash Tilawat: when you have an interaction with a chatbot, like Claude or ChatGPT, that interaction is
Ash Tilawat: Defined by the session, meaning that little window in which you're talking back and forth.
Ash Tilawat: And that's scoped to a single conversation, maybe a single task. It has some information, but everything is imprompt, in context.
Ash Tilawat: What happens with Agentic rag and archival memory is that we can store certain pieces of memory as the agent is progressing through the task.
Ash Tilawat: Meaning, let's say it went out and grabbed something from a SQL database, we can quickly store that.
Ash Tilawat: Maybe it went out and actually decided, hey, what are the different aspects I need for this? And made a little state object to store its progress.
Ash Tilawat: The goal here is this second type of memory lets us grab different things across the board, store those things for use later on, and makes it so that our context window, when we're actually searching for all this information, isn't diluted with all this noise and random tokens.
Ash Tilawat: So, that is another thing that Agentic Rag enables, the fact that we can leave a single session, a single conversation, and have disconnected, offloading our memory onto something else.
Ash Tilawat: Now, if we were using Agentic RAG, we can actually run all of our tools in parallel. And if you wanted multiple agents, you can also do that in parallel. Because what we're doing is these are parallel retrieval tasks for data analysis. So let's say we had a question about what is the refund policy here at Stripe.
Ash Tilawat: Describe any changes made to that refund policy, and what steps you might take as a result of those changes.
Ash Tilawat: What we can do is we can decompose those into a series of calls to different databases, different knowledge sources. We can run all of those in parallel, and then come back and say, hey, what is the information retrieved from this? What is the information got from this? What is the information from this? We can bring all that information together, and then have something on the other side.
Ash Tilawat: You want to start with a single agent for your Agentic rag system, but your Agentic rag system can scale horizontally, meaning you can have different sources tied to different agents, and all of those agents can perform in parallel.
Ash Tilawat: Okay, before I get to trade-offs in the eval, I'm gonna take a step back here, and we're gonna take some of these questions that we're getting here in the chat.
Ash Tilawat: We have a question here that says, I've noticed that RAG is embedded into the LLM is outdated, and that a lot of time I need to re-prompt and get the latest information. Has anyone tried maintaining a cloud version of RAG that stays up to date, that a local agent can pull from with the latest info?
Ash Tilawat: Great point. The hardest part about a database is keeping the database fresh, whether it's a SQL database, a vector database, or some sort of other type of information. Now, what I've seen individuals do, for example, when you build a code review agent, they have to refresh that information as soon as different code is merged into a repository.
Ash Tilawat: In that scenario, what individuals are doing is, maybe every week, every couple of days, that code is being re-vectorized, reorganized, and added to a database. You would have to do the same thing for your SQL, clean that data, preprocess that data, and maybe on a cron job every single week, make sure that data is up to date.
Ash Tilawat: Now, if this is in the cloud, what I've seen teams do is run a job every night, run a job every 7 days, so that that new piece of information is then being added directly.
Ash Tilawat: What are other questions?
Ash Tilawat: Aaron answering a ton of questions.
Ash Tilawat: A gente grack basically works like taste in a lot of ways, very interesting.
Ash Tilawat: Question from Valerie is, will the recording be available to watch later? Yes, on our Learn portal, you guys can actually access all of my recordings across all of Night School.
Ash Tilawat: Hmm.
Ash Tilawat: Someone is asking, can we talk more about the hard deterministic checks, since the RAG agent production pipeline will need more of those? 100%. In fact, if you go into my RAG cookbook, I have evals in there, and I'm just about to talk about evals now. If you take nothing away from this session.
Ash Tilawat: The thing that you should take away is that evals are important on top of any production AI system, meaning you need checks or deterministic scores in place to ensure that system is performing really well. So this could mean that every time I get a response out of my RAG system, I can actually score that response on maybe different criteria, and as a result.
Ash Tilawat: evaluate those scores, change and update prompts as a result of those scores, and then make sure that it's performing inline. But I will cover that next.
Ash Tilawat: Someone says the Learn Portal is cracked. Love it. I'm glad… I think at this point, inside of our Learn Portal, we have maybe 7 night school sessions, or 6 night school sessions, a couple of sessions and lectures from Gauntlet, and I think what we're gonna keep doing is making all those recordings available to you. But again, the thing about AI,
Ash Tilawat: is that every single week, there's something new. You know, something is changing, something is becoming deprecated, something is no longer in use. So what we like to do with our live sessions every Wednesday, give you timely, accurate, and hands-on material that you can actually apply that week.
Ash Tilawat: And so, coming to the session live every Wednesday, and then using those recordings as you need, would be my recommendation.
Ash Tilawat: Okay, we'll keep going. If there's more questions in the chat, I'll be spending another 10-15 minutes at the end of the session answering those questions.
Ash Tilawat: Now, the thing about adding agents to any system is that there's trade-offs, and there's evals on top of their performance.
Ash Tilawat: Evals are a deterministic way to ensure that an agent or rag, or graph is actually performing at the level you think it should.
Ash Tilawat: Evals are, by far, are becoming the moat for some of the best agents in the world. Your harness, your eval, is the reason why your system performs better than everybody else's system.
Ash Tilawat: So, when you think about Agentic rag, the first thing you need to think about is latency.
Ash Tilawat: How long does it actually take to give an answer back?
Ash Tilawat: Because if an agent is gonna go out and grab all this information, it might take
Ash Tilawat: 10, 15, 30 seconds, or maybe even 2, 3, 4 minutes to get that answer back.
Ash Tilawat: I will say that if you think about deep research inside of OpenAI's ChatGPT, deep research can take
Ash Tilawat: 15, 20 minutes. And if you run that over and over again, it could be even longer.
Ash Tilawat: Oftentimes, people don't always want the answer right away. What they want to maximize for sometimes is accuracy. So if it's something more research-heavy, more accuracy-driven, then a larger time horizon is expected.
Ash Tilawat: But, if it's simpler, easier to retrieve, just an API call, then that information can come back directly.
Ash Tilawat: What Agentic Rag lets you do is have tools across the board for any of these scenarios.
Ash Tilawat: I was working with a company that was building a SQL agent, it was a moving company, and they wanted this agent to be able to get information and analysis back, but at the same time, do actions on top of their tools, meaning schedule a driver, schedule a task, make sure, mark a task as complete.
Ash Tilawat: Or, if they wanted to have their weekly numbers and a really nice report generated, then they can go out and grab that information as well, inside of the same agent.
Ash Tilawat: But the expectations of time from one area to the next are vastly different.
Ash Tilawat: For example, if the question is, how many moves are we scheduling this week? It could be very little. If the question is, how many moves are we scheduling over a month, that could be 10x, 100x.
Ash Tilawat: If the question is, hey, can you just go update this on the task? Then hey, I'm gonna go update this and come back.
Ash Tilawat: The goal here is you need to understand what the user is asking, and what time is required for that task to be done. Set that expectation up front, and then have an eval on top, measuring how long it takes to answer different types of queries. The goal should be that you land in this area.
Ash Tilawat: Anywhere from 1 to 3 seconds, or maybe even telling the user to go away for a little bit to run something completely async and come back and check.
Ash Tilawat: You have enough time, you can be transparent with the users, but this is the first eval I would 100% add. The latency and the latency as a result of the thoroughness required on the other side.
Ash Tilawat: Fuzzier deterministic, meaning Oftentimes, if you let the LLM
Ash Tilawat: just assume a lot of things, understand your schema, maybe, understand the different documents associated with this, then the LLM is using its context window and its understanding to actually make all the decisions when it comes to retrieving data.
Ash Tilawat: You can have deterministic ways of grabbing information. Instead of letting the LLM decide which API calls you should make in sequence, you can make a tool to just call those three API calls in sequence and grab that information.
Ash Tilawat: This is the difference between being fuzzy or deterministic. Oftentimes, as engineers, we want to be deterministic and say, hey, this tool retrieves this type of information, so I just need to make these three API calls available.
Ash Tilawat: That's important. Not all the time do you need to hand off thinking to an LLM.
Ash Tilawat: If you do decide to hand off something to an LLM, then what the LLM is doing is taking that information in, digesting that information, understanding it within the context of your RAG system, and then deciding what to do with that RAG system going forward.
Ash Tilawat: when you're designing this, you want to have evals in place that are deterministic, are meant for deterministic tools, and fuzzy tools. If it's deterministic, it means that you can have maybe a unit test, or a check on those API routes, or maybe the data coming back could be a certain format, and you can check that format.
Ash Tilawat: Or you can have acceptance criteria for that call. If it's fuzzy, it might be, hey, you can have an accuracy score, you can have a score on top of a rubric, or you can have a score on top of the different tools that are being used, so that it can evaluate whether or not the thing coming back is even relevant, or give it more information on how to evaluate it.
Ash Tilawat: No matter how you do this.
Ash Tilawat: This is a disconnected tool inside of your AGENTIC RAG system, meaning you retrieve information with one tool, but you evaluate that information with a different tool.
Ash Tilawat: And the reason for this is you can associate evals with that different tool, and then make it perform really well.
Ash Tilawat: So, something to keep in mind as engineers, or when you're trying to build this, is does this tool need to be fuzzy or deterministic?
Ash Tilawat: can I just retrieve all the SQL through a SQL… all the information through a SQL query? Do I need to let the LLM go out and decide what the schema looks like, and actually see what's happening up front?
Ash Tilawat: Evals are non-negotiable, in my opinion, and the way I would start with evals, especially on top of RAG, is thinking about the output and whether or not it's actually grounded in any information.
Ash Tilawat: There's 3 evals to keep in mind for any RAG pipeline, and that's accuracy, Accuracy or precision?
Ash Tilawat: Groundedness, and recall.
Ash Tilawat: I'm gonna start with groundedness.
Ash Tilawat: Oftentimes, people create these rack systems, and they're not actually using any of the information that's being retrieved. What tends to happen is, people create these rack systems, they add all this information to context windows, and then the LLM is unable to discern what information is important, so it just uses its training data to answer the question.
Ash Tilawat: For example, if you took Warren Buffett's investment letters over the last 24 years, and you imported that through a RAG pipeline, added that to the context window, and tried to get the answer on the other side.
Ash Tilawat: Then…
Ash Tilawat: It would be important to make sure that everything that was being answered, you can trace back to an actual letter from those series of letters. But oftentimes, what tends to happen when you analyze this is that the LLM already has some understanding of Warren Buffett's public investment letters, and it just uses that to answer the question.
Ash Tilawat: If that's the case, then you have a poor groundedness score, and that eval is telling you that you need to have a better prompt
Ash Tilawat: more information and remove noise from your RAG pipeline.
Ash Tilawat: The next one is something called precision. Means that if I'm retrieving all this information, and maybe it's, like, 7 pieces of information, then if I'm retrieving 7 pieces of information, of the 7 that I've just retrieved, how many are actually relevant to answering that question? It could be 3, it could be 4, it could be 5, it could be 6.
Ash Tilawat: And the goal here would be to get your precision as high as possible, so you know that your agent is actually retrieving useful information, and not just noisy information.
Ash Tilawat: The last one is… Recall.
Ash Tilawat: Which means that if you have all this information, let's say it's inside of a SQL database, a vector database, maybe it's documents on top of everything that you're trying to do.
Ash Tilawat: Then there's so much information that oftentimes it's really hard to go out and find where something is located. Does the agent… is the agent able to recall where information is? Can the agent locate where that new information is? Does the…
Ash Tilawat: knowledge base that we have, have coverage across all the pieces of information. So this becomes really important to track.
Ash Tilawat: Now, we can also do binary evals on top of just user satisfaction, meaning I get a response back on the other side, did the user like it or not? I can do it on how well different queries are completed, meaning if I have a series of completed queries.
Ash Tilawat: Of those completed queries, which ones were completed successfully, and which ones maybe had incorrect answers.
Ash Tilawat: Maybe latency or time to resolution is my eval. How long does it take me to gather all the information? How long does it take me to actually give out an answer? And then maybe it's the rate at which we escalate to humans. Like, hey, can we go out and give this to a human being after maybe 3, 4, 5 retries?
Ash Tilawat: The core metrics that, again, you need to think about are precision, recall and coverage, groundedness, latency, and cost. For anybody building an injected RAG pipeline, this is where I would start. Again, precision is, of all the information that I've gathered.
Ash Tilawat: How much of that information is actually useful?
Ash Tilawat: Recall and coverage represents, of all of the relevant chunks in my corpus of data, was my agentic rag system able to find coverage and find all of them?
Ash Tilawat: Groundedness means that, hey, I've retrieved all this information, I have all this information from different sources, is that actually being used to actually generate an answer? And then latency and cost refer to the time and money spent trying to actually build out the system and give an answer.
Ash Tilawat: Again, if you build out these evals, they tend to compound. And as these evals tend to compound, you're able to then see the cost.
Ash Tilawat: the… see the accuracy, see the precision, see the recall, and see the groundedness. I would use a platform like Langsmith or Langfuse to track all of these. Langsmith even has an MCP, can actually set all this up for you. And then you can see the scores of your RAG pipeline, see the scores of your agents, and then make improvements to your agent.
Ash Tilawat: Now, there's some terminology in the field now that I won't talk through directly, which is training your agent backwards. What does that even mean? What does it mean to make your agent better,
Ash Tilawat: What does it mean to make your agent better month over month? Execution over execution. It means that you're updating a prompt, changing a model, maybe changing a tool, maybe changing the API or software associated with that tool. These are all steps you can take to make your RAG pipeline better and better.
Ash Tilawat: In terms of production AI,
Ash Tilawat: There's so many demos out in the space. Hey, this is supposed to be a demo for a chatbot to build a website. Oh, this is a demo for a SQL agent to get an analysis back.
Ash Tilawat: These demos are not being tracked and scored and evaluated. The way you make something production is that you evaluate whether or not it's performing correctly.
Ash Tilawat: Is it actually aligning with the goals of the user or the company? Is it resulting in output on the other side? And measure that output in an observability platform, and then change things to make sure those numbers are the highest they can be.
Ash Tilawat: You need to treat this like an experiment, treat this like a data science project, treat this as a way to make your system better over time.
Ash Tilawat: Okay, there's some agent-specific metrics I do want to talk about. One is tool selection, query decomposition, and stop quality, which just means… tool selection means, is it choosing the right tool for the right task at the right time?
Ash Tilawat: what an observability platform lets you do is it gives you production logs of when a tool is called, why that tool was called, and what the context was when that tool was called. So you can actually walk through and see, hey, this tool was called incorrectly, this tool was called correctly, maybe I need to change the prompt, so that that changes and makes it much better.
Ash Tilawat: In terms of query decomposition, you can actually see, hey, this was the question given by the user, these were the questions then created by the agent.
Ash Tilawat: excuse me, then the agent went over and tried to find all the information associated with it, and see if the decomposition of that query into steps was actually done correctly or not.
Ash Tilawat: Finally, stop quality means that, hey, something's not working, means it escalate to a human, we're using way too many tokens, we're just doing retries over and over again. Is there a metric we can track here to make sure it's working properly?
Ash Tilawat: Okay, that is what I wanted to cover. This is the introductory lesson in sort of my Agentic RAG series inside of the Gauntlet program. But again, if you want to keep going, I'm going to walk you through our repository that I think is going to do a really, really good job of walking you through all the different types of RAG.
Ash Tilawat: Again, if you have not seen this, this is an open-source project, it's called RAG Cookbook.
Ash Tilawat: And inside of my cookbook, there are 5 types of rag.
Ash Tilawat: If you go inside, you can see each of the steps inside of a RAG pipeline.
Ash Tilawat: ingestion means that you're actually setting up your knowledge base. You're taking all your data, and you're putting it inside of a specific vector database, or it could be a SQL database, or it could be anything, right? You can retrieve from any type of knowledge base.
Ash Tilawat: This ingestion function is… what it's doing is it's taking information and ingesting that inside of a vector database for you for naive RAG.
Ash Tilawat: Then you can do something called retrieval. So you can track and see, hey, what is the information being given back to me?
Ash Tilawat: You can just run these files one-off, and see, hey, what is the information given back to me, and is that information relevant to the question I asked?
Ash Tilawat: Finally, then you can see generation, where you can say, hey, this was all the information retrieved.
Ash Tilawat: That information was added to a context window, and then an answer was generated.
Ash Tilawat: If you go into my evals folder.
Ash Tilawat: Excuse me. You can see two evals per different type of RAG system.
Ash Tilawat: Groundedness and precision are two I show you here. And if you put this entire repo inside of a coding agent, the coding agent can actually be your tutor. It can guide you through how I set up my retrieval pipelines, how I set up my evals, and how to use those evals on top of what you're doing.
Ash Tilawat: Inside of my cookbook, I cover 5 different types, which are just adding complexity till we reach a gentic rag.
Ash Tilawat: The next one is something called metadata filtered rag, which just means that we're filtering our vector area, our search area, and making sure that we're only looking for, let's say, things are tagged in a specific way.
Ash Tilawat: So let's say I had all of the physics textbooks in the world inside of my knowledge base. I can just use metadata filtered RAG to get the textbooks from 2026. So I don't have to go out and search all of the textbooks, just the ones in the year created in 2026. So this is what metadata-enabled RAG filters.
Ash Tilawat: And inside of the evals, again, you'll get the same latency eval, a precision delta eval. Each of my evals are building on top of each other, so you can actually walk through, step by step, exactly what to do.
Ash Tilawat: Next, after metadata, I do something called hybrid search, where I combine keyword search and regular search with RAG, then we do something called graph rag, and then end up at Agentic rag. I'm using letters from
Ash Tilawat: Berkshire Hathaway, as my source of information, I take these letters, I chunk them up, I ingest them, and then I build that RAG pipeline out directly for you. So, this was an introductory class.
Ash Tilawat: Please drop this repo in any coding agent you prefer.
Ash Tilawat: Tell the coding agent, turn on plan mode in Cloud Code, turn on, plan mode on top of cursor, and have it walk through
Ash Tilawat: exactly what to do, step by step, how I've set things up, why I set some certain things up a certain way, why I set up the eval, and what you can do step-by-step is learn exactly what it takes to go from a basic RAG system all the way to Agentic rag.
Ash Tilawat: I also have some information here that you can use to better set up the repository. The amazing part about the world we live in today is I know there's some people who are in sales, some people in go-to-market, in marketing, I know there's a bunch of engineers in the room as well.
Ash Tilawat: All you have to do is take your coding agent and now use it as an AI tutor, and this repository has everything set up for you to learn.
Ash Tilawat: I want to thank you guys for your attention. This was the end of my lecture, and if you want to keep going.
Ash Tilawat: I'm gonna do Q&A for at least another 10 minutes, and then after that, I'm gonna give you the topic of next night's call.
Ash Tilawat: Alright, Q&A.
Ash Tilawat: Please add your questions to the chat.
Ash Tilawat: And…
Ash Tilawat: Sorry, I'm trying to find more questions.
Ash Tilawat: I have a question here, is that, is the Asian good at long horizon queries? So, I think what's happening, or what the question's trying to entail, is that if you walk over
Ash Tilawat: like, a really hard, deep research task across, let's say, 6, 7 data sources.
Ash Tilawat: You can actually add a skill
Ash Tilawat: on top of anything you're building, and you can actually have it traverse that data, lake, database, exactly the way you want it. So, a nice way I like to put it is, let's say you have, I don't know,
Ash Tilawat: create a HR policy depending on all the guidelines set out in my company. Go into Slack, go into Salesforce, go into our SQL database, get all the information here, and come back.
Ash Tilawat: What you can do is you can take a skill. So, a skill on top of the Agentic RAG agent you're building, and that skill can teach the agent how to traverse all those different data sources, grab information efficiently, and run a task longer than 10 minutes.
Ash Tilawat: So the answer to your question is if something is really, really hard, and really hard to find and search for, then can you go out multiple times into those areas with specific skills and system prompts to get that information correctly, run that task for more than 10 minutes, and actually have a result on the other side.
Ash Tilawat: So, it is totally possible for a long horizon query to be resolved by an agentic rack system.
Ash Tilawat: Alright… Sorry.
Ash Tilawat: Why do you think of using LLM Wiki from Carpathy as a monitoring framework to keep Synced with…
Ash Tilawat: or collaborate with the agents. It feels too far out there, but as a primary actor in control of a workflow, I want to be able to go and kind of log that I can read the agents
Ash Tilawat: It wouldn't be the same type of memory. I want an LM griding away to recall and pull from. It…
Ash Tilawat: is more like a dynamic agent MD with more history.
Ash Tilawat: And have… bring into its context, but I would think it through a system analyst. So, I think carpathy loops are really, really interesting. I think, as humans, what are we inherently doing when we do knowledge work? We use our memory, our experience doing a task.
Ash Tilawat: And take all of that information to decide what information don't we have. Then we go out and do research. We keep doing research, whether that's talking to people, understanding that task from other angles, or maybe finding that research online. Once we have all the information, then we make a plan on how to execute a task, and then we move forward.
Ash Tilawat: I think it's really interesting to think of a Carpathy loop as doing research for us.
Ash Tilawat: And then, when it's done doing research, transitioning to doing tasks for us.
Ash Tilawat: Because what the research part does, kind of similar to Agentic Rag, is grab all of the information across the loop.
Ash Tilawat: decide from all the information that has been retrieved, hey, what should I grab and use inside of the context window, and then import that into some sort of task loop. So, I think it's really interesting, I think it's really early. I think to try this out, what I would do is, whatever…
Ash Tilawat: query I'm trying to solve, whatever information I'm trying to gather, whatever task I'm trying to complete.
Ash Tilawat: Have a deep understanding of all the data sources.
Ash Tilawat: Understand exactly how you would research information across those multiple data sources, in what sequence of tasks, in what ways, and what methodology you would follow. Once you're okay with that sequence, once you understand that sequence, see if you can get AI to do it.
Ash Tilawat: If you can get AI to do that, then you can give it more complex tasks, do it… do research across multiple days, and then as a result, do something that is more meaningful.
Ash Tilawat: So, I think it's interesting, but I think it's early.
Ash Tilawat: Thank you, guys.
Ash Tilawat: Should human-in-the-loop approach be considered at some point? Today, you showed 5. Where would you say this fits in? Great question from Jorge. I think human-in-the-loop fits in…
Ash Tilawat: in every single approach I showed you today.
Ash Tilawat: Now, this was the introduction to Agentic RAG, right? If we were in Gauntlet right now, there would be 3 more sessions on this. And maybe the next session would be deeper into evals, deeper into human loop. But what you're trying to do here is…
Ash Tilawat: You're using your eval as the human-in-the-loop methodology, meaning you can make a rubric to score how well your RAG system is working.
Ash Tilawat: That rubric can be created by a human being, a subject matter expert that actually understands that retrieval process, and then what you can do is see if you get an LLM to grade exactly like a human would. Calibrate the grading.
Ash Tilawat: like the human subject matter expert, and then use the LLM to make your system better over time.
Ash Tilawat: So, where does human in the Loop come in here? I think in all five, it can be used in the eval space. Now, if you feel like an answer coming from a RAG pipeline is inefficient, or maybe it doesn't have all the information, then you can route that question to a subject matter expert. For example, I worked on a tutor.
Ash Tilawat: for this education company, and every time the tutor was unable to gather all the information to answer a question, it was routed in Slack to a teacher. The teacher would then answer the question, we would save that answer inside of a knowledge base, and then we would continue forward.
Ash Tilawat: So, it's always an option to have a failsafe that goes to a human, but human in the loop is always possible inside of your EDAS.
Ash Tilawat: Rajiv is asking, can you spend some time on pointers for senior non-AI engineers that want to get started with it?
Ash Tilawat: Don't dabble, immerse yourselves completely. That means that every Saturday, if you're gonna spend 6 hours on learning AI, do that. I think the big problem that we're seeing right now is there's a lot of people learning a little bit across too many different areas. If you're a senior engineer and you're not AI first, take one topic and go super deep.
Ash Tilawat: That depth is more important than breadth.
Ash Tilawat: It's not that I want you to be perfect at everything, I just want you to be really, really good at a certain amount of areas. And in my opinion, those four areas are RAG, agents, graphs, and evals. RAG, Agents, Graphs, and evals. If I were you, a senior engineer who have a strong technology background, focus on those four areas.
Ash Tilawat: Every Saturday, join Night School, join me, find more insights into these areas, and go deeper on those subjects.
Ash Tilawat: Trevor's asking, but running a long Horizon query gets you token management issues, right? I suppose that would be the next thing to learn. Great question from Trevor. Trevor, you can change the model.
Ash Tilawat: Right? So sometimes, when I'm retrieving something super simple, that's an easy API call, I just use a really, like, a nano or a mini model. That doesn't matter, right? Like, first, that will save a bunch of cost, also usually results in a much bigger token window, so I can do that over time. So you can manipulate the model, you can manipulate the input tokens, output tokens, you can manipulate the system prompt, but the first step would be, you need to have visibility across your entire system.
Ash Tilawat: So using an observability platform, any observability platform of your choice, mapping the entire system so that you can actually see what's going wrong, and then maximizing the models you're using, maybe using dumber models, using models with bigger context windows, to make sure that the tokens are being managed well.
Ash Tilawat: Josh is saying, any advice for building production-grade projects on the side on a budget to help learn Agentic RAG, Agentic AI software engineering? If your current place of work doesn't support AI, you can't interact with to learn the latest AI development practices. Great question from Josh.
Ash Tilawat: Josh, it could just take… it takes a $20 account on Cursor for you to get started. If you really want to do, save money here, I would just get Open Router, where you can, put in tokens and money directly, and actually set that limit, and use something like OpenCode, or some sort of open source, coding agent that connects to…
Ash Tilawat: an API.
Ash Tilawat: And your goal would be that you want to go down…
Ash Tilawat: Each of these thought processes, how to set up a RAG pipeline, how to set up an agent, how to set up, graphs, how to set up evals, and you want to do this using the most popular tools in the market right now. If it's agents, it would be the AI SDK by Vercel, the Claude SDK by Anthropic, the OpenAI Agents SDK, or LangGraphLang Chain.
Ash Tilawat: and you want to be able to go down these four different areas, you want to be able to just use that, you know, $20, $30 budget, so you have just enough tokens, use a dumb model. If you're using a dumber model, it forces you to think, it forces you to prompt better, and it results in you actually learning the process and being able to build something yourself.
Ash Tilawat: I also want to say, as we're getting to the end of the session, that if you're interested in becoming a challenger for Gauntlet.
Ash Tilawat: Cohort 5 starts on Monday, and we do this every week. This was an introductory class. We would go forward with doing two, three more classes after this, deep into the subject, completely immersing yourself in AI.
Ash Tilawat: It's 10 weeks, 3 weeks remote, 7 weeks on-site, and what we want to do is build the biggest, and the best, and the most elite community of AI builders in the world here in Austin, Texas, and if you want to be a part of that, we would love to have you.
Ash Tilawat: Okay.
Ash Tilawat: Next question.
Ash Tilawat: Questions for Kyle. Kyle's asking if he's having some trouble with his application, what should he do? You could just do support at gauntletHQ.com, and that should be able to answer your question.
Ash Tilawat: Kieran is asking, what are the most common gaps in the Gentic rag during eval versus during production?
Ash Tilawat: So, what happens is when you set up evals for Agentic RAG, what you do is you build a golden set with your product manager, maybe with your subject matter experts, by having conversations with your users.
Ash Tilawat: And that golden set is only a subset of information that actually accounts for all the queries that are going to come in. Meaning, you built your system for 300 possible test cases, but the actual test cases are completely different, or maybe there's 600, 1,000 test cases. So the difference between production and development is, you…
Ash Tilawat: Haven't done enough research, user testing.
Ash Tilawat: user queries, interviews, to figure out what are all the possible questions and queries I'm going to get in my Agentic RAG system, and I haven't built out different edge cases associated with those. So, the difference in production is there's so many unexpected questions. The questions you thought were going to be asked, or never asked.
Ash Tilawat: And what tends to happen is you want to launch as soon as possible, and you can do it as a beta. You can do it and say, hey, we're only going to give this to 5 customers, and we're going to do it as a beta. Because when customers actually start using it, you realize your training data is actually way more than you thought, and all the queries that you had to cover are way more than you thought initially.
Ash Tilawat: Lauren's saying, how do I sign up for next week's session? I've been running into deployment issues with deploying agents on Hugging Face and Railway.
Ash Tilawat: Great question from Lauren here. It's every Wednesday at the same time, but what we're gonna do is we're gonna send out the link right after the session for the next session, so you can sign up. Lauren, I would argue if you… not argue, I would recommend that you connect the Railway plugin on top of your Cloud Code instance.
Ash Tilawat: This should really help in deploying agents. If you deploy them, if you already compose them inside of a Docker container, the Docker container is sort of a sandbox for you to test everything out, make sure everything's working properly, and then it becomes really easy to just host that Docker container wherever you want.
Ash Tilawat: I would recommend using the Railway plugin on top of Cloud Code, it makes things really, really easy.
Ash Tilawat: Or using Agent Core on top of AWS, if this is something that you want to ship to production much faster.
Ash Tilawat: Johnny's asking, can you repeat that, the Claude SDK? If there are 4 frameworks I would learn, if I were you, and this is what we focus on here at Gauntlet as well, the Claude SDK by Anthropic.
Ash Tilawat: the AI SDK by Vercel, the OpenAI Agents SDK, and Langchain Langgraph. These are the different areas that are really popular in the market right now. We tend to plant our flag in the areas that our current partners are actually interested in, and actually go deeper on those specific subjects.
Ash Tilawat: Question from Rajiv is, what Saturday school are you talking about? There's no Saturday school specifically, I was just saying you can use Saturdays to actually upskill yourself in AI. I just want you guys to…
Ash Tilawat: go all in, and actually dedicate an entire day to doing this. Because oftentimes what happens, and I think that what's gonna happen for most of you in this room, is you're gonna watch me for a long time, and none of you are gonna take action and apply the things I'm telling you to do.
Ash Tilawat: Instead, if you really want to break into AI engineering, if you want to become an AI-native software engineer, you have to immerse yourself. You have to go all in. You have to dedicate a day, two days, to doing so. That's why at Gauntlet, we are so…
Ash Tilawat: We are big believers, and we think it's so important that immersive education in person, in an environment where you're surrounded by like-minded, intelligent engineers, is important to taking you to the next level, and that's how you can definitely stand out.
Ash Tilawat: And take it to the next step.
Ash Tilawat: I have a question from people who are saying they've applied, but don't know what's happening. If that's the case, please email support at gauntletHQ.
Ash Tilawat: Question here is, is there a cohort more focused on applied AI or frontiers in AI research? Great question! Our cohort is more focused on applied AI. All of our hiring partners are looking for software engineers that are going to add AI features or build AI product.
Ash Tilawat: We're not training models, I mean, we use a little bit of fine-tuning, we teach reinforcement learning, but our goal is, again, applied AI, making sure that we're taking the existing systems and taking them to the next level.
Ash Tilawat: Alright, I want to thank everybody for joining Night School today. If you're interested in watching a recording, please check out our Learn Portal. The Learn Portal is going to be your stop for our recordings, the dates for the next Night School session, and more free content for you to upskill yourself.
Ash Tilawat: If you're interested in going all-in and becoming an Gauntlet-trained AI champion, then please consider applying to Cohort 5, and after that, Cohort 6, and after that, Cohort 7, because we're going to be doing this for a long time. You can do that at apply.gauntletai.com.
Ash Tilawat: And our goal is to build the best elite community of builders in the world, and I think there's some of you in the room right now.
Ash Tilawat: I want to thank you for your time, I want to thank you for your attention, and I'm excited to see you next week at Night School.