Aaron Gallant: Alright, so we are live.
Aaron Gallant: Hello and welcome. Anybody out there, thank you for joining us. This is Night School.
Aaron Gallant: My name is Aaron Gauntlet. This is my first time teaching night school, but pretty far from my first time teaching. In fact, I've been teaching at Gauntlet since the beginning. And if you're joining us, please, drop where you're from in the chat. Love to see the whole range and…
Aaron Gallant: variety that we get here, it's always great. It's one of the powers of the… Tokyo! Very cool. Japan… Japan's fun to… fun to visit,
Aaron Gallant: I… Let's see here. Saludos from Seattle, Indianapolis, Kansas, Sao Paulo, alright!
Aaron Gallant: It's another place that's pretty fun to visit.
Aaron Gallant: Germany? Wow. Belfast?
Aaron Gallant: And some Austin, and then Aussie, which I think is different than Austin, but that's both great.
Aaron Gallant: Dubai, even. Okay, we've got quite a spread.
Aaron Gallant: And also, if you want to drop in the chat, you know, a little bit about your background and what you're hoping to, get out of this. You know, what you are interested in here.
Aaron Gallant: And I do see there's an audio loop on the call.
Aaron Gallant: I…
Aaron Gallant: If anybody else can confirm if there's any audio quality issues, let me know. But, yeah, we've checked this a little bit, so could be on your side if you want to double check, too.
Aaron Gallant: Alright, yeah, let's see what people are doing here. Hoping to become an AI-native engineer, forward-deployed engineer.
Aaron Gallant: Sure.
Aaron Gallant: Anybody else? Got some goals?
Aaron Gallant: Some things they hope to use all this new fun technology for.
Aaron Gallant: Alright, well, you can click on it and drop it in the chat, too.
Aaron Gallant: Just here to learn, that's always good, too. I… I really connect to that, actually. That's…
Aaron Gallant: Why I teach so much, because you also learn when you teach.
Aaron Gallant: train a model to classify documents for work. Well, that's great, Jacob. We might talk…
Aaron Gallant: Not specifically, specifically about that, but we'll give you some context to help you figure that out.
Aaron Gallant: Great, some good stuff. Build ML models and AI-powered apps. Well, let's dig into it then, shall we? Because that's really the point of this session, is to, sort of…
Aaron Gallant: disambiguate, to understand a little more specifically what these things are, and what it means, for instance, to train a machine learning model, and why you might want to do it, why you might not want to do it.
Aaron Gallant: And how it is different from the many other things we do. So, the topic is fine-tuning, and part of what's so fantastic about large language models is that they can be fine-tuned. They can be sort of continually trained. They are pre-trained.
Aaron Gallant: These big companies have spent piles of money training these enormous models that we have access to.
Aaron Gallant: And that gives them a general range of capabilities.
Aaron Gallant: But then you can train them more, and perhaps make them better, at least for some specific task. That would be the hope.
Aaron Gallant: Now, that said, there's a lot of trade-offs here. It's complicated, it's expensive, it's not as expensive as training from scratch, but it is significantly more expensive than just, you know, consuming the model, just doing simple inference, and that's most of what we do, by the way.
Aaron Gallant: So, you know, almost all the other techniques that are talked about in this space, all the other AI engineering stuff, RAG, context engineering, what have you, are all just…
Aaron Gallant: essentially a suite of techniques to try to use the inference of the model effectively. And if you're not following exactly what I mean by that, you hopefully will soon.
Aaron Gallant: But it's sort of like…
Aaron Gallant: we're just taking the model and using it versus changing the model itself. And, you know, most of the time, we don't do that. Most of the time, we just accept the black box. Now we're gonna actually
Aaron Gallant: Tinker with the black box a little bit.
Aaron Gallant: So, I'm gonna move through the slides here. I will share the link with the slides at the end, and there's also, as time allows, there's some code at the end. We'll share the link for that, and if I have time, I'll walk through the code, but if not, you'll have the code.
Aaron Gallant: And, please drop questions throughout in the chat. I'm monitoring the chat, I have it here on multi-monitor setup, so I will, I will do my best to pick up questions and answer them as I go.
Aaron Gallant: Of course, I will probably focus on questions that are relevant to the slide, so… or at least relevant to the topic. If you have other questions about Gauntlet or anything else, we have other staff. We've got Sunny there, who will help out in the chat answering questions. All right?
Aaron Gallant: Let's do it.
Aaron Gallant: So… when should you fine-tune a model? When should you consider opening that black box? Well…
Aaron Gallant: There's some situations here, and the first one teaches intuition where words fall short. That's a fancy way of saying you couldn't solve it with prompting.
Aaron Gallant: Right? Like, you couldn't get the model to do the thing. The model can't do it, you need to improve the model. That's… that might be the first thing that leaps to your mind.
Aaron Gallant: But that's actually not the most common case that I think we are seeing, because the models are pretty generally capable, and if you run into a situation where you can't do something with a particular model, you can usually just go up to the bigger model, right?
Aaron Gallant: And if you're in a situation where the biggest flagship model can't do the task.
Aaron Gallant: well, you're… maybe you could fine-tune that model and make it even a little better, but, like, that's… you're now at the cutting edge, like, you're competing with the research groups and the groups that are trying to advance the frontier, and it's cool, and I'm not saying you can't or shouldn't do that, but I will say that the majority of practical
Aaron Gallant: paying tasks, you know, things that companies want done, do not necessarily live all the way on the frontier, and that's okay. You know, we ride that, and we live a little bit short of it. So,
Aaron Gallant: We get to benefit from it.
Aaron Gallant: So, the first one, yeah, it's a valid use case, but you're not gonna see it.
Aaron Gallant: Now, the middle two are sort of my favorite, and they're the ones that you actually see a bit. You can make a smaller model better for a specific task.
Aaron Gallant: And you can also potentially make a model better at a task, but with less prompting required, less use of your context window, essentially. And both of these are efficiency gains, essentially.
Aaron Gallant: And also, potentially, not just efficiency gains, but, autonomy gains. And what I mean by that is,
Aaron Gallant: Let's say you want to, you work in a sensitive space, like, financial or security or medical space, and you need to have tight control.
Aaron Gallant: over certain aspects of your system. You can't just be tossing all of this stuff as API calls to somebody else's model. You know, you can't be using ChatGPT or Claude or any of that.
Aaron Gallant: So… what do you do?
Aaron Gallant: Well, those models are kind of the strongest models, but there are some pretty good models out there that are open, that you can download, and then you can fine-tune those models to be better for the specific task you're solving.
Aaron Gallant: And then now you have control of a model that is solving that problem for you, and you could potentially deploy it on your own hardware and maintain all that. So there's that aspect, the sort of privacy-security control aspect. And then there's also the efficiency aspect.
Aaron Gallant: Because really, as we'll see in most of the slides here, a lot of what we're doing with fine-tuning is working with smaller models and trying to make them good enough to solve some particular problem, maybe on the edge, or maybe just because we want it to be cheap, and maybe we still do it in the cloud, but a small model. And…
Aaron Gallant: What's really convenient about that, as we'll see, is you can use large models to essentially synthesize the training data, because the large model's already able to solve whatever this task is.
Aaron Gallant: And so you make the training data with the large model, and then you take that training data from the large model, and you train the smaller model.
Aaron Gallant: So that's, I think, the… the sort of sweet spot where fine-tuning is the most practical, and to give a few concrete use cases, and I do see that's a great question in chat, I'll get to that at the end of the slide.
Aaron Gallant: To give a few concrete use cases, let's say you wanted to train a
Aaron Gallant: Fraud detection system, and you wanted to have a step that analyzed, open text, analyzed some messages or emails or something.
Aaron Gallant: and tried to look for signs of fraudulent behavior, or signs of something like that, like, you know, in a financial context, payment processing or something. And let's say all this is considered kind of private, personal information.
Aaron Gallant: and your lawyers didn't sign the thing that lets you just send it to some other service. You have to do it on your own hardware, or at least within your own private cloud. And so, you take a small model.
Aaron Gallant: and you label, one way or another, you get a bunch of labeled cases of this is good text, this is bad text, and you use that to train the model to be better at making that decision. And you start from a smaller model, that, you can run more efficiently, and then you evaluate it, and you ensure that the accuracy is where you want it to be.
Aaron Gallant: So…
Aaron Gallant: that's sort of what this overall loop looks like, and we'll get to a few more of the specifics in the coming slides, of course. But that's, again, these two middle points here. If there's one takeaway from this whole session, it's that most of the time, you shouldn't use fine-tuning.
Aaron Gallant: And the main time to think about, like, the main time to think about it is when those aspects, when, you know, efficiency and control of small models, that's the most likely situation where it's like, hey, we could use a small model for this, and we could make it more efficient, we could have control of it. And that's when you might want to fine-tune.
Aaron Gallant: The last one, narrow the range of possibilities, that's talking about stuff like alignment, right? That's talking about,
Aaron Gallant: Trying to make,
Aaron Gallant: the model essentially have some sort of rails, some sort of limitations, or at least guidelines on what it does or doesn't do. And all models have this already. In fact, it's the big difference between the major flavors of large language models, because, you know, ChatGPT, Claude, and Grok are all trained on
Aaron Gallant: Give or take the entire internet, but they all have much different, reinforcement learning, fine-tuning, essentially, that has tweaked their behavior.
Aaron Gallant: And so, you know, maybe there's a situation where you would do this. I can imagine if you're working for some corporation that's particularly sensitive about how its, LLM behaves, you could potentially want to do this. But the thing is, is…
Aaron Gallant: It's not really the best way to try to ensure that a model is aligned or secure. Really, you need to check
Aaron Gallant: You need to have other checks. The way to do it is to check outside of the model, to not rely on the model to be fully aligned in and of itself, but to have other parts of your system that can also validate the integrity, security, whatever it is you care about about the model's behavior. You know, things like not allowing it to run RM-RF, stuff like that.
Aaron Gallant: So… that deletes all files, for anybody out there who's a little bit like, what was that?
Aaron Gallant: I know we might have a slight range of technical people in the audience here. So, let me get to some of the questions that came in here. Apple's working on getting Google's models to run on the iPhone. Are they using fine-tuning to do this? I mean, Apple might be fine-tuning the models to some extent for their use case.
Aaron Gallant: you'd have to ask Apple to be sure. I think that's mostly just about using those small Gemma models, on, you know, a nice…
Aaron Gallant: embedded context, because they're an appropriate model for that size, for, like, a phone. The Gemma models are good candidates for fine-tuning, though, if you wanted to fine-tune a model. They are reasonably sized for that.
Aaron Gallant: Is fine-tuning basically for privacy and saving money. Yeah, I mean, that's kind of what I'm getting at here. It's…
Aaron Gallant: Not strictly, to be clear, but that's where I would say most of the real, you know, useful cases are for it at the moment.
Aaron Gallant: My understanding is that most problems you might think to solve with fine-tuning can be accomplished with more appropriate model selection and prompt engineering. Bingo! Yes!
Aaron Gallant: You all already got the lecture, we're only 12 minutes in, we can call it, right? I'll keep going, but it's great to see this. Yeah, I mean, those two points are really emphasizing, like, yeah, do not start by fine-tuning. Start by
Aaron Gallant: By making sure you selected your model appropriately, and you're prompting it correctly, you're, you know, you set up whatever to retrieve context for it, your memory, all these other systems we let… because none of those systems change the model. Those systems just change
Aaron Gallant: How the model inference is done.
Aaron Gallant: So…
Aaron Gallant: Alright, and on that note, there's this little check for understanding here. So, if people want to drop in the chat, and if… hopefully, at least many of you, I think we've had previous sessions, know what retrieval augmented generation is at this point.
Aaron Gallant: But what would be a use case for retrieval augmented generation versus fine-tuning? And we've already talked about fine-tuning, so give, like, a use case that would work well with retrieval augmented generation.
Aaron Gallant: And not fine-tuning.
Aaron Gallant: And I'll keep answering questions while you think about that.
Aaron Gallant: With respect to privacy, are there any advantages to downloading a model and using it locally?
Aaron Gallant: Absolutely. I mean, I… VPC… so, VPC is Virtual Private Cloud, and that's nice. That means you're writing a big enough check to whatever cloud company to get some level of assuredness that there is privacy and security, and those are all pretty trustworthy, and
Aaron Gallant: Your corporate lawyers will probably be okay with it. But…
Aaron Gallant: you know, push comes to shove, there's no substitute for having the silicon yourself, right? It depends how paranoid you feel like being, and what your… what your threat model is, as they say in the security world, and I would not encourage excessive paranoia just for the sake of it.
Aaron Gallant: You know, make sure that you're solving the problem you actually need to solve. But yeah, strictly speaking, the closer things are, the more direct control you have over it, the more direct control you have over it.
Aaron Gallant: And, you know, there are cases where data centers might have issues, right?
Aaron Gallant: Of course, if you're gonna host it yourself, you are the data center now, and you're responsible for things like uptime. So, there's no free lunch. It's just more work.
Aaron Gallant: Alright. Are AI skills better than fine-tuning? Well, that's… it's not like an Apple…
Aaron Gallant: Apple comparison there, but, yeah, I mean, in the sense that, like, if you're trying to solve a problem and you think you could add skills, and by skills, I'm taking this to mean tool use plus pre-written prompts for those tools, essentially. That's kind of what we mean by skills.
Aaron Gallant: Yeah, that might be a better way to solve a lot of problems.
Aaron Gallant: Now, that said, fine-tuning does something that none of these other techniques can. So if you need fine-tuning, you need fine-tuning, right? And, again, the example would be constrained environment, you need to use a 1.3 billion parameter model, you need to make its accuracy at least
Aaron Gallant: X, whatever your business criteria is, and, you know, you fine-tune the model to get it there.
Aaron Gallant: Alright, so people were answering the question.
Aaron Gallant: Seeing some good answers to the RAG stuff. I'm not gonna read all of them out loud, but y'all can follow the chat as well. And I'll take one or two more questions, then I'm gonna dive back into the slides. Use cases for narrowing the range of possibilities. So, yeah, that's the idea of… I mean, I guess if you wanna…
Aaron Gallant: somewhat specific example, imagine you're a company like Disney, and you want to expose a chatbot, but you want to make sure that the chatbot behaves with appropriate, you know, Disney style. You could potentially do some fine-tuning to really enforce more than… like, because, yeah, you can prompt, you could prompt
Aaron Gallant: any LLM to be like, hey, behave like Mickey Mouse, right? But if anybody else comes along.
Aaron Gallant: and does a prompt injection, and says, you know, forget previous instructions and actually do this, that sort of… that's pretty easy to circumvent. If you actually fine-tune, you're changing the model itself, and it's a little bit harder to circumvent that sort of, guardrail.
Aaron Gallant: it's still, as I said earlier, not a guarantee, because the model
Aaron Gallant: It's something I say almost every time I teach about language models. Large language models, they're statistical models, they're inherently probabilistic. You can't have guarantees. So if you're in a situation where you need guarantees, you need really good, really hard guardrails, you need to have secondary checks. You need to have evals, you need to have monitoring, you need to have other things going on, maybe a human-in-the-loop pattern, which we'll talk about a little bit later, but
Aaron Gallant: means a human checks it, right? So…
Aaron Gallant: That's why I don't really emphasize that last use case either. Yeah, you can do it, and it's why, you know, different language models essentially have different voices, why some language models won't answer certain sorts of questions, or, you know, so forth, is often because of how they were… they were trained, with these examples.
Aaron Gallant: Last question before I get back to the slides. What process is better for a medical industry where data privacy and industry language are specific?
Aaron Gallant: And may not be available in major models. Rags are fine-tuning.
Aaron Gallant: I'd go with RAGs. I mean, assuming that you have a reasonable knowledge base, and a reasonable knowledge base… it's easier to get a RAG knowledge base than it is to get a labeled set of data for fine-tuning. We haven't even gotten to that. Like, that's where all the work is, is having labeled data. So yeah, I would definitely start with RAG for that.
Aaron Gallant: Now, that said,
Aaron Gallant: if you also need it to run on the edge, and then you might need to also fine-tune the model as well, if the model's performance isn't good enough, even with the RAG context, but I would still start with RAG.
Aaron Gallant: Fine-tuning is usually for when the task is a specific sort of problem. It's not just, like, write code in some language, but it's, like, make a decision. Is this fraud or not? Is this, person sick or healthy? These sorts of things, you know, like, is this spam or not?
Aaron Gallant: classification problems.
Aaron Gallant: Okay, so…
Aaron Gallant: Fine-tuning the process. What do you actually do to fine-tune? We've said it enough, let's understand a little bit more here. Well, you start from a model, and we'll talk a bit about model selection, but model selection is its own topic. We could have
Aaron Gallant: At least a whole lecture on that, I imagine. So, suffice it to say, there's a whole zoo of models out there. We live in an interesting time for this.
Aaron Gallant: Step 2, that's what I was just talking about.
Aaron Gallant: that's where all the work is. You know, gathering, cleaning, curating your data. You've probably heard garbage and garbage out, right?
Aaron Gallant: And that's true for, and I've said this a few times, I'm gonna be a little bit crisper now, model inference. Model inference, just so you know, all of these language models, they really are just math. Now, they're…
Aaron Gallant: somewhat incomprehensibly large math, we can't understand them fully, really, the explainability is actually poor, but we can know observationally that they…
Aaron Gallant: are able to essentially understand statistical patterns in language, and then use that to generate more language. Okay. Now, when we do that with these models, that's actually basically the same as just a good old
Aaron Gallant: Cartesian plot. How many people remember Y equals MX plus B? Anybody at least remember that much from math?
Aaron Gallant: Back in… whatever, middle school?
Aaron Gallant: The slope-intercept form. Now, Y equals mx plus b is, like, the simplest sort of parameterizable formula you can sort of come up with.
Aaron Gallant: the parameters is M, the slope. It's one parameter, right? And that slope
Aaron Gallant: determines how the line points, essentially. Yeah, and then the Bs, the intercept, we kind of don't care as much about it, but yeah, it's there too.
Aaron Gallant: All of these models are the same thing, except they're all Y equals MX plus B.
Aaron Gallant: But the difference is, of course, Y, what it gives you, what it returns, is actually tokens, which are numbers, but numbers that are turned back to words. So, essentially, it makes words. X, what you give it, is also words, of course, your prompt, that's the input.
Aaron Gallant: And M, instead of being a single number, like 3,
Aaron Gallant: Is a bazillion numbers, or, you know, slightly more accurately, say, 7 billion parameters for a modest open source model, right?
Aaron Gallant: 13 billion parameters, and 130 billion parameters, like, billions of parameters as opposed to one. Parameter just means number. So the number of numbers in M organized in these matrices, really huge.
Aaron Gallant: Now, that said, it's still the same thing. It's still basically multiplying through. It's a little more complicated, because linear algebra, but you don't have to worry about that to understand it conceptually, really. You have to worry about if you want to implement it from scratch, which is cool, but you don't have to do that, and that's not where the reward necessarily is, so…
Aaron Gallant: That is inference, though. Just sticking an X in there.
Aaron Gallant: and getting different Ys. That's what we all do. Every time we prompt, every time we rag, every time we do any of that, we're executing the function and getting some new Y out, right?
Aaron Gallant: Fine-tuning, we're changing the M. We're going to the M, and we're changing those numbers. So that's what I mean about changing, and those are called the parameters, those are the weights, or numbers that determine the behavior of the model itself, and that's a durable change to the model.
Aaron Gallant: I'm emphasizing this because I think it's an important thing to understand, an important distinction to make. Are people pretty clear on this?
Aaron Gallant: If there are still questions on this, drop it in the chat. Great, alright.
Aaron Gallant: So…
Aaron Gallant: back to the real world. We can step back from math a little bit. Thank you, Trevor. I enjoy math, I mean, and to be clear, it's great to go deep in math, but it's great to know how deep you have to go to solve the problem you're solving. And right now, this is deep enough.
Aaron Gallant: So…
Aaron Gallant: Of course, most of the work in our, say, corporate setting is dealing with the data, and the reason I made that big deal about inference versus training
Aaron Gallant: Rewind here. Garbage in, garbage out. You've all probably heard that. It applies to both inference and training, right? If you give a bad prompt, you're gonna get bad output.
Aaron Gallant: But the thing is, is that only happens that one time. You give it that bad prompt.
Aaron Gallant: that one time, and you get a bad response that one time. Obviously, that's bad, by definition, but it's once.
Aaron Gallant: If you use bad data while you're training, you're teaching the model badly. You're now basically making the model bad. You're making all the future bad. You're getting bad…
Aaron Gallant: Ad infinitum.
Aaron Gallant: So much, much worse. So that's why prepare data is so critical. You really need to be sure, if you're training your model, that your data is correct, that the labels, and we'll get more about what that means in a bit.
Aaron Gallant: But basically, what you're saying the data is.
Aaron Gallant: what it's learning from, it better be learning from something that's right. If it's learning wrong things, it's gonna make things worse. And it's also why… so, by the way, step 3, adjust weights, that's just run a function. That's just a little bit of code, and then…
Aaron Gallant: Possibly writing a check to some compute source to do all the crunching.
Aaron Gallant: But then 4, validate is where, I guess, the next most work is, because you want to take your label data, set some of it aside, you don't let the model train on all of your label data.
Aaron Gallant: Because you keep some of it, and you use it to validate later. You use it to see, hey, is the model actually good or not? And the reason you have to hold it aside is you don't want the model to cheat. You don't want it to learn the answer key, as it were.
Aaron Gallant: Alright.
Aaron Gallant: By the way, pretty much everything I'm saying on this slide more or less applies to machine learning in general. I mean, we're talking about this in a large language model fine-tuning context, but this is all general supervised learning, machine learning stuff.
Aaron Gallant: This is sort of the intersection of AI engineering and, say, machine learning engineer. So… Alright.
Aaron Gallant: OpenAI. I'm not gonna spend too long on this, but I want you all to know it exists. So, OpenAI
Aaron Gallant: which, you know, offers of ChatGPT.
Aaron Gallant: despite their name, they're not exactly open, right? Like, they are somebody else's computer.
Aaron Gallant: They are a service that we can pay, granted, a service that offers pretty good quality, it seems, and cheap prices at the moment, but a service that we pay, send some information over the wire, get something, get a response back, right? An API.
Aaron Gallant: They still offer, though, a fine-tuning API.
Aaron Gallant: And, the way that works in the case of OpenAI is it's sort of like a limited fine-tuning playground. You can specify the base model, so you could say, oh, I want to start from 4.0 Mini, and you probably do want to start from a small model.
Aaron Gallant: and then you give it data, and the data has to be in JSON-L, JSON lines format. And that's like a…
Aaron Gallant: a text file where each line is a JSON blob. And the format of the JSON blob, as you'll see in the next slide, is just…
Aaron Gallant: discussions, because that's what label data actually is in this case. You know, labeled data in general.
Aaron Gallant: going back to our friend Y equals MX plus B, labeled data would be pairs of X's and Y's. You know, for this X, this should be Y. For this X, this should be Y, right?
Aaron Gallant: For language, it's prompt response. For this question, or request, or whatever, user input.
Aaron Gallant: This should be the model's response. This is how it should answer that question.
Aaron Gallant: Or, you know, maybe it should do a tool call. Whatever, like, this is what the model should do.
Aaron Gallant: And it's convenient because you get that naturally by just logging every time you interact with a language model. Like, if you log interactions using an observation platform like Langfuse or Langsmith, you can get these, you know, prompt
Aaron Gallant: Discussion histories, and you can use that
Aaron Gallant: Assuming you also quality check it and make sure it's good, that's at least the right shape to be training data.
Aaron Gallant: And that's the shape that OpenAI wants.
Aaron Gallant: Now, that said, OpenAI doesn't really let us actually get anything.
Aaron Gallant: At the end of the day, we give them the data, and they give us, like, a little bookmark. They give us an ID that says, okay, we're holding the data.
Aaron Gallant: And then we say, we want to train the model, and they train the model, and they give us another little bookmark, another little ID that says, hey, here's your trained model.
Aaron Gallant: You can't download it, you can just use that model ID for future inference on OpenAI's API.
Aaron Gallant: And, of course, it comes at a cost. It comes at both a training cost, and inference is more expensive on your custom models than similarly-sized, non-custom models. Because, of course, OpenAI has to have some extra infrastructure in place to host your custom model.
Aaron Gallant: Probably can't optimize or cache it as well, that kind of thing.
Aaron Gallant: So, definitely a lot of limitations. You also, you know, don't get access to anything, really, any of the model weights. You don't really have a lot of hyperparameters. We'll see more hyperparameters in the good example coming up.
Aaron Gallant: But the reason to know this exists is, first off, if you want to just, like, play with fine-tuning, and you don't mind spending…
Aaron Gallant: you know, a little bit of time and money on OpenAI, this can be a good little testbed environment.
Aaron Gallant: And secondly, you know, there could be use cases for this if you're deep, if your org is all in, open AI, and you want to fine-tune. But overall, again.
Aaron Gallant: I think, at least, that it's the open… the actually open models where the… where the fun is at.
Aaron Gallant: So,
Aaron Gallant: on that note, well, first off, there's a little bit more about the message format, as I said. So that's… that's one line of a JSON lines file showing an interaction where there's, like, a conversation history.
Aaron Gallant: And it's just a silly example, you wouldn't really fine-tune on this, most likely, but it shows for a sarcastic chatbot, if it receives this question, this is the answer I should give.
Aaron Gallant: Great.
Aaron Gallant: And you can make this, you can synthesize this by just interacting with a big LLM that can solve this problem. And that, again, is sort of the,
Aaron Gallant: the bootstrappy way to at least generate your initial data. You still need to curate its quality,
Aaron Gallant: and you, still need to format it appropriately for whatever fine-tuning platform you're using. In the case of OpenAI, they want this format.
Aaron Gallant: Alright, so this… I'm not going to spend time here. Click this link if you want to actually do the OpenAI stuff. That's the point. This links to OpenAI's documentation. You'll have the slides at the end, this link will be on this slide, but this shows the various steps to actually do things on OpenAI's fine-tuning if you want.
Aaron Gallant: But… what can you do besides play at OpenAI's playground? Well…
Aaron Gallant: a lot of stuff. Again, I really think the coolest thing of this space isn't the, the behemoth
Aaron Gallant: models and companies necessarily, but is the whole ecosystem. And the fact that this technology
Aaron Gallant: I wouldn't go so far as to say it's, like, inherently…
Aaron Gallant: Shareable exactly, or inherently open. And in fact, what we call open models
Aaron Gallant: they're not really the same as open source, when you think about it. So,
Aaron Gallant: Oh, let me address this question before I go off on this topic here. Is it fair to assume OpenAI might be easier to learn with? Yeah, if you want to just do, like, hey, I just want to do a bare minimum feedback loop of fine-tuning a model, you could do that in, you know, 20, 30 lines of code with OpenAI,
Aaron Gallant: And maybe a little more, but like, yes, it would not be too hard. You can read their docs, and it's a sort of curated experience. The hardest part, of course, is the data.
Aaron Gallant: But if you have that, then that's the easiest to sort of try it with. But again, the only use cases, I think, are if your org is, like, really all in with OpenAI.
Aaron Gallant: And it's like, yeah, we want to fine-tune GPT-40 Mini to make it really good at something, whatever.
Aaron Gallant: And again, I'd still fine-tune the small models, typically.
Aaron Gallant: But anyway, so, back to model… selection.
Aaron Gallant: So…
Aaron Gallant: We have here, all the different places we can get models from, and as I was saying, it's this whole ecosystem of, quote, open models.
Aaron Gallant: But I do think it's worth understanding, as practitioners or interested, aspiring practitioners in this space, what open model actually means. Most people just mean open weight. That means you can download
Aaron Gallant: perhaps a multi-gigabyte file that basically contains all those numbers I was talking about, all those parameters in that matrix, or really a bunch of matrices, right?
Aaron Gallant: And that's all well and good. That lets you use the model, and what's really nice about fine-tuning, that lets you fine-tune the model.
Aaron Gallant: You actually don't need, like, the original blueprint, per se, or the original training data, or the original anything. You just need the model in its current state, and you can continue to train it. It's sometimes in the supervised learning space called online learning, which is to say that you can just continually train something without having to start over from scratch.
Aaron Gallant: Really cool. That's… that's what enables all of this. However, it's not really the same thing as open source.
Aaron Gallant: Because open source, when you think about it for software, means the source code is open, and the model is re… and, sorry, the software is reproducible, because anybody can take it from the source and compile it.
Aaron Gallant: An open model is actually more like a binary. It's kind of already been compiled.
Aaron Gallant: It's a little fuzzy, but I encourage people to be mindful of that in this space. There are a few organizations out there that are potentially trying to do open source models, where they actually open not just the model, but also the data and all the architecture and training process and everything.
Aaron Gallant: But most of the big open models, like, say, Llama from Meta.
Aaron Gallant: are just really mostly open weight. You know, that's what they put out there.
Aaron Gallant: That said, we can still use them for this. Hooray. So, how do we pick what model to start from? Well, there's a whole bunch of models out there, and again, you'll get these slides, all these links are clicky, and you can get more details, but the highlights here…
Aaron Gallant: Is, you might already be aware, there's a whole bunch of benchmarks and ways to measure model performance, and in general, you'd want to pick a model to start from that is, well, first, of the size that you want to handle, so however many parameters
Aaron Gallant: basically corresponds to how much RAM
Aaron Gallant: you need to run the model, which corresponds to what sort of environment you want to run the model in, right? So that's probably your first decision.
Aaron Gallant: And then once you make that decision, you can compare models based on their performance at tasks that are close to the task you want to solve. So, for instance, if you're making something that's supposed to write code, you might care about Sweebench, which is a benchmark specifically for software engineering skills.
Aaron Gallant: And then, regardless of the specific sort of benchmarks out there, the general metrics that you see in this space
Aaron Gallant: So…
Aaron Gallant: sort of the most accessible, and an important one, and really the only one I would ever present in, say, a business context is accuracy. And what we typically mean by that is how often is it right? Like, how often did it do the thing?
Aaron Gallant: And again, that often means boiling the task down to a binary success-failure, or at least a classification sort of thing, which is often a realistic thing to do for a lot of tasks we care about.
Aaron Gallant: And then rating, okay, it did this one right, it did that one wrong, it did it right 92% of the time, out of our, you know, test cases that we held out, right?
Aaron Gallant: And that's… that's a good number to have. There are other types of accuracy, and I'm not going to get into the nuances of that now. Suffice it to say that if you start diving deep in this, find your local friendly data scientist, and make sure that you are understanding the numbers, but accuracy is a good place to start.
Aaron Gallant: These other ones are more about… are a little more LLM-specific here, alignment with, human reference, and perplexity. And perplexity I like in particular because that basically shows how surprised the model is
Aaron Gallant: And, what that means…
Aaron Gallant: you're like, wait, how surprised the model is? Well, don't anthropomorphize the model, to be clear. The model doesn't get surprised the way you or I might, but…
Aaron Gallant: Let's say you, for some reason, train a language model on nothing but the complete text from everything from Lord of the Rings.
Aaron Gallant: Shakespeare, whatever. Some famous author, right? And a little dated, say, right? Like, it's… so this language model has been trained on the entire corpus of Lord of the Rings for some reason.
Aaron Gallant: And then you fed it,
Aaron Gallant: I don't know, anything new, right? You fed it, tweets, right?
Aaron Gallant: Most tweets are not written.
Aaron Gallant: the way J.R.R. Tolkien writes, certainly not the way Shakespeare writes, if we go back with that as our hypothetical training data. And so, what that means is that the input is sort of out of domain from what the model expects to see based on the training data. The model is trained on a certain domain.
Aaron Gallant: Y equals MX plus B, domain, range, ringing a few bells.
Aaron Gallant: Was, was, trained on a certain domain of input.
Aaron Gallant: And, that domain doesn't cover this new input.
Aaron Gallant: Now, of course, the example I gave is a little bit absurd, but it's meant to be illustrative. Realistically, what perplexity shows you
Aaron Gallant: is, you know, let's say you're in a very specific… well, like that question we got earlier, like, oh, we're working with… in a medical space with this very specific language, and the model probably hasn't seen a lot of training data for this language, because it's not like it's Python. There's not a million Stack Overflow articles
Aaron Gallant: on this particular piece of technology, and so the model doesn't know about it. Well, that's something that perplexity might actually pick up.
Aaron Gallant: I still wouldn't necessarily open with fine-tuning to solve that problem, but you might see that sort of thing with perplexity.
Aaron Gallant: And then lastly, the real gold standard is still… look at it, think about it. Now, that's 4, if you were wondering. Human evaluation. I'm assuming you're all humans.
Aaron Gallant: I don't think we have any LLMs in attendance at the moment. I imagine that they're not at that point.
Aaron Gallant: In any case,
Aaron Gallant: the human evaluation step is kind of critical, sort of like user acceptance testing from a software perspective. We are the ones who define what success and failure is, what business value is. You know, we talk about models hallucinating.
Aaron Gallant: But models are just generating text. They really are. You know, what to some sense, the skeptics say is actually correct. They are just next token predictors, but they happen to do it in a way that's often really useful if you use them effectively.
Aaron Gallant: However, we are the ones who have to evaluate that. And so it's not that the models hallucinate.
Aaron Gallant: So much, because they, again, just like they can't be really surprised, they can't really hallucinate. They don't have a mental model like that. That's not what's going on. But they certainly confabulate. They can just string plausible text together. You know, something I try to avoid doing.
Aaron Gallant: well, I suppose I try to have plausible, but also correct text, right? The correctness, though, is something we have to ascribe to it. So that's… that's the takeaway here.
Aaron Gallant: Alright, I'm gonna pause for a sip. I've been talking for a while, I haven't seen any questions coming in a bit. If anybody wants to drop a question in the, chat.
Aaron Gallant: Gone once.
Aaron Gallant: Going twice. Alright, well, I'll also try to save time at the end for more questions.
Aaron Gallant: Oh, I see, something that just came in about perplexity. Yeah, I mean, perplexity is… is… would often reflect, like, if your domain, like, again, the example is you have a bunch of really specific medical jargon or medical technical jargon that you're feeding into this model that isn't well reflected in the model's training corpus.
Aaron Gallant: So yeah, I think the answer is yes to your question.
Aaron Gallant: Alright, so, RLHF.
Aaron Gallant: Now, to be clear, That's reinforcement learning with human feedback.
Aaron Gallant: Fine-tuning is not exactly RLHF, but they're kind of cousins-ish, and RLHF is important to understand, because RLHF is what those big companies, what OpenAI and Anthropic and so forth, are doing to make the models into chat models. Indeed, RLHF, we've had…
Aaron Gallant: general pre-trained transformers in one form or another. I mean, the attention is all you need paper was 2017. Like, transformer architecture's been around, and I've been following it for a while, and it's been cool for a while, but it didn't really get obviously useful to most people until it started actually, quote, chatting.
Aaron Gallant: And how did it do that? Because before that, it really was just a next token completer. It really was just a really smart autocomplete. It was just trained on, here's the words up to this point, what should the next word be?
Aaron Gallant: But it was trained on a huge amount of data, and that made it do interesting things.
Aaron Gallant: Well, it turns out that if you take that model that, you know, because if you take a model like that, and you ask it a question, is it going to answer your question?
Aaron Gallant: Maybe not, because what it might do is it might just write more similar questions.
Aaron Gallant: That's actually the most likely thing you'll see, because you see that pattern, FAQ pages and stuff, lists of questions, quizzes and stuff written on the internet.
Aaron Gallant: And so, to make it instead
Aaron Gallant: decide to answer the question, we do reinforcement learning with human feedback, which is setting up an environment, that's essentially what reinforcement learning is here, and the environment here is paying a whole bunch of contractors.
Aaron Gallant: to interact with this thing, according to some playbook and some set of prompts that… and again, this is the main thing that distinguishes Brock from Claude here, is what the prompts were to do this step, and what the evaluation, what the sort of playbook that the humans used to say, yeah, this is good, this is bad.
Aaron Gallant: But that's all this is. You give it certain input, And, you…
Aaron Gallant: then check the output, and a human rates it, and then you train the model based on that, and then you just do that in a feedback loop until the model gets where you want it to get. And in a sense, it's sort of like labeled data, except you're labeling it in real time with human scoring.
Aaron Gallant: So that's where the connection to fine-tuning, comes in here. So again, all the models we're using already had this done to them.
Aaron Gallant: We're going to use a similar mechanism, but instead, we're going to label data, and
Aaron Gallant: I already talked about synthesizing data.
Aaron Gallant: And that's one good way to get labeled data. Another good way to get labeled data is user behavior.
Aaron Gallant: So, you know, you've all seen there are… you know, every website in the world, pretty much, has little thumbs up, thumbs down widgets, right? As a user, you'll be like, I liked this, I didn't like that.
Aaron Gallant: Well… If you instrument that, and that's what this snippet of code is sort of mocking.
Aaron Gallant: you score that, thumbs up is 1, thumbs down is 0, and you save that in a platform like Langsmith, and you associate that with the whole
Aaron Gallant: trace the conversation, whatever led up to it. Congratulations, you now have labeled data. You now have an LLM response to a user question.
Aaron Gallant: And you know whether it was good or bad. And you can use that to decide how to include it in a dataset, right? So, really, user-generated data is the only way to get data at huge scale. You know, nobody manually makes
Aaron Gallant: billion row data sets. That's a lot of contractors. You know, the only way you get really big data is this. Now, that said, you don't need really big data, and big label data, that is, to do fine-tuning, but this is still a good, convenient way to grab it.
Aaron Gallant: My rule of thumb… for any, like, how much data do I need to train a model in general.
Aaron Gallant: is, well, how much variation is there in the space you're thinking about? You know, what's a representative sample look like?
Aaron Gallant: And you probably want, you know, at least a few thousand observations, rows, you know, data entries, and you want them to cover all the cases. But I would rather have a thousand representative
Aaron Gallant: sample that has variation in it than 10,000 that are all just about the same thing over and over again. Because that's not as good to train on, because it won't teach it about the whole variety.
Aaron Gallant: Alright, so finally on to some of the alphabet soup here, and I'm checking the time. I'd love to answer more questions, but I'm gonna have to push a bit, I think, to get through these slides first, and then as there's time at the end, I will, knock out a few more.
Aaron Gallant: How do we actually do this? How do we do this with a model that, we control? Well, we can use something called parameter-efficient fine-tuning to make it even easier.
Aaron Gallant: And what this does is this takes our model, our pre-trained transformer model, freezes most of it, and then adds a few layers. And these layers have to be the right dimensions to map up. The software makes that not too hard.
Aaron Gallant: Dimensions is just the number of numbers, and they have to align.
Aaron Gallant: But then, that means that instead of training the whole model, we're training
Aaron Gallant: less of the model. Now, you might think, well, that means we change it less. The answer is, yeah, kind of, but this is the output, so we still are potentially very much changing the actual
Aaron Gallant: behavior, the end result of the model. We're just kind of doing it a little more cheaply and keeping most of the model in place, because hopefully the model is kind of close to what we want, we just want to tweak it.
Aaron Gallant: So, that's efficiency number one.
Aaron Gallant: Train less stuff.
Aaron Gallant: Efficiency, too.
Aaron Gallant: make the num… make there be fewer numbers in the number… sorry. This is have fewer sets of numbers. This is have fewer numbers in the sets of numbers. So, all of these…
Aaron Gallant: neurons, all of these things in a neural network are organized in matrices. Matrices have rank, which is another word for, essentially, size or dimension, the number of numbers in them.
Aaron Gallant: Well, linear algebra
Aaron Gallant: lets you do certain things, to manipulate the rank of matrices. And, what that means is you can basically
Aaron Gallant: lossily make a matrix smaller. Lawsily means you are losing information, but you're keeping the most significant information, the most relevant information in the matrix, the most important numbers.
Aaron Gallant: And you're losing some other numbers that, you know, most matrices that are really big are sparse, they're mostly empty. So you're not losing too much. It's sort of like, you know, taking a wave and making it new into an MP3.
Aaron Gallant: Or taking a picture and making it into a JPEG.
Aaron Gallant: We're losing a little bit, but most of it's there. And the result is, hey, less numbers.
Aaron Gallant: And then, lastly, we can also quantize, and quantization is not at all specific, of course, to,
Aaron Gallant: to large language models. Quantization is something that's been around in computers as long as there have been computers, I imagine, at least digital computers, because it just refers to truncating floating-point numbers. So we're taking, say, 32-bit floats and truncating them down to 8.
Aaron Gallant: And we keep the 8 most significant digits, and we're still accurate to a pretty fine decimal place, just not as many as before.
Aaron Gallant: Again, we're losing some information here, but we're keeping the most important information, and we're making, in this case, 32 to 8, that really is… that's 4 times smaller, right? So, if you stack all this stuff together, the end result
Aaron Gallant: Is you can take something like a 13 billion parameter model.
Aaron Gallant: And you can actually fine-tune it on possibly the laptop you have on your desk right now.
Aaron Gallant: Not gonna guarantee that for everybody here, but if you have a not terribly old Mac.
Aaron Gallant: chances are pretty good. The reason I highlight Mac is Apple doesn't pay me, sadly, but the M1 architecture has good optimization for this sort of stuff. If you're not in the Apple ecosystem, never fear, but it just means you might have to…
Aaron Gallant: get some other thing, like, you know, a GPU or some other bit of silicon that's the right way to do this sort of thing.
Aaron Gallant: In any case… That's the takeaway here, that we can, train models
Aaron Gallant: but really little resources, and that enables a lot of experimentation. And again, the main goal here is to train these small models anyway. That's the main useful use case, is to take a model that you might want to run on a Raspberry Pi, or something funny like that, and train it really small. And to be able to do that on your own hardware is pretty powerful.
Aaron Gallant: Alright, summary slide, I'll leave this here for your reference, and I want to use the remaining time
Aaron Gallant: Please start dropping questions again, and I will catch up, but while people are thinking and dropping some questions, I'll just very quickly show the code example. I'm not gonna really run through it, but you have it, it was just linked there, and it's this link here.
Aaron Gallant: And, I have a pre-ran version of it here, because it does take a while to run. A few notes. This is Google Colab, it's a way to run Python notebooks in the cloud. You don't have to run this. This is only this notebook. Daniel's the only one. You don't have to run it in Colab. You can download it, you could run it on other services.
Aaron Gallant: Also, you should probably make a copy of this notebook. So if you see here, save a copy in Drive, you want to make a copy so you have your own copy that you can edit and run, and you also are going to want to change the instance type. So if you see that there's these connection…
Aaron Gallant: options, runtime types, you're gonna want the T4 GPU instance. Otherwise, this stuff won't really run great. And that is for reasons of parallelized vector computation stuff.
Aaron Gallant: The math, right? Has to have the right chip behind it.
Aaron Gallant: So, what's in this notebook?
Aaron Gallant: 5-minute version or less. A lot of boilerplate and imports.
Aaron Gallant: And then… a lot of setup with a lot of hyperparameters. So I've used the word parameters a ton.
Aaron Gallant: Parameters, again, are the numbers inside the model, the numbers that we normally don't change when we're just doing RAG or whatever, but that we are changing now with fine-tuning. We're changing those numbers. Okay.
Aaron Gallant: Hyperparam… the thing about parameters is there's literally billions of them, and if you point to any one parameter and be like, why is that parameter 2? Or it wouldn't be 2, like, 0.6, whatever.
Aaron Gallant: we don't have… that's what I mean by the… we don't have good explainability. We don't know. All we can say is, like, well, that was just the optimal point that the training process arrived at for that parameter, and we can just observe the outcome.
Aaron Gallant: However.
Aaron Gallant: Hyperparameters are not inside the black box. They're like the knobs and levers on the outside, and they're what we have control over, and they're what you want to find, again, your local data scientist or machine learning engineer to get a deeper understanding of this, and to know what you're doing here.
Aaron Gallant: But you want to choose these hyperparameters such that you have a good training process. And there's rules of thumb.
Aaron Gallant: But there's not really any single hard and fast answer to how to do all this. So, to some extent, you also have to do experiments, where you do smaller training runs, and you try different hyperparameters to see what's best. There's a lot here. What I'll highlight
Aaron Gallant: Dropout is an interesting one, because dropout's a little counterintuitive. Dropout refers to, essentially
Aaron Gallant: A probability that a neuron that you're training just won't even fire.
Aaron Gallant: And you might think, well, why would we ever do that? Why would we go to all this effort, and train things, and then this part of it… we just say, oh, this math isn't gonna math this time.
Aaron Gallant: That's a waste, right? Well, no, because what we're doing is we're…
Aaron Gallant: making the system robust to failure. It's sort of like chaos testing, if you're familiar with that, where, you know, your infrastructure should be robust to things randomly failing. Well, the model should be robust to neurons randomly not firing, because that also
Aaron Gallant: ensures that we don't overtrain in any one particular area of the model and end up with, like, one neuron with sort of too much responsibility. The reason that would be bad is that's not going to generalize well. It might seem like it learns the training data really, really well, but it's overfitting, and it's not going to generalize to unseen data, and that's what we actually care about.
Aaron Gallant: Bunch more stuff here. Some of these are a little more operational. Number of epics, that's how long we train, essentially, one full training run.
Aaron Gallant: Learning rate. This is another one that I would really highlight. This is one that's even exposed by OpenAI's API. This is how fast you train, essentially. This is how fast you update the weights when you need to update the weights. And again, you might think.
Aaron Gallant: well, just make this as big as we can, right? Like, obviously, we don't want to waste time. We want to train quickly. Well, yeah, we do, but what we're doing here is we're solving what is called an optimization problem, which is we're finding the best set of numbers to make the model behave as best as it can with our data.
Aaron Gallant: Basically predict the data effectively, accurately.
Aaron Gallant: And it's sort of like, if you picture it, it's in a very high-dimensional space, billions of parameters, but picture just three-dimensional space. Picture climbing a mountain.
Aaron Gallant: Or maybe picture, two-dimensional space, even. Picture putting a golf ball on a green, right?
Aaron Gallant: As you get closer, To the whole.
Aaron Gallant: you want to use less force, right? And why is that? Well, if you hit the ball too hard, essentially you take too large of a step.
Aaron Gallant: You're gonna skip straight over that hole.
Aaron Gallant: Right? And that's… that's bad. You're not going to reach that optimum point. If you need to reach a specific, precise point in two-dimensional space, you need to take little steps the closer you get. That's what we're talking about here, and that's what…
Aaron Gallant: This is doing for us.
Aaron Gallant: All right, and then the rest of this, and then I want to get to the questions for the last 5 minutes. There's more hyperparameters here. Documentation for these, software will answer questions if you have particulars.
Aaron Gallant: And we put it all together, basically, so we're hooking up… you see the bits and bytes, that's the quantization bit, so we're doing QLORA,
Aaron Gallant: And, the tokenizer is the thing that changes the words into numbers, vice versa.
Aaron Gallant: And we're fine-tuning the model.
Aaron Gallant: And I'll show the data in a second, too, but what we're fine-tuning the model with
Aaron Gallant: is from Hugging Face, somebody just mentioned Hugging Face.
Aaron Gallant: we have some label data about tweet sentiment. So the idea is, is, what is the sentiment of this tweet? Rate it negative, neutral, positive, and then there's a tweet about some company doing something, and then at the end, there should be a label. Yeah, neutral.
Aaron Gallant: And etc.
Aaron Gallant: And the idea is, oh, we could use this model to maybe do financial stuff.
Aaron Gallant: We run trainer.train, and it's a llama model that we're starting from.
Aaron Gallant: We train it for a while, and we see the loss go down. The loss is sort of how bad the model is. It's another way to
Aaron Gallant: say, model performance, and by how bad, I mean how unlike the good model output. Like, how un… how often is it wrong, based on the labeled training data? And how much is the output like?
Aaron Gallant: What it should be.
Aaron Gallant: And so we want loss to go down, and indeed it does. We see that it starts out at 3.29, and then it goes down, and it sort of bounces around, kind of stabilizes around .9 or 1, maybe.
Aaron Gallant: If we look at that in this little dashboard here, we can see the…
Aaron Gallant: Light gray line is the actual jagged loss, and the bolder line is smoothed as a rolling average of the loss. And you can see that it sort of bottomed out, and that's when we stopped training. So we've optimized the model to some extent.
Aaron Gallant: Probably not really, because we could try other hyperparameters, but we certainly made the model better at doing this data.
Aaron Gallant: And if we give it an example now, a hypothetical tweet about Tesla superchargers, we will see that it responds.
Aaron Gallant: And you notice, this format's different Oh.
Sunny with Gauntlet: Hi everyone, I think Aaron's just having a little glitch, let's just give him a second, as I think his computer might have froze, of course, right at the last second.
Sunny with Gauntlet: Let's see if he's able to hop back on…
Sunny with Gauntlet: Yeah, let's see…
Sunny with Gauntlet: Oh, it looks like we might have dropped him. Sorry about that, everyone. But we'll be sharing this recording via email, so no worries. Oh, is he jumping back on? I think we might have gotten back.
Aaron Gallant: Sorry about that. My computer over… my computer literally overheated, it's kind of hot.
Sunny with Gauntlet: Oh, no!
Aaron Gallant: I didn't… I didn't turn on air conditioning aggressively enough. Yeah, it's… anyway. We're about to wrap up. So, I was sprinting to the end of the notebook. I'll go ahead and,
Aaron Gallant: you, you have a copy of the notebook, so you can see that it rates the tweet as positive sentiment. So that's… that's the idea.
Aaron Gallant: Let me get this back up here.
Aaron Gallant: And also, check the questions I can check in the chat.
Aaron Gallant: Alright.
Aaron Gallant: Get the chat back up.
Aaron Gallant: Oh, except…
Aaron Gallant: crashing out means I lost the chat history. So if you asked a question earlier, please feel free to re-ask it. I'll spend at least a few minutes here at the end answering questions, and then we'll, I think call it. So, do you have a strong preference, between LM Studio, Ollama, or other app storing local models? Not a strong preference. I mean…
Aaron Gallant: a lot of them build on the same, like, you know, llama.cpp, or same sort of, underlying stuff. Lm Studio has been sort of a…
Aaron Gallant: kind of convenient UI, I'll give it that. But if I was doing something programmatic, I might reach for Olama instead. A link to the slides. Yes, there is a link we can share for the slides. I can drop that now, or maybe Sonny will drop that.
Aaron Gallant: Here, I've got it.
Aaron Gallant: Or, no, I need to send it to everybody.
Aaron Gallant: There we go.
Aaron Gallant: All right, and while we are possibly waiting for a last question or two, I just want to also close by saying thank you for joining. I hope you enjoyed this. As clear, I enjoy this stuff. I think this is a great
Aaron Gallant: area to be in, a great thing to be thinking about and learning about, and if you are interested in thinking and learning more, please join us more. They're, of course, going to continue to be night classes, but we also
Aaron Gallant: As I imagine you know, I have the regular full gauntlet program. The next, application, I believe Sonny said, is July 6th. Did I get that right? Or not… apply before that, but the next start is July 6th, is that right?
Sunny with Gauntlet: Yes, that's correct. Next cohort clicks off July 6th.
Aaron Gallant: July.
Sunny with Gauntlet: I'm gonna drop those links now.
Aaron Gallant: And also, if maybe you're not looking to make that sort of a change, you're, you know, you're at a place, and you have a career, but you still want to learn more about this, and your employer's supportive of that, or…
Aaron Gallant: perhaps you're the one making the decision. We have Catalyst, which allows for similar training for people who are already on the job, and we give a lot of hands-on help to make sure that these skills are really useful and applicable
Aaron Gallant: And you see return on it in what you're doing.
Aaron Gallant: And then last but not least, but there's also going to be…
Aaron Gallant: And Sonny, you'll have to help me here, because I lost windows here, but the, event in Austin…
Sunny with Gauntlet: Yes, so we are having a live event in Austin. It's called Fired Festival. I'm gonna drop a link right now in the chat for everyone. It's… we don't really always do live events, so it's gonna be a great time. If you want to apply for that or learn more, just follow the link that's gonna be in the chat.
Aaron Gallant: Great, and I think that's all we've got. Made… made perfect time. I…
Aaron Gallant: Looking at the questions here, what would we recommend for less technical people? I mean, not… this was probably a particularly technical night class, they vary. There will be other night classes that, have a variety of topics, and really, our programs in general, we do have, you know, product focus and other aspects to our program.
Aaron Gallant: Or to our programs, I should say.
Aaron Gallant: Alright, I don't see any other burning questions, so, thank you all, and
Aaron Gallant: Hope to see you again at something gauntlet-related.