Teaching a small LLM model how to “think” (or at least getting close to it)

How to “Level Up” a Small AI Model (Without Waiting for a Bigger Brain)

I’ve been building my own local AI models to write AutoHotkey code. I’m on version six now, and honestly? It writes pretty clean code. I’m impressed with how far a small model running on my own machine has come.

But I ran into a wall. And I bet you’ve hit the same wall if you’ve ever tried this yourself.

A small model can write code it’s seen the shape of. What it can’t do is reason. It can’t stop, think through a tricky problem, catch its own mistakes, and work its way out of a corner the way the big models (like Opus) do. Those big models had ENORMOUS amounts of time and money poured into teaching them how to think through problems. My little local model just doesn’t have that horsepower, and no amount of extra training data is going to magically give it a bigger brain.

So I almost gave up on the idea. Then I asked a better question. And the answer changed how I think about ALL of this.

Now on with the show. 🚀

Simulate Reasoning Using Mechanical AI Support
Simulate Reasoning Using Mechanical AI Support

The Thesis: Stop Trying to Add a Brain. Steal What the Brain Does.

Here’s the shift that made everything click.

Reasoning isn’t magic. It’s a means to an end. When a smart model “reasons,” you don’t actually care that it’s reasoning. You care about the RESULTS you get:

  • It catches its own mistakes
  • It handles several rules at the same time
  • It doesn’t waste time going down dead ends
  • It knows when it’s stuck

Those are the ends. Reasoning is just how the big models get there.

So the real question isn’t “How do I add reasoning to my small model?” (You can’t. Not really.)

The question is: “How do I get those same RESULTS some other way?”

And it turns out you can get most of them. Not by making the model smarter, but by building smart stuff around the model. The single idea that ties it all together:

Move the thinking OUTSIDE the model.

Reasoning is really three things bundled together: trying options, checking them, and remembering useful facts. Every one of those can happen outside the model, in tools and processes YOU control. Let me show you what I mean, benefit by benefit.

1. Catching Its Own Mistakes ✅ (You can basically nail this one)

A reasoning model notices when its code is broken before you even see it. A small model won’t. It’ll hand you broken code with total confidence. 😊

The fix? Don’t ask it to notice. Run the code, catch the error, and hand the error right back to it.

Here’s the cool thing: small models are TERRIBLE at spotting their own mistakes, but they’re surprisingly GOOD at fixing an error when you point right at it. “Line 14 is broken, here’s the message” is something even a little model can handle.

So you build a loop: generate the code, run it, feed back the exact error, let it try again. That one move gets you almost all the way there on plain correctness bugs. It’s the closest thing to “the model caught its own mistake,” except a tool caught it instead.

2. Juggling Several Rules at Once ✅ (Mostly solvable)

Ask a big model to write something that follows five rules at the same time (“use version 2, only one hotkey, no global variables, handle the missing-window case…”) and it holds all five in its head. A small model? It’ll nail two and quietly forget the other three.

The trick: stop asking for all five at once. Two ways that work:

  • Generate several versions, then run each through a checker that throws out the ones breaking a rule. Keep the survivor.
  • Or fix one rule at a time, one pass each.

This one needs you to build a little “rule checker” of your own. A small script that scans the output and flags anything that breaks your rules. That checker turned out to be the MOST valuable piece of the whole thing. It’s the tireless quality-control inspector that never forgets a single rule, even when the model does.

3. Not Going Down Dead Ends ✅ (Mostly solvable)

A reasoning model picks a good approach BEFORE it starts writing. A small model just starts typing and hopes.

The fix is almost funny in how simple it is: don’t let the little model plan at all.

Instead, figure out what KIND of request it is (a hotkey remap? a small tool with a window? an Excel automation?) and hand it a proven template for that exact type. Now it’s not inventing an approach and wandering into a dead end. It’s filling in a blank that already works.

This works great because so much of what we build follows familiar patterns. Be honest though: for something genuinely NEW that doesn’t match any template, you’ve got nothing to hand it. More on that in a second.

4. Knowing When It’s Stuck ✅ (Fully solvable, and easier than you’d think)

Here’s a fun one. A reasoning model realizes “this isn’t working, let me back up.” Your small model will NEVER realize that. It’ll happily spin in circles forever. 😃

But you don’t need the model to notice. You watch from the outside:

  • Cap it at 3 tries, then stop
  • If try #3 looks almost identical to try #2, it’s clearly spinning, so pull the plug

Then you escalate: hand it off to a big model or to a human. Spotting “stuck” is actually EASIER from the outside than from the inside. You’re just watching for repeats.

5. Handling Vague Requests ⚠️ (Partly solvable)

When you give a smart model a fuzzy request, it fills in the blanks sensibly. A small model fills them in badly and confidently.

Best you can do here: a simple intake form. Force the request through a fixed set of questions (What triggers it? Which window? What should happen? Any edge cases?) with sensible defaults for anything left blank. Structured questions work. Letting the little model ask its own questions does not.

6. Deep Detective-Work Debugging ❌ (This one’s weak)

Some bugs need a chain of reasoning: “the hotkey fires, but the window shows up late, so the keystrokes land in the wrong place, so I need to wait for the window first.” That’s A-causes-B-causes-C detective work.

This is the first place a small model genuinely falls apart. You can help a little by adding logging, but honestly? Route these to a big model. Don’t fight it.

7. Truly Brand-New Problems ❌ (No substitute exists)

Anything the model has never seen before (a weird algorithm, an unusual technical trick) needs REAL reasoning. There’s no tool, no template, no loop that fakes it. This is just the honest ceiling. Your little local model is a fantastic pattern machine, not a problem solver. Send these to the big model too.

So Here’s the Whole Picture

Out of the seven things reasoning gives you, you can reproduce FOUR of them with tools and process, get PART of a fifth, and you’re stuck on the last two.

That’s the beautiful part. Four-and-a-half out of seven, without making the model one bit smarter. 🤖

The pattern behind all of it:

  • Move the “trying” outside: generate a bunch of options and let a checker pick the winner. That beats one “thoughtful” attempt.
  • Move the “memory” outside: templates and saved examples instead of hoping the model remembers.
  • Move the “knowing when to quit” outside: retry limits and an escalation button.

And those last two you CAN’T fake (deep debugging and brand-new problems)? Don’t treat them as failures. Treat them as a feature. That’s your “call in the big model” button, and it should be built in on purpose, not bolted on in a panic.

Where I’d Start

If you’re playing with your own small model, build two things first:

  1. The rule checker (scans the output, flags broken rules)
  2. The run-it-and-fix-it loop (run the code, feed back the error, try again)

Those two alone get you about 70% of the value. Everything else is polish on top.

One thing I’ve learned doing this: the win isn’t a bigger model. The win is a better process wrapped around a modest one. That’s the same lesson automation teaches over and over. You don’t always need a fancier tool. You need a smarter setup around the tool you’ve got. ⚡

How about you? Have you been trying to squeeze more out of a small local model? I’d love to hear what’s worked (and what blew up in your face, because mine sure has 😊).

Teaching Claude to work YOUR way- Creating Claude Skills

Teaching Claude to Work YOUR Way: Creating Claude Skills

I was messing around with Claude the other day, and I caught myself typing the same instructions for the third time that week.

“Use AutoHotkey v2, not v1. Put the functions at the end. Include the header lines. Dark theme the GUI.”

Every. Single. Time.

And it hit me: why am I re-explaining all this? Claude is smart, but it doesn’t magically know how I like things done. It’s like hiring a brilliant new assistant who’s never met you. Talented, sure, but you still have to show them the ropes.

That’s exactly what Skills solve. And if you’re new to AI and new to Claude Code, this is one of those things that clicks and suddenly makes the whole tool 10x more useful.

Let me walk you through it.

What is a Claude Skill?

Think of a Skill as a little instruction packet you hand to Claude: your rules, your preferences and a few reference files, written once so Claude follows them every time. No more re-explaining “use v2, put functions at the end” in every chat.

Out of the box, Claude knows a LOT. But it doesn’t know your quirks, your rules, your preferred way of doing things. A Skill fills in that gap. It augments the model’s knowledge with YOUR knowledge.

Here’s the key idea: the model is the raw brainpower. The Skill is the specialized playbook. You’re not making Claude smarter in general, you’re making it smarter about your stuff.

Want every AutoHotkey script to include your header lines and put functions at the end? Write it once into a Skill. Now Claude does it automatically, forever, without you nagging.

The cool thing is you stop repeating yourself and you start getting consistent results. That’s the whole game.

How do you create a Claude Skill?

Just ask Claude. Here’s the part people miss: Claude has a built-in Skill Creator skill.

You don’t need to learn some scary format or hand-write config files. You literally ask Claude to help you make a Skill, and it walks you through it. Tell it what you want the Skill to do, and it scaffolds the thing for you.

It’s a Skill that makes Skills. 🤖

So the barrier to entry is basically zero. If you can describe what you want in plain English, you can make a Skill.

Why doesn’t Claude use my Skill?

Nine times out of ten, it’s the description. This trips up a lot of beginners, so pay attention.

Claude decides which Skill to use based on a waterfall load. Picture your Skills lined up, each with a short description. When you ask Claude to do something, it reads down that list of descriptions and picks the one that matches.

That means your description is everything.

If your description is vague, Claude won’t know when to grab the Skill, and it just sits there unused. Ever hired someone, given them a tool, and they never touch it because they didn’t realize it applied? Same problem.

So spend real time on the description. Load it up with the trigger words and phrases you’d actually say. If you want a Skill to fire whenever you mention “build a form” or “add a settings window,” put those exact phrases in the description.

A great Skill with a bad description is a Skill that never runs. Optimize the description. It matters more than you’d think.

How do you know your Claude Skill actually works?

This is where Tests and Evals come in. Don’t let the fancy words scare you, they’re simple.

Tests = the examples you throw at it

A Test is just a realistic request a user might make.

“Here are 5 example prompts someone might send this Skill.”

You run Claude-with-the-Skill against each one and look at what comes out. That’s it. This is the eyeball test. You read the output and go “yep, that’s what I wanted” or “nope, that’s wrong, fix it.”

Tests give you the examples to run.

Evals = the grading rubric

Now, eyeballing works for a while, but it’s slow and it’s a vibe check. Evals make it a number.

An Eval is a checkable pass/fail statement about the output. Things like:

  • “The output is valid AutoHotkey v2.”
  • “It includes the required header lines.”
  • “The functions are at the end of the file.”

Instead of squinting at each result, you write these assertions and the outputs get scored automatically. You get a score instead of a feeling.

Evals give you the rubric to judge by.

Put them together and you can actually improve

Here’s the loop, and it’s beautiful:

  1. Run your Tests (the examples)
  2. Check the Evals (the rubric)
  3. See what failed
  4. Tweak the Skill
  5. Run it again

Now you’re not guessing whether your Skill got better. You’ve got a score that says so. This is how you go from “eh, seems okay” to “this Skill nails it every time.” ⚡

The Trick That Almost Nobody Uses

Here’s my favorite tip, and it costs you nothing.

After you finish a session with Claude, if something felt clunky, if you had to correct it a bunch or manually fix stuff it should’ve gotten right, just ASK Claude about it.

Something like:

“Given our last conversation and the changes we had to make, what changes should we make to our Skill to avoid them in the future?”

Claude will look back at where things went sideways and tell you exactly what to add to the Skill so it doesn’t happen again.

Think about that. Your Skill gets better every time you use it, because you’re feeding real friction back into it. That’s compounding. One thing I’ve learned: the tools that improve themselves are the ones worth building.

Oh, And You Can Just TALK To It

One more thing, because I don’t want you sitting there typing paragraphs like it’s 1998.

Hit the microphone button and literally have a conversation with Claude.

I do this constantly. I’ll be describing a Skill, thinking out loud, rambling about what I want it to do, and Claude just goes with it. It feels less like operating software and more like talking to a coworker who happens to be really good at their job.

If you’re new to this, try it. Talking is faster than typing, and it lowers the barrier to actually experimenting. 😊

The Bottom Line

Skills turn Claude from a generic assistant into YOUR assistant.

  • They augment the model with your rules and knowledge.
  • The built-in Skill Creator makes building them easy.
  • Tests give you examples, Evals give you a rubric, and together they let you actually improve.
  • Your description decides whether the Skill ever gets used, so make it good.
  • And when something goes wrong, just ask Claude how to fix the Skill.

You don’t have to get it perfect on day one. Remember: life is an experiment. Try one small Skill, test it, tweak it, and watch how much time you get back.

Now go build something. 🚀

Deciding your AI approach: Rag, Skill or Fine-tuning an LLM

Train a Model, Build a Skill, or Set Up RAG? Here’s How to Actually Decide

I hit this exact fork in the road last week. I had a big pile of my own material sitting on my hard drive, and I wanted AI to actually USE it. So I asked myself the question everybody eventually asks: “Do I need to train my own model?”

Turns out, that’s usually the WRONG first question. And the reason why saved me a ton of time (and money). So let me walk you through how I think about it now. 🤖

What are the ways to get AI to use your own data?

When people want AI to work with their own stuff (documents, notes, courses, company data, whatever), there are basically three ways to do it. Here they are in plain English, no jargon:

1. Training (fine-tuning) a model
Rewiring the AI’s brain with thousands of examples so it permanently behaves a certain way by default.

2. Building a Skill
Handing the AI an instruction sheet plus a few reference files right before you ask your question.

3. Building a RAG system
Giving the AI a private search engine for your files, so it looks up the relevant bits before answering. (RAG just means “retrieval augmented generation,” which is a fancy way of saying “search first, then answer.”)

Should you fine-tune, build a Skill, or use RAG?

Start with a Skill. Then ask one question: is this about FACTS (what the AI knows) or BEHAVIOR (how it acts)? Facts problems get a Skill, or RAG once you have lots of documents that change often. Fine-tune only for behavior that better instructions can’t fix. Most people have a facts problem.

Here’s the thing that unlocked it for me. I asked a handful of the top AI models this same question, and they ALL landed on the same distinction:

Skills and RAG are about FACTS (what the AI knows).
Fine-tuning is about BEHAVIOR (how the AI acts).

Once that clicked, the whole decision got a LOT simpler. If you want the AI to know your stuff, that’s a facts problem (Skill or RAG). If you want the AI to consistently act or sound a certain way no matter what, that’s a behavior problem (fine-tuning).

Most people THINK they have a behavior problem. Almost always, they have a facts problem. ❗

When should you use a Skill, RAG or fine-tuning?

ApproachPick it when…Cost & upkeep
SkillYour material is small and curated (a few docs, a style guide, an FAQ) and doesn’t change much.Cheap. Fast. You update it by editing a file.
RAGYou’ve got a LOT of documents (or they change often), and you need answers pulled from specific sources.Medium. Some setup, ongoing tuning, but it scales well.
Fine-tuningYou need a very specific behavior, tone, or output format repeated reliably every single time, and prompting alone can’t nail it.Expensive. High effort, and you retrain every time your data changes.

The mistakes people make (so you don’t have to)

  • Fine-tuning: Trying to use it to teach the AI new FACTS. It doesn’t really work. The model still forgets and makes stuff up. Fine-tuning changes how it behaves, not what it knows.
  • Skill: Cramming in too many files and hoping the AI finds the good parts. More context is NOT always better. Too much junk just confuses it (and slows it down and costs more).
  • RAG: Building it before you actually need it. It’s easy to over-engineer a whole search system for a pile of documents that would’ve fit in a simple Skill.

Do you need to train your own AI model?

“You have to train your own model to make it sound like you (or know your business).”

Nope. A well-written Skill (a good style guide plus a few solid examples) gets you about 90% of the way there, for a tiny fraction of the cost and effort. Don’t get me wrong, fine-tuning has its place. But it’s rarely where you should START.

My rule of thumb: start small, escalate only when you have to

Here’s the exact order I’d tell a friend to follow:

  1. Start with a Skill. It’s the cheapest, fastest, easiest to change. It honestly solves most problems in an afternoon.
  2. Move to RAG when your documents outgrow what fits in a single prompt, or they change too often to keep pasting in by hand.
  3. Fine-tune LAST, and only when the AI already has the right facts but keeps getting the tone or format wrong in a way you can’t fix with better instructions.

Notice the pattern? You climb the ladder only when the rung you’re on actually breaks. That’s the whole trick. Most people jump straight to the hardest, most Expensive option because it SOUNDS impressive. Then they spend weeks on something a simple Skill would’ve handled in an afternoon. 🚀

The bottom line

One thing I’ve learned over and over with this stuff: the fanciest option is almost never the right first move. Try the simple thing. Measure how it does. Only reach for the heavy machinery when the simple thing genuinely can’t keep up.

Ask yourself the facts-vs-behavior question first. It’ll point you the right direction almost every time. Then start on the cheapest rung and climb only if you have to.

How about you? Have you been sitting on a pile of your own material, wondering how to get AI to actually use it? Start with a Skill this week. I think you’ll be surprised how far it gets you. 😊

P.S. The best part of starting small isn’t just saving money. It’s that you LEARN what you actually need before you commit to building it. Every experiment teaches you something. And that beats guessing every time.