What to Look for in an LLM
Picking an LLM sounds simple until you actually have to choose one.
There are plenty of models available, and they all seem to promise something impressive. Different individuals have varying strengths; some may excel in coding, while others are skilled at managing long documents. Additionally, some may provide services at a lower cost or complete tasks more quickly.
So, how do you decide?
The answer isn’t necessarily to find the most powerful LLM. It is important to find the option that makes sense for what you are trying to build.
Start With What You Need
Before comparing models, figure out what you actually want the model to do.
A customer support chatbot does not have the same requirements as a coding assistant. Likewise, a system that analyses long documents may need different capabilities from one that generates short responses.
Start with the task.
Maybe you need an LLM to:
- Write or review code
- Summarise documents
- Answer customer questions
- Extract information
- Analyse data
- Work with images
- Support an AI agent
- Return information in a specific format
Once you know the job, you have something useful to compare models against.
Don’t Get Stuck on Benchmarks
Benchmarks are useful. They can give you a general idea of how models perform.
But they aren’t the whole story.
A model can score extremely well on a benchmark and still not be the best option for your application. Your users won’t be giving it benchmark questions. They’ll be sending real requests, including vague instructions, unusual situations, and questions the model wasn’t specifically tested on.
That’s why testing models with examples from your own project is so valuable.
Give several models the same tasks and compare the results. You may be surprised by what you find.
Accuracy Matters, but So Does Consistency
Getting an answer right is obviously important. However, you also want a model that behaves reliably from one request to the next.
Pay attention to whether it:
- Follows your instructions
- Uses the information you provide
- Keeps to the required format
- Handles unclear requests properly
- Avoids making things up
- Knows when it doesn’t have enough information
This becomes even more important when the model is connected to other software.
A slightly better answer isn’t much use if the model regularly ignores instructions or returns information in a format your application can’t process.
Think About Context
Context is another important part of the decision.
If your application deals with long reports, extensive codebases, or lengthy conversations, it’s important to know how much information the model can process and how effectively it works with that data.
But bigger isn’t automatically better.
If your users only send short requests, you may never need a huge context window. Paying for a capability you don’t use doesn’t make much sense.
Speed and Cost Can Change the Decision
A powerful model can be great, but it can also be expensive.
Imagine an application handling thousands of requests every day. Even a small difference in the cost of each request can become noticeable over time.
Speed matters too. Someone asking a chatbot a simple question probably doesn’t need the most powerful model available if a faster one can provide a good answer.
This is why some applications use more than one model.
A simpler task can be assigned to a smaller, less expensive model, while a more complex request can be directed to a more capable one.
What About Tools?
An LLM doesn’t always need to work alone.
Modern applications can give models access to tools such as APIs, databases, and other software. This allows a model to do more than generate a response.
For example, an AI agent might need to check information in a database, call an API, and then use the result to complete a task.
If you’re building something like this, look closely at how well the model handles tool use and structured outputs.
A model that is great at writing text isn’t automatically the best model for an application that needs to take actions.
Don’t Forget Images and Documents
Some projects need more than text.
If you’re working with PDFs, screenshots, charts, diagrams, or other visual information, multimodal capabilities may be important.
For example, a business system could use a model to extract information from an invoice or analyse a diagram inside a document.
If your project doesn’t need these features, though, there’s no reason to make them a deciding factor.
Test Before You Commit
This is probably the most practical advice: test the models yourself.
Take a selection of real tasks your application will need to handle. Run them through the models you’re considering and compare the results.
Look at:
Accuracy. Does it get the job right?
Consistency. Does it behave reliably?
Speed. How quickly does it respond?
Cost. What will it cost when usage grows?
Usability. Does it actually fit into your application?
A benchmark can tell you how a model performed on a particular test. Your own testing tells you how it performs on your work.
The Best LLM Depends on the Job
There isn’t one LLM that makes sense for every project.
The right choice depends on what you’re building, who will use it, and what you expect it to handle.
For some projects, a fast and affordable model will be enough. Others may need stronger reasoning, better coding capabilities, longer context, or more advanced tool use.
And because models and pricing continue to change, the choice you make today may not be the choice you make next year.
Instead of asking “Which LLM is the best?”, ask a more useful question:
“Which LLM is the best fit for what we’re trying to do?”
That’s where the comparison becomes much more useful.
