How to Make LLMs Meet Enterprise Business Needs

Although large language models (LLMs) are powerful, they’re not perfect. By themselves, LLMs fall short in certain situations and need to be combined with other technologies. Many enterprises that want to implement generative AI believe they only need LLMs, only to find out later that the LLMs they’re using aren’t accurate, produce generalized responses, and fail to understand their content. In a recent tech talk, Anant Bhardwaj, Founder and CEO of Instabase, and Clemens Mewald, Head of Product at Instabase, discussed the limitations of LLMs and the strategies and technologies that organizations should use to overcome them.

Compound AI Systems Make LLMs More Reliable and Accurate

We’re often asked by customers why they can’t just use ChatGPT to understand and work with their documents. LLMs like GPT don’t always provide accurate answers tailored to your specific business needs. For example, GPT’s broad knowledge can result in overly generalized responses that miss the particular business context provided. 

This is where compound AI systems come in. Given this limitation of standalone LLMs, it’s important to build upon base models and integrate additional logic layers to enhance their AI capabilities. In addition to advanced retrieval-augmented generation (RAG), organizations need to also build content digitization, parsing, data cleaning, validation, human review, and other components. By adding all these additional layers to LLMs, models perform much better and produce reliable, useful results that organizations can trust. 

Overloading Context Windows Is Costly and Inefficient

Another major limitation of LLMs is their context window. While context windows are getting larger, there are a couple of issues when you input vast amounts of data into them. There’s the “lost in the middle” problem, where models are good at retrieving information from the beginning and end of the context window, but they’re really bad at doing so in the middle. Overloading context windows also gets expensive. 

However, compound AI systems, like the ones used in Instabase AI Hub, effectively address this constraint and improve efficiency. Compound AI systems also have diminishing costs as the number of pages or tokens increases. 

Large Models Are Usually Better Than Small Fine-Tuned Models

As models scale, they develop emerging abilities and are able to reason better. This is one of the reasons why developers keep on making larger models. Also, time to value is radically reduced since you don’t need to collect data, annotate it, and then train and fine-tune models. 

Fine-tuning small models has its place, but it’s not the right choice for most companies. The vast majority of use cases can be solved with zero or few-shot prompting and large foundational models. When you also consider effort, total cost of ownership, time to value, and other factors, there are very few situations where it makes sense to fine-tune a model. 

Open-Source Models Are More Trouble Than They’re Worth

With open-source models significantly improving over the past few years and even matching the performance of proprietary models, some think that proprietary models are no longer needed. However, organizations underestimate the cost of open-source models. Running an open-source model at the scale of something like ChatGPT would cost nearly $60 million a month, while the cost of proprietary models is heavily subsidized by companies like OpenAI. Additionally, running an open-source model yourself requires a lot of work that OpenAI and other companies have already handled. As a result, enterprises with massive volume and throughput won’t see much benefit with using open-source models. 

Also, don’t forget that with compound AI systems, model performance is only a very small part of the overall performance. If you’ve built a good compound AI system, you’ll actually experience comparable performance with different models, which means your model choice doesn’t really matter.

While these takeaways show how enterprise organizations should approach implementing LLMs in order to achieve the results they need, there’s more nuance to these topics than what we’ve covered here. Dive deeper into this discussion by watching the full webinar on demand here.