How can enterprises lower costs in GenAI for projects at scale? As projects go from prototypes to scale, many enterprises are starting to see that economics are a key issue, sometimes a key gating issue.
There’s multiple parts of the stack that come into play. And enterprises probably need to push on all of them to make the economics work.
1. Option of GenAI at the UX/UI layer
Some UIs generate an output by default. This is the case with some LLM-generated text, for example. In contrast, the experience could be one where the user is given the choice to see the generated text by clicking a button. By not producing the text by default and instead allowing only users who WANT to see the GenAI experience to get it, one could reduce the compute requirements by 2x or more.
2. Reduce costs in GenAi by right-sizing the quality of the LLM model
There’s no point in always driving in a high-end car. There are different quality levels for LLM models with vastly different compute requirements (5x-50x differences). Not only can one choose a lower quality level for different use cases, but one can also even change the quality level for different customer segments in the same use case. This can lower the compute requirements substantially with the trade off of quality and / or somewhat more code complexity (if you’re going to be doing segmentation).
3. Proprietary vs open source models
Open source LLM models are increasing in quality at an astounding speed. So one can also do the comparison of proprietary vs open source and see if this lever will help in lowering the cost. Not only are ipen source LLMs increasing in quality, but also the infrastructure and ease of deployment is growing with the option of leveraging open source models in many commonly used clouds.
4. Quantization of the LLM processing
There is a technique one can use to reduce the fidelity of the computation the LLM needs to do called “quantization“. This means that instead of using all the “bits” in the computation, one compresses the computation into lower number of “bits”. Sometimes there is some loss in the quality but this can be managed with various techniques and the trade off can be worth it.
5. Traditional finops
Being smart about how you use your cloud resources always pays dividends. You might be oversizing your compute resources and having substantial idle capacity. This is more traditional cloud frugality that many enterprises are wise too already. So there is no point in spending much time on this.
And then there’s ways to lower costs around your vector database. In short, there are various parts of the stack that can be explored to lower the costs of your compute costs for your GenAI project at scale. Any glaring ones you think that I missed?
As always, views here are personal and not related to my employment.