The Rising Importance of Orchestration in Gen AI Development

At last year’s Microsoft Build conference, CTO Kevin Scott talked about the importance of orchestration when building generative AI apps. In the early days, each team would be creating their own layer. But through this process, Scott noticed many commonalities.

“One of the things that we did that greatly affected our ability to get these Copilots out to market at scale and to do more ambitious things was to decide that inside of Microsoft, we are going to have one orchestration mechanism that we will use to help build our apps,” he said. 

He did go on to mention that orchestration was far from a solved problem. But he said there would be much innovation in the category, such as from open source projects.

Orchestration in computing refers to the automated arrangement, coordination and management of complex computer systems, applications and services – similar to an orchestra conductor directing various musicians.

It involves coordinating multiple tasks and processes to work together seamlessly across different environments such as cloud platforms, data centers and software applications.

How orchestration works with generative AI apps

In simple terms, an application will have a request and response. An example is where a user can fill out a form and the information will be stored in a database. This can be handled in a single code file.

But with more complex applications, there will need to be orchestration to help manage different actions. This is common for generative AI applications that rely on techniques like Retrieval-Augmented Generation (RAG). It uses semantic search, such as with vector databases, to sift through and process data. This can help improve the accuracy and relevancy of the content generated.

“A developer can build a naive RAG application that calls an LLM once or twice and returns a response to a user’s question,” said Davor Bonaci, who is the CTO of DataStax. “But sophisticated RAG applications benefit from orchestration, where multiple interactions with the LLM and other systems are run and results are aggregated into the best possible answer to the user’s question.”

The platforms

Orchestration is a murky topic when it comes to generative AI. But generally, it is about coordinating and managing components, services and processes in the development and deployment of applications.

“Orchestration tools can abstract away many of the details of prompt chaining, interfacing with external APIs, retrieving contextual data from vector databases and maintaining memory across multiple LLM calls,” said Atindriyo Sanyal, who is the co-founder and CTO of Galileo. “Orchestration layers make sure all of a GenAI application’s moving parts work seamlessly together.”

There are various popular open source projects that help developers with the orchestration process. “The major players in the open source community are LangChain, LlamaIndex and Haystack,” said Iggy Gullstrand, who is the CEO of Triform. “There are a few up and coming as well, but these three are on my watchlist.”

The open source projects have attracted fervent and large communities. They have also gained the attention of investors.

Last week, LangChain announced a $25 million Series A round of venture capital, with Sequoia as the lead investor.

“Today, more than 50,000 LLM applications have been built using LangChain, with use cases ranging from internal apps to autonomous agents to games to chat automation to security scanners and more,” according to a Sequoia blog. “And while the platform is already a mainstay at weekend hackathons, we have also been excited to see many of our portfolio companies moving their LangChain use cases into production. Soon, your delivery order or customer support or gaming experience may be powered by LangChain.”

Traditional orchestration tools like Kubernetes or Docker Swarm can be useful too.

“While Kubeflow is an open-source platform designed for machine learning and MLOps within Kubernetes, Kubeflow Pipelines is a platform for orchestrating machine learning workflows from start to finish,” said David Coffey, who is the vice president of product management, Software Networking, and NS1 chief product officer at IBM.

“Then there is Open Data Hub, which is a full collection of open-source tools that uses the power of an OpenShift Container Platform to streamline the end-to-end GenAI application development lifecycle.”

The challenges

The category of generative AI development tools is nascent as many of them are less than a year old. These solutions also pose challenges for developers. They may need to use a combination of LangChain and LlamaIndex, along with different LLMs and SLMs, vector databases and context data.

“Given orchestration layers are yet another layer in the stack, they can create added complexity when it comes to evaluating your system, scaling experimentation, and debugging,” said Sanyal.

“This is part of the reason why some developers say they are a thin layer,  which does not really impact the overall functionality and performance of GenAI applications. But I believe orchestration layers are here to stay. People appreciate the convenience they afford and as these products become more mature, I expect developers to find more value in them.”