Taming the LLM Beast

Large language models (LLMs) are the mythical beasts of AI, capable of conjuring human-quality text, translating languages and producing different kinds of creative content, as if by magic. But like their mythical counterparts, building an LLM from scratch is no easy feat. It's a difficult journey fraught with technical challenges, from data collection and preparation to training and fine-tuning the model. This isn't a quest for the faint of heart.

For the brave researchers and engineers willing to take on this challenge, this article serves as an essential field guide on how to take on this quest, tame the LLM beast and build a model from scratch.

1. Data 

LLMs consume vast amounts of data and multilingual support is scarce, so building a multi-stage data pipeline takes time. Data lineage tracking tools help teams understand data origin and changes for quality and reproducibility. It is also important to track various data versions across different preprocessing steps. Data versioning tools such as Data Version Control (DVC) can help maintain consistency and manage updates.

Data pipelines transform the raw data into various formats for better processing. Keeping track of the instructions for data pipeline versions helps teams experiment with different approaches on existing data sets or new versions and revert to the old recipe when they don’t. Open-source tools like Spark empower teams to scale the execution of data processing across a large number of computers. Others like Airflow and Prefect can orchestrate complex data pipelines and are essential for a robust data preparation process. Nebius’ own TractoAI is an end-to-end solution for data preparation and exploration, which helps anyone daring to take their first steps on this journey and connects these different capabilities.  

Related:The Future Workforce will be Powered by AI

2. Experimentation

The next step on the hero’s journey lies in experimenting with tools to help extend the utilization of what seems like a good process to work at a greater scale. There are many ways things can go wrong in trying to scale up a new LLM, including problems with the training data, the choice of LLM models and how they are scaled across multiple computers. Developers must consider scaling the training process across multiple computers, assessing data quality and validating model architectures.

Teams need to maintain detailed records for reproducibility and track how changes in the training process affect the final results. Tools such as MLFlow or Weights and Biases can be used at this stage. When experimenting, researchers need to focus on two key aspects – whether the idea works and whether the idea scales. With that in mind, researchers want to start small – on as few as eight GPUs – to test feasibility. If this works, they can scale it up to 32-64 GPUs for a day to validate scalability. Next, scale it up to 128 or more GPUs for week-long training to ensure robustness.

Related:Conquering the Fear of Embracing AI

3. Pre-training

Pre-training requires a Herculean amount of computational power, often forcing developers to hunt for external clusters. Subtle differences in data center architectures can sometimes slow or break in different ways, introducing stability issues that cause time-consuming and expensive restarts.

There are many different ways to run batches of data across GPU clusters and the options can vary depending on each cloud provider’s approach. The best architectures use Nvidia’s Collective Communication Libraries (NCCL), which allows GPUs to share updates using a peer-to-peer approach. This keeps each compute node on the same page with less networking overhead. Teams should consider agreeing on a proof of concept, rigorously testing the cluster performance on a variety of real workloads and tests, e.g., NCCL, then, if the tests pass, shortlist the most reliable providers and move to a long-term contract  

4. Checkpoint

It’s important to save intermediate checkpoints every hour on large training runs in case a training run crashes. This ensures you can restart from where you left off without requiring days or weeks for a large run. You don’t necessarily need to save each one. Still, it's also a good idea to save daily checkpoints in case some of the training assumptions about model architecture lead to problems like gradient explosion.

Also, you should explore model and infrastructure architectures that allow you to back up checkpoints from RAM during the training process, which allows the training process to continue during backup. Model sharding and different combinations of data and model parallelism can improve the backup process. Open-source tools like Jax Orbax or PyTorch Lightening can help automate the checkpoint process.  In addition to this, utilizing storage, which is optimized for checkpoints is key.

5 Achieving Alignment and Optimal Performance

The final stage involves further experimentation but with a lighter computational footprint. It's important to track and benchmark experiments to achieve successful alignment and optimal performance. It is also important to use universal methods that can streamline the alignment process.

Taming the LLM beast doesn’t need to be the twelve labors of Hercules. While it requires careful consideration of the many steps to build models that provide good results for new use cases, languages and domains, it is a feat that can be accomplished by mortal men and women. As with all quests, what’s needed is a plan – in this case, one that makes sure data preparation, model validation and experimentation, pre-training on big clusters, implementing checkpoints and securing alignment are considered, to ensure that the model is robust, efficient and fair, ultimately leading to a more reliable and impactful AI platform.