Even after AI developers spend millions on computing power to train a model on billions of human-written words, it still takes an intensive regimen of human feedback and fine-tuning before the system is ready to interact with humans.
That process is called post-training, and there’s a big gap in the level of sophistication between open-source AI and often-better-funded closed rivals, according to the Allen Institute for AI (AI2). It’s why the nonprofit research lab is releasing its latest family of LLMs, Tülu 3, along with a post-training toolkit encompassing datasets, code architecture, and an evaluation framework to help other open-source developers replicate the regimen the team used for those models.
Open models like Meta’s Llama family, Mistral AI, and Google Gemma are especially useful for companies that need more control over models and businesses dealing with sensitive information. But AI2 said even most open models don’t release full post-training details.
“[Post-training] is a challenging process,” Hannaneh Hajishirzi, senior director of NLP research at AI2, told Tech Brew. “The big companies have figured it out. The open-source community, I would say, haven’t figured this out. Why? Because it’s a big effort.”
Post-training also could become more important as Big Tech companies have reportedly hit walls in the gains they see in the initial pretraining process.
A big undertaking: Most people familiar with AI training know about pretraining, when the model ingests huge troves of data, according to Hajishirzi.
“When the model comes out of the pretraining phase, it is not designed to do complex tasks like math reasoning, or it is not able to follow human instructions precisely, or it is not safe,” she told us.
Post-training adds this kind of polish, but it’s an intensive process that requires multiple teams working together. One team might focus on evaluating the model and seeing where it needs improvement, another curates datasets for specific stages of post-training, and yet another could build training infrastructure, Hajishirzi said. “All these things need to be done in tandem.”
Keep up with the innovative tech transforming business
Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.
Memory lapses: Even dicier is the risk that post-training can lead the model to lose other important previously obtained knowledge, an outcome AI researchers call “catastrophic forgetting.”
“So for example, if you teach the model how to code or…answer complex science questions, it would forget doing simple question answering, like, ‘Who is Barack Obama?’ or writing a poem, the type of tasks that language models do these days,” Hajishirzi said.
The Tülu toolkit shows developers how to balance core skills like coding, reasoning, safety, and multilingual capabilities through post-training. It also allows for customization; for instance, a company might focus the model on answering science questions, with less regard for safety if only used internally, Hajishirzi said.
“You can pick the pieces of data that are useful for your own goals,” she said. “You can use the evaluation settings that we have developed for those particular core skills, and then mix and match and then train the model to follow these recipes based on your need.”
Tülu 3 also includes an online demo playground. As the model rolls out, Hajishirzi said the team will watch to see how the AI is used in unexpected ways and how it might be improved.
“This is what we are hoping to get—what type of queries we are not aware of or we are not prepared for, and then collectively, we would start focusing on using the same pipeline to add new capabilities to our system,” she said.