OpenAI applies new techniques to train models as performance plateaus: report

Microsoft-backed (NASDAQ:MSFT) OpenAI is applying new techniques to improve the performance of its upcoming large language model, code-named Orion, as it is reportedly not demonstrating the same leaps of improvement as prior models, according to a report by The Information.

Although Orion is outperforming ChatGPT-4 in terms of ability to complete tasks and answer questions, the improvements are allegedly much smaller than those made from GPT-3 to GPT-4, according to the report, which cited anonymous sources familiar with the matter.

A primary reason for the alleged slowdown in its latest artificial intelligence model is the rising scarcity of high-quality data available to train large language models, the report said. AI developers have already absorbed the bulk of publicly available data.

This has led to new models, such as Orion, being partially trained on AI-generated data, which is known as synthetic data, an OpenAI employee told The Information. This is prompting new models to behave similarly to prior models. One remedy is the use of human evaluators to test pre-trained models with various coding and problem-solving questions, followed up by human feedback.

To orchestrate this more hands-on approach to training LLMS, OpenAI and other AI developers are contracting out to other startups such as Scale AI and Turing AI, the report said.

“For general-knowledge questions, you could argue that for now we are seeing a plateau in the performance of LLMs,” said Ion Stoica, co-founder of Databricks, in the report. “We need factual data, and synthetic data does not help as much.”

Constraints in overall computing power are also an issue for the latest wave of AI models, said OpenAI CEO Sam Altman earlier this month.

“All of these models have gotten quite complex, and we can’t ship as many things in parallel as we’d like to,” Altman said. “We also face a lot of limitations and hard decisions about how we allocate our compute towards many great ideas.”

Late last month, OpenAI said it does not plan to release an AI model code-named Orion this year, refuting recent reporting on the company’s product roadmap.