Keep up with the innovative tech transforming business
Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.
With AI companies rolling out models more gradually, transparency about each update has tended to suffer.
That “documentation drift” is one of the findings from a policy group called Americans for Responsible Innovation (ARI), which advocates for AI regulation. The report also found that closed models tended to be more tight-lipped about technical details, but that tech giants are generally more open than AI startups.
The findings were based on an analysis of seven different models from OpenAI, Meta, Anthropic, Google, and xAI across 21 different transparency metrics.
ARI policy analyst and report author David Robusto said there are a number of factors behind why companies might be reluctant to flesh out documentation details for every update. Doing so takes extensive time and resources, and the pace of progress makes it hard to maintain, he said. There’s also the possibility that rivals might reverse-engineer work based on certain details, Robusto said.
“There is a large competitive advantage to be gained from secrecy,” Robusto told Tech Brew. “Any sort of inch that is given by these companies can be picked up by their competitors. And so as a result, there is at least a partial incentive to maybe not keeping all of this stuff as up to date as they might want to, for example, if they were in a more traditional academic context.”
Clear technical details are important tools for policymakers and third parties attempting to understand how the latest frontier models work, especially in high-stakes areas like healthcare and defense, the report said. Big foundation models already tend to be opaque by nature because their massive size makes divining any rationale behind decision-making difficult.
Robusto said some straightforward fixes to some of these issues could come from regulations and industry-wide standards.
“There’s a conversation to be had about the merits of different types of mandatory disclosures. But there’s also a lot of low-hanging fruit here,” Robusto said. “Can we put together standards around things like making consistent the way these companies actually do benchmark evaluations and report on their methodology, as well as things like just standardizing the actual format of disclosures that these companies do?”
While President-elect Trump has vowed to roll back some of the Biden administration’s AI regulatory efforts, Robusto said he’s “cautiously optimistic” that transparency measures like these have bipartisan support.