Robotic Automations

While investors were preparing to go nuclear after Sam Altman’s unceremonious ouster from OpenAI and Altman was plotting his return to the company, the members of OpenAI’s Superalignment team were assiduously plugging along on the problem of how to control AI that’s smarter than humans.

Or at least, that’s the impression they’d like to give.

This week, I took a call with three of the Superalignment team’s members — Collin Burns, Pavel Izmailov and Leopold Aschenbrenner — who were in New Orleans at NeurIPS, the annual machine learning conference, to present OpenAI’s newest work on ensuring that AI systems behave as intended.

OpenAI formed the Superalignment team in July to develop ways to steer, regulate and govern “superintelligent” AI systems — that is, theoretical systems with intelligence far exceeding that of humans.

“Today, we can basically align models that are dumber than us, or maybe around human-level at most,” Burns said. “Aligning a model that’s actually smarter than us is much, much less obvious — how we can even do it?”

The Superalignment effort is being led by OpenAI co-founder and chief scientist Ilya Sutskever, which didn’t raise eyebrows in July — but certainly does now in light of the fact that Sutskever was among those who initially pushed for Altman’s firing. While some reporting suggests Sutskever is in a “state of limbo” following Altman’s return, OpenAI’s PR tells me that Sutskever is indeed — as of today, at least — still heading the Superalignment team.

Superalignment is a bit of a touchy subject within the AI research community. Some argue that the subfield is premature; others imply that it’s a red herring.

While Altman has invited comparisons between OpenAI and the Manhattan Project, going so far as to assemble a team to probe AI models to protect against “catastrophic risks,” including chemical and nuclear threats, some experts say that there’s little evidence to suggest the startup’s technology will gain world-ending, human-outsmarting capabilities anytime soon — or ever. Claims of imminent superintelligence, these experts add, serve only to deliberately draw attention away from and distract from the pressing AI regulatory issues of the day, like algorithmic bias and AI’s tendency toward toxicity.

For what it’s worth, Sutskever appears to believe earnestly that AI — not OpenAI’s per se, but some embodiment of it — could someday pose an existential threat. He reportedly went so far as to commission and burn a wooden effigy at a company offsite to demonstrate his commitment to preventing AI harm from befalling humanity, and commands a meaningful amount of OpenAI’s compute — 20% of its existing computer chips — for the Superalignment team’s research.

“AI progress recently has been extraordinarily rapid, and I can assure you that it’s not slowing down,” Aschenbrenner said. “I think we’re going to reach human-level systems pretty soon, but it won’t stop there — we’re going to go right through to superhuman systems … So how do we align superhuman AI systems and make them safe? It’s really a problem for all of humanity — perhaps the most important unsolved technical problem of our time.”

The Superalignment team, currently, is attempting to build governance and control frameworks that might apply well to future powerful AI systems. It’s not a straightforward task, considering that the definition of “superintelligence” — and whether a particular AI system has achieved it — is the subject of robust debate. But the approach the team’s settled on for now involves using a weaker, less-sophisticated AI model (e.g. GPT-2) to guide a more advanced, sophisticated model (GPT-4) in desirable directions — and away from undesirable ones.

A figure illustrating the Superalignment team’s AI-based analogy for aligning superintelligent systems. Image Credits: OpenAI

“A lot of what we’re trying to do is tell a model what to do and ensure it will do it,” Burns said. “How do we get a model to follow instructions and get a model to only help with things that are true and not make stuff up? How do we get a model to tell us if the code it generated is safe or egregious behavior? These are the types of tasks we want to be able to achieve with our research.”

But wait, you might say — what does AI guiding AI have to do with preventing humanity-threatening AI? Well, it’s an analogy: The weak model is meant to be a stand-in for human supervisors while the strong model represents superintelligent AI. Similar to humans who might not be able to make sense of a superintelligent AI system, the weak model can’t “understand” all the complexities and nuances of the strong model — making the setup useful for proving out superalignment hypotheses, the Superalignment team says.

“You can think of a sixth-grade student trying to supervise a college student,” Izmailov explained. “Let’s say the sixth grader is trying to tell the college student about a task that he kind of knows how to solve … Even though the supervision from the sixth grader can have mistakes in the details, there’s hope that the college student would understand the gist and would be able to do the task better than the supervisor.”

In the Superalignment team’s setup, a weak model fine-tuned on a particular task generates labels that are used to “communicate” the broad strokes of that task to the strong model. Given these labels, the strong model can generalize more or less correctly according to the weak model’s intent — even if the weak model’s labels contain errors and biases, the team found.

The weak-strong model approach might even lead to breakthroughs in the area of hallucinations, claims the team.

“Hallucinations are actually quite interesting, because internally, the model actually knows whether the thing it’s saying is fact or fiction,” Aschenbrenner said. “But the way these models are trained today, human supervisors reward them ‘thumbs up,’ ‘thumbs down’ for saying things. So sometimes, inadvertently, humans reward the model for saying things that are either false or that the model doesn’t actually know about and so on. If we’re successful in our research, we should develop techniques where we can basically summon the model’s knowledge and we could apply that summoning on whether something is fact or fiction and use this to reduce hallucinations.”

But the analogy isn’t perfect. So OpenAI wants to crowdsource ideas.

To that end, OpenAI is launching a $10 million grant program to support technical research on superintelligent alignment, tranches of which will be reserved for academic labs, nonprofits, individual researchers and graduate students. OpenAI also plans to also host an academic conference on superalignment in early 2025, where it’ll share and promote the superalignment prize finalists’ work.

Curiously, a portion of funding for the grant will come from former Google CEO and chairman Eric Schmidt. Schmidt — an ardent supporter of Altman — is fast becoming a poster child for AI doomerism, asserting the arrival of dangerous AI systems is nigh and that regulators aren’t doing enough in preparation. It’s not out of a sense of altruism, necessarily — reporting in Protocol and Wired note that Schmidt, an active AI investor, stands to benefit enormously commercially if the U.S. government were to implement his proposed blueprint to bolster AI research.

The donation might be perceived as virtue signaling through a cynical lens, then. Schmidt’s personal fortune stands around an estimated $24 billion, and he’s poured hundreds of millions into other, decidedly less ethics-focused AI ventures and funds — including his own.

Schmidt denies this is the case, of course.

“AI and other emerging technologies are reshaping our economy and society,” he said in an emailed statement. “Ensuring they are aligned with human values is critical, and I am proud to support OpenAI’s new [grants] to develop and control AI responsibly for public benefit.”

Indeed, the involvement of a figure with such transparent commercial motivations begs the question: Will OpenAI’s superalignment research as well as the research it’s encouraging the community to submit to its future conference be made available for anyone to use as they see fit?

The Superalignment team assured me that, yes, both OpenAI’s research — including code — and the work of others who receive grants and prizes from OpenAI on superalignment-related work will be shared publicly. We’ll hold the company to it.

“Contributing not just to the safety of our models but the safety of other labs’ models and advanced AI in general is a part of our mission,” Aschenbrenner said. “It’s really core to our mission of building [AI] for the benefit of all of humanity, safely. And we think that doing this research is absolutely essential for making it beneficial and making it safe.”