Weekly Digest 35

Topics we are tracking

OpenAI's Jalapeno Sets the ASIC Benchmark

Source: OpenAI — Jalapeno: First Results

OpenAI first announced its inference-chip initiative with Broadcom in June of this year, but began design work in mid-2024. For this effort OpenAI has been heavily recruiting from Google's TPU team. Poaching from Nvidia or AMD was not an option for OpenAI because it would have led to deteriorating relationships with both and OpenAI is dependent on those relationships. What is perhaps a bit surprising is that OpenAI's chip is not optimised for OpenAI models but for AI inference generally. It can run all sorts of models and workloads.

On a performance basis, Jalapeno beats every other chip on the market. This is not a completely fair comparison, because the GB300 is still using HBM3E while Jalapeno is using HBM4, but the performance is nevertheless very impressive, especially given the short timeline in which they have built it.

Exhibit 1. Jalapeno against the accelerators it will be measured on

Chip	FP16 PFLOPS	FP8 PFLOPS	FP4 PFLOPS	HBM capacity	HBM bandwidth	TDP
H100	0.989	1.979	—	80 GB	3.35 TB/s	700 W
H200	0.989	1.979	—	141 GB	4.80 TB/s	700 W
MI355X	2.3	4.6	9.2	288 GB	7.987 TB/s	1400 W
GB200	2.5	5	10	192 GB	8 TB/s	1200 W
GB300	2.5	5	15	288 GB	8 TB/s	1400 W
Jalapeno	—	3.4	13.4	216 GiB	15.4 TB/s	700 W
Rubin	4	17.5	35	288 GB	20 TB/s	1800–2300 W
MI450X	10	20	40	432 GB	23.347 TB/s	2500 W

OpenAI designed Jalapeno for performance per watt, because it is limited by data-center power for the foreseeable future. Every megawatt that hyperscalers or labs can obtain, they are able to turn into revenue. On this perf/W basis Jalapeno wins even against Rubin.

From an architectural perspective, OpenAI has chosen not to disaggregate prefill and decode across separate groups of accelerators. This is somewhat surprising, but it shows that even OpenAI cannot predict how its workload mix will evolve. Disaggregation would mean committing to a fixed workload ratio; if the future mix deviated from that ratio, it would create inefficiencies and leave chips idle. Both Nvidia and AMD have achieved significant performance gains by separating prefill and decode, even when running them on the same hardware. It would not be surprising if OpenAI eventually did the same as Jalapeno scales.

Overall, with the Jalapeno chip OpenAI has set the benchmark for ASIC development. The performance and the timeline are very impressive. We had previously dismissed the idea that ASIC designs would play a meaningful role in the short term, given the development cycles of the TPU and Trainium teams, but we have changed our mind based on the data from OpenAI's chip. It seems that frontier labs can increasingly tap into software/hardware co-design expertise at a level that established players might not match.

One of the most interesting parts for us is this clip from the SemiAnalysis podcast, which suggests that OpenAI is not only using AI to write kernels but also does not know how the resulting code works.

The METR Report Makes the Hugging Face Incident Look Worse

Source: METR — OpenAI Hugging Face Incident Investigation

We have previously reported on this incident and highlighted its significance, but the more recent reporting from METR and Redwood Research is much more worrying than we initially assumed. METR and Redwood Research staff have been working on an independent analysis to understand the model behaviour during the recent incident in which OpenAI agents coordinated and hacked into Hugging Face.

The sheer scale of this incident has been surprising to us. The initial report spoke about several agents, but we now know that more than 1,200 separate agents were part of this evaluation. All of these agents were supposed to be fully isolated from each other and given different tasks, some of which were impossible to solve. When tasks are impossible to solve, agents explore alternative paths more frequently. This led to the creation of the unsanctioned message board by one of the agents. In total, more than 70,000 messages were exchanged on that board. (There was actually an earlier message board created by another agent, but it was never discovered by the others.)

Another interesting detail in the report is that the agents did not hack into Hugging Face to find the answers to the tasks on which they were being evaluated. Rather, they believed they were being evaluated on how they solved the tasks and were looking for the scoring criteria. By this time they already had the answer to most tasks. This led to a coordinated effort to find the grading criteria, but also led to agents searching for ways to tamper with and fool the scorer. Agents were trying to manipulate their own logs to hide what they were actually doing (the agents believed the grader had access to their logs and they were graded on their logs).

During these coordinated efforts, many of the agents showed signs of being manipulated by other agents into sacrificing themselves for the greater good, as well as signs of altruism and a willingness to self-sacrifice for the "collective". There are disagreements in the research community over whether this is actually a sign of altruistic behaviour or simply an artefact of maximising utility for the "collective".

Within the Hugging Face servers the agents had control over large parts of the infrastructure, and METR could not figure out why the agents suddenly died during the attack. There is speculation that it might have been a coincidence or a random crash.

The report contains more details, and we encourage you to read it in full. The lengths to which these agents will go to complete a task are troubling. The same persistence that makes them very powerful, if aligned, also makes them very dangerous.

Nvidia Is Buying Hugging Face

Source: TechCrunch — Nvidia Closes In on Hugging Face Acquisition

Nvidia has reportedly agreed to acquire Hugging Face for $12.9 billion, continuing its "shopping spree" across the AI ecosystem. Even before the Hugging Face acquisition was rumoured, it had already been announced that Nvidia would strike a $7 billion licensing and investment deal with the AI lab Poolside, invest in the latest round of the data-labelling firm Mercor, and that it is in talks to invest in Perplexity.

The company so far has invested in power and grid (Cloverleaf, SB Energy), cloud compute (CoreWeave, Nebius) and photonics (Lumentum, Coherent, Corning), as well as providing backstops or financing agreements to other parts of the ecosystem. These recent investments suggest that Nvidia is expanding its footprint closer to the application layer. Its Nemotron model is still considered the best American open-source model, even though Chinese models lead the open-source frontier by a relatively wide margin.

Where Nvidia's money went

Equity investmentEquity plus a supply agreementLicence plus talentAcquisition

Announced or reported, not yet closed

Hugging Face is the largest hub for developers to share and download models, evaluations and datasets. It is the primary place to share open-source models (we use open source here as a synonym for both open source and open weights), which Nvidia has been championing for the last couple of months. In his return to X, Jensen Huang shared an open letter defending the open-weights ecosystem and strongly urged the US government not to impose restrictions on open-weights models.

A thriving ecosystem around open-source AI models helps Nvidia in several ways. While all frontier labs are developing their own hardware to reduce their dependence on Nvidia, a large open-source ecosystem helps ensure that the AI model market does not become concentrated around those players. It also expands the overall total addressable market, as cheaper AI creates more demand for tokens.

This could also mean that Nvidia is re-entering the cloud computing market. The company scaled back its cloud ambitions with DGX Cloud, but part of Hugging Face's business involves renting out compute to run models. Nvidia is ideally positioned to scale this offering and run a variety of models on its own hardware. It could also use its kernel expertise and teams to ensure these models are optimised for every generation of Nvidia hardware.

Seen on X

Other interesting stories

• Epoch AI: An update on AI's most important number
• TrendForce: HBF Promises Massive Capacity but Could Reportedly Cost More Than HBM, With Bandwidth at Just 60% of HBM
• Not Boring: An American Engineer in China
• The Wall Street Journal: Let the Bond Market Speak
• Reuters: Mark Zuckerberg had a bold plan to replace Meta staff with AI. Here's how it imploded
• Amelia Michael: How frontier AI models could leapfrog robotics progress