On Benchmarks
Topics worth your time: a framework for understanding what CAD and PCB benchmarks measure today, rules from Skunk Works on getting to a prototype, and new funding for solar installation robots, battery systems, and geothermal wells.
P.S. We’re hard at work on our SF Tech Week event on 10/09 - apply to attend!
How Should We Evaluate Benchmarks?
An uncanny realization with the new frontier models, specifically Fable 5/5.1 and Astra 6, is that both are now capable (to varying degrees) of designing parts in CAD and routing PCBs. As such, a variety of new benchmarks have been established to measure these emerging capabilities like CAD Arena and EEBench, both of which are worth reading for their methodology and what they measure.
A fair criticism of benchmarks in hardware—and one that, if we were to guess, reflects the broader consensus—is that the true litmus test in our industry is seeing the design through production. That means generated 3D models need to survive manufacturing, with all the constraints on manufacturability and part production that come with it, and boards need to be fabricated without requiring a respin. Others have put the same sentiment a little less delicately.
With that in mind, how should we interpret benchmarks and what do they truly tell us about a model’s ability to design working hardware? Here’s our evolving perspective on treating benchmarks as rungs on a ladder.
Rung #1: Can it operate the compiler (CAD/CAM/CAE) software well?
Astra and Fable already perform well, and operation of these tools might be largely solved provided frontier labs prioritize the right training. Grok 4.7, for example, is expected to launch with training on troves of SpaceX’s engineering data.
Progress at this rung is also relatively quick and cheap to benchmark: CAD Arena evaluates both model correctness and feature tree editability, among other criteria.
Rung #2: Can it evaluate what it has designed is correct or not, relative to how a person does?
This distinction matters because human design involves reasoning in both design space (sketches, CAD, DFM, basic prototypes) and physical space (tolerances, DFM). No sufficiently complex design moves directly from design space into reality without iteration.
The ability of today’s models to “see” and have a representation of 3D geometric relationships is much improved, but by no means good. Some of these limitations are inherent to the harness: a hybrid of the human interface of CAD and APIs, rather than a pure symbolic system.
Rung #3: Can it see a design through manufacturing and verify that it works?
Operating the software and evaluating a design are necessary steps, but as mentioned above, the true qualification is whether the design works as intended. Can the model produce (or coordinate with others to produce) a physical part, and verify that it meets its functional specifications?
There’s plenty still to learn, and we’re seeing in real time how successfully these designs work in real operation (see here for WIP testing on a GPT-6 Astra/EasyEDA designed flight controller board).
This is where benchmarks need to move beyond the design itself and measure whether the resulting hardware works as intended. Following designs through manufacturing, assembly, testing, and iteration will be slower and more expensive, but those benchmarks are worth building. As we’ve seen in software, models can improve quickly against a clear, verifiable metric. It’s also important to note that mastery of a benchmark is only evidence of the capabilities it tests against.
Expecting current benchmarks to measure the last rung shouldn’t diminish the value of measuring the first. Most of the progress in the field felt like (and was) slop even compared to the beginning of 2026, but it’s now an uncanny valley and the implications are worth addressing.
TL;DR: Treat benchmarks as rungs on a ladder. “Vibe hardware” isn’t here, but models are getting better at parts of the work. The reality, as usual, is somewhere in the middle.
Interesting Chart: Get to the Prototype
On a related note, Skunk Works makes the case for getting to a prototype quickly.
“The last 10 percent of performance generates one-third of the cost and two-thirds of the problems.” — Norman R. Augustine, Augustine’s Laws
Presented by Xometry: Inside a Battlefield Refrigerator
A refrigerator designed for the battlefield has to do more than stay cold. Working with Xometry, Delta Development Team’s APRU had to keep blood between 36°F and 43°F while withstanding temperature fluctuations, chemicals, vibration, and even the G-forces of being airdropped from a plane.
As Delta moved from a functional prototype toward production, Xometry supported the full range of manufacturing — turning a demanding set of requirements into a manufacturable medical device.
See how Delta engineered the APRU for conditions far beyond the lab.
Manufacturing & Startup News
OpenAI acquired GLASS Imaging, a Los Altos startup building AI software that improves smartphone image quality, in a deal valuing the company at more than $300M.
Planted raised a $31.8M Series B to expand its fleet of robots that install solar panel foundations along natural terrain, including slopes that conventional solar layouts leave unused.
Branch Energy raised a $33M Series B to expand its battery systems into new markets, deploying batteries at commercial buildings that store electricity for use on-site or sale to the grid.
Mazama Energy raised a $135M Series B to develop horizontal geothermal wells in Oregon’s superhot rock, targeting 15 MW per well and initial power generation in 2027.
Tools From Our Partners
Aris Machina Protos – Design & validate anything, with the AI-native workspace for industrial science and hardware R&D.
SolidWorks for Startups – Full access to SolidWorks’ product development tools, free for early-stage startups.
Zoo – AI-native CAD with a modern geometry engine, readable code, and editable feature trees for precise, editable models.
Access Every Link We’ve Ever Shared 🎞️
Refer three friends to Hardware FYI and unlock our full archive.
Missed an issue? Hunting for that one link? Every tool, guide, and article we’ve ever shared is in here.
P.S. If you find something interesting—an article, tool, funding round, project, really anything—send it our way!
P.P.S. 20,000+ engineers, founders, and operators read this newsletter. If you want to reach them, get in touch.