Running TRELLIS.2 on a Mac (1/3): Porting a CUDA-Only Image-to-3D Model to Apple Silicon

Series: TRELLIS.2 on Apple Silicon (1/3) Part 1 · Porting (this post) — Part 2 · Measuring and Dead Ends → — Part 3 · What Actually Won →

This is the story of getting a model that turns a single photo into a fully textured 3D mesh to run on a Mac — with no NVIDIA GPU in sight. Part 1 is about one thing: how I got a CUDA-only model to stand up on Apple Silicon in the first place.

What TRELLIS.2 is, and why running it on a Mac is fun

TRELLIS.2 is an image-to-3D model from Microsoft Research. Feed it a single photo and it spits out a GLB file: a mesh with 400K–500K+ vertices and fully baked PBR textures — base color, metallic, roughness, the works. The quality is best-in-class for this kind of thing. There's just one catch. It's CUDA-only. It's bolted tightly to NVIDIA-exclusive libraries like flash_attn, flex_gemm, nvdiffrast, and o_voxel — so on a Mac, it won't even get past the import statements.

So this project is about running it on Apple Silicon instead — on top of PyTorch's MPS (Metal Performance Shaders) backend and the Metal stack. Shivam P Kumar's trellis-mac was the starting point for the CUDA→MPS port; from there it forked off into its own independent project, trellis-silicon, where I cleaned it up into an installable Python package and layered a measurement-first performance campaign on top. The performance story belongs to Parts 2 and 3; this post covers the step before all that — the port itself.

Why is running it on a Mac interesting? Because it's not a case of "swap the CUDA calls for a device variable and you're done." You have to replace the CUDA-only kernels one by one with Metal ports or pure-PyTorch stand-ins, and the stack you end up with has completely different performance characteristics from the original. Above all, the performance intuition you built up on CUDA barely transfers to MPS. That's the real subject of this series — but first, let's just get the thing to float at all.

Don't fork the original — an idempotent patcher architecture

The very first principle I settled on was: "don't fork the upstream code." TRELLIS.2 is a large codebase, and it keeps getting updated. The moment you copy the whole thing and start editing it in place, keeping up with upstream changes becomes its own private hell.

So in this project, TRELLIS.2/ is not tracked by git. Instead, it works like this:

1. setup.sh clones upstream pinned to a specific commit (the commit is pinned for reproducibility, and you can override it with the TRELLIS2_COMMIT environment variable).
2. trellis-silicon-patch (the patcher) rewrites that source in place via string substitution — swapping CUDA-only paths for Apple Silicon ones.
3. Every patch is guarded by a marker string. Because the patcher checks that marker to see whether a spot has already been patched, you can re-run it as many times as you like and get the same result (idempotent).

In other words, what lives in our repo isn't the original code — it's a collection of idempotent transformation rules that say "transform the original this way." Upstream stays clean, and bumping to a new version is just a matter of changing the clone commit. The actual patcher is split into several modules by concern (path resolution, device substitution, mesh processing, attention, loading, stub installation, and so on), all orchestrated with a single python -m trellis_silicon.patches.

Swapping out the CUDA libraries one at a time

The heart of the port comes down to "replace each CUDA-only dependency with something that runs on Apple Silicon." Here's how I handled the five core ones.

On top of that, every .cuda() call hardcoded throughout the code was patched to use the active device instead. A few of these are worth expanding on.

Sparse 3D convolution. This is a submanifold sparse convolution: it builds a spatial hash of the active voxels, then for each kernel position it gathers neighbor features → matmul with the weights → scatter-adds them back. The neighbor maps are cached per tensor. For environments where the Metal port (mtlgemm) doesn't behave, there's also a pure-PyTorch fallback (conv_none.py).

Mesh extraction. The original pulls the mesh out of a dual voxel grid using a CUDA hashmap. I reimplemented this on top of Python dicts — coordinate→index tables, connected voxels per edge, quad triangulation based on normal alignment. (Spoiler: this pure-Python implementation makes a return appearance in Part 3. It was slow.)

Attention. I bolted an SDPA backend onto the sparse attention module. It pads the variable-length sequences into a batch, runs them through torch.nn.functional.scaled_dot_product_attention, and then un-pads. Whether or not this was the right call becomes the central argument of Parts 2 and 3.

Texture baking. By default it uses @pedronaugusto's Metal stack (mtldiffrast, mtlbvh, mtlmesh, plus his CPU fork of o_voxel) to expose to_glb. The mesh the decoder emits is heavy — on the order of a million faces — and the Metal BVH (Bounding Volume Hierarchy — a tree structure that accelerates ray/triangle intersection tests) builder is known to be unstable on large inputs, so before baking I pre-reduce it to roughly 200K faces with fast_simplification. (That "unstable on large inputs" premise also gets re-examined in Part 3 — whether it's actually true.) If there's no Metal toolchain, a pure-Python fallback (xatlas UV unwrap + scipy cKDTree inverse-distance weighting) steps in instead.

Some things I gave up on entirely. Hole filling at decode time needs cumesh, but its Metal port segfaults on decoder-sized meshes, so I skipped it. That means the output mesh can be left with small holes. It's not a perfect port — it's a port that runs.

First generation

Once the whole stack is standing, it comes down to a single command.

trellis-silicon photo.png

An image goes in; it pulls conditioning with DINOv3, samples the sparse structure, samples the shape and texture SLat (sparse latent — a latent representation that holds compressed shape and texture information per voxel), decodes twice with the VAE, extracts the mesh, bakes the texture with Metal, and exports a GLB. Run it on a sample image from the public repo (the Metropolitan Museum's CC0 public-domain piece "Brighella on a pedestal") and out comes a textured mesh with around 500K vertices and around a million triangles. No NVIDIA GPU — all of it on the Mac's GPU.

That's the port. But run it once and the very next question jumps out at you. It's slow. Even warm on an already-warmed-up machine, loading takes over 100 seconds and generation takes over 3 minutes. So where's the bottleneck?

At the time, I thought I knew where the bottleneck was. The README even said, "the biggest remaining bottleneck is attention." How that confidence got shattered five times in a row is the story of Part 2: Measuring and Dead Ends.

All the code and benchmark scripts are public → github.com/sanchez-kim/trellis-silicon