Mesh Tokenization for Transformers

The MismatchA Mesh Is a Graph, a Transformer Reads a Sequence

A mesh can be reindexed or its faces reordered without changing the surface. A naive token stream treats those equivalent arrays as different sentences. Canonicalization reduces this ambiguity but cannot remove it entirely for symmetric or disconnected shapes.

QuantizationCoordinates Become a Finite Vocabulary

Normalize the object into a stable frame, clamp it to a known box, and quantize each coordinate into integer bins. More bins preserve precision but enlarge the modeling problem. Fewer bins make sequences easier while creating stair-step error and merging nearby vertices.

import numpy as np

def quantize_vertices(vertices, bits=9):
    levels = (1 << bits) - 1
    lo = vertices.min(0)
    hi = vertices.max(0)
    span = np.maximum(hi - lo, 1e-8)
    normalized = (vertices - lo) / span
    tokens = np.rint(normalized * levels).astype(np.int32)
    return tokens, lo, hi

def dequantize(tokens, lo, hi, bits=9):
    levels = (1 << bits) - 1
    return lo + (tokens / levels) * (hi - lo)

A production tokenizer also has explicit start, stop, face-boundary, component, and invalid-transition behavior. It must decide how normals, materials, quads, holes, and UV seams enter—or stay outside—the geometry sequence.

MeshGPTLearn a Vocabulary of Local Geometry

MeshGPT encodes local geometry and topology with graph convolutions, quantizes latent embeddings into a learned vocabulary, and trains a decoder-only transformer to predict the next embedding. A mesh decoder reconstructs triangles from those tokens. The important shift is that tokens can represent learned local structure rather than raw coordinate integers alone.

AdjacencySpend Tokens on New Information

Faces in a coherent surface often share an edge with the previous face. MeshAnything V2’s Adjacent Mesh Tokenization exploits that continuity: where possible, the next adjacent triangle can be specified by its new vertex rather than repeating all three. The authors report roughly halving sequence length on average and doubling the supported face limit without increasing compute.

Pedagogical TraversalWalk Faces Through Shared Edges

def adjacency_walk(faces):
    edge_to_faces = {}
    for fi, (a, b, c) in enumerate(faces):
        for edge in [(a, b), (b, c), (c, a)]:
            edge = tuple(sorted(edge))
            edge_to_faces.setdefault(edge, []).append(fi)

    visited, order, stack = set(), [], [0]
    while stack:
        fi = stack.pop()
        if fi in visited:
            continue
        visited.add(fi)
        order.append(fi)
        a, b, c = faces[fi]
        neighbors = []
        for edge in [(a, b), (b, c), (c, a)]:
            neighbors += edge_to_faces[tuple(sorted(edge))]
        stack.extend(n for n in neighbors if n not in visited)
    return order

This is only a traversal, not MeshAnything’s released AMT implementation. Real tokenizers define component restarts, consistent winding, nonmanifold cases, deterministic tie-breaking, and detokenization constraints.

Failure ModesSyntax Can Be Valid While Geometry Is Bad

A decoded sequence may repeat faces, flip normals, open holes, self-intersect, or create nonmanifold fans. Apply grammar masks during decoding where possible, then run geometric validation. Conditioning on point clouds or images constrains broad shape; it does not guarantee printable topology.

Sequence BudgetWhy Compact Topology Matters

Attention cost grows sharply with sequence length. A token spent repeating a known shared vertex cannot describe a new feature. Efficient traversal, local vocabularies, hierarchy, and artist-like topology are therefore model architecture—not post-processing trivia.