One for All: Generalist Foundation Model
for Cross-Sensor Skeleton Representation Learning

(a) Previous methods couple the architecture to a fixed joint count and rigid absolute positional embeddings, so each sensor needs its own isolated encoder and the resulting features are mutually incompatible — a sensor-specific representation.
(b) SOfA (Ours) introduces a fixed-size set of learnable Canonical Joint Slots S_c. An arbitrary sensor sequence enters only as a condition, and a single shared encoder dynamically fills the slots to emit a standardized, sensor-unified representation regardless of the original joint count.

SOfA establishes a sensor-unified representation within a teacher–student architecture, where both networks dynamically fill a fixed-size set of Canonical Joint Slots. Processing a masked view, the student encoder minimizes the canonical reconstruction error (ℒ_Cano) to match the slot representations of the unmasked teacher, alongside a global distillation loss (ℒ_DINO) that aligns high-level semantics. Because the varying sensor tokens W_p enter only as a dynamic condition, they are discarded after encoding and the model exposes exclusively the unified canonical representation W = [w_cls | W_c] to downstream tasks.

(Left) Semantic Joint Embedding (SJE). Absolute positional embeddings assign rigid numerical indices, so the same index refers to different anatomical joints across sensors — or to empty, zero-padded slots. SJE instead builds a prompt from each sensor's joint metadata ("A human skeleton joint of the {Left Wrist}") and encodes it with a pre-trained T5 text encoder, giving anatomically identical joints a consistent latent identity across every dataset and inheriting semantic relations between joint names.

(Right) Canonical Joint Slots (CJS). The class token, the canonical tokens and the condition patch tokens propagate jointly through masked self-attention, so the slots are filled by aggregating information from the visible joints. An attention mask assigns −∞ to zero-padded joint positions, so the final representation is computed strictly from valid skeletal information regardless of the input's original joint count.

Representative normalized skeletons from the ten heterogeneous datasets. Despite variations in joint count, tracking protocol and acquisition environment, all sequences share a common hip-centered frame and upright orientation after our unified preprocessing.

t-SNE of the slot assignments across sensors. The five slots autonomously converge toward the semantic centers of distinct anatomical clusters — the torso and the four limbs — rather than acting as a generic compression bottleneck. If the slots were merely compressing, their optimal count would be arbitrary; instead performance peaks at exactly five, matching the five major regions of the human body. The learned canonical space is anatomically meaningful.

Because SOfA pre-trains a single encoder on a combined corpus, NTU-60 and NTU-120 need care: NTU-120 contains every NTU-60 sequence. We pre-train on NTU-120 only, keeping just the sequences assigned to training by both protocols (X-Sub ∩ X-Set) — 25,053 sequences, only 22% of NTU-120.

The official NTU-120 X-Sub / X-Set and NTU-60 X-Sub test sets are then usable directly. NTU-60 X-View is the exception: it splits by camera ID while X-Set splits by setup ID, so some official X-View test sequences appear in the pre-training corpus. We remove all 5,392 overlapping sequences by sequence ID before evaluation, leaving 13,568 sequences reported as X-View^*. The filtering is completed before evaluation and does not depend on model predictions.