SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition

Skeleton-based action recognition, which classifies human actions based on the coordinates of joints and their connectivity within skeleton data, is widely utilized in various scenarios. While Graph Convolutional Networks (GCNs) have been proposed for skeleton data represented as graphs, they suffer from limited receptive fields constrained by joint connectivity. To address this limitation, recent advancements have introduced transformer-based methods. However, capturing correlations between all joints in all frames requires substantial memory resources. To alleviate this, we propose a novel approach called Skeletal-Temporal Transformer (SkateFormer) that partitions joints and frames based on different types of skeletal-temporal relation (Skate-Type) and performs skeletal-temporal self-attention (Skate-MSA) within each partition. We categorize the key skeletal-temporal relations for action recognition into a total of four distinct types. These types combine (i) two skeletal relation types based on physically neighboring and distant joints, and (ii) two temporal relation types based on neighboring and distant frames. Through this partition-specific attention strategy, our SkateFormer can selectively focus on key joints and frames crucial for action recognition in an action-adaptive manner with efficient computation. Extensive experiments on various benchmark datasets validate that our SkateFormer outperforms recent state-of-the-art methods.

Attend to the key joints and frames

Self-attention within four partitions, neighboring or distant joints × local or global motion: state-of-the-art at 3.62 GFLOPs.

Left: a 3D skeleton whose joints are grouped by dashed outlines into neighboring joint partitions (arms, legs, torso). Middle, skeletal relation types: for 'make OK sign' the key joints are physically neighboring joints, the fingers of one hand; for 'clap' they are physically distant joints, both palms, linked across partitions by red attention arrows over four frames. Right, temporal relation types: for 'brush teeth' the key frames are neighboring frames (local motion); for 'sit down' they are distant frames, the first and the last (global motion). Each example shows the RGB frame next to its skeleton sequence. — **Partition-specific attention.** Key joints can be physically neighboring (fingers of one hand in ‘make OK sign’) or distant (both palms in ‘clap’); key frames can be neighboring (local motion in ‘brush teeth’) or distant (global motion in ‘sit down’). Left: the neighboring joint partitions.

Paper figures

Skate-Type importance scores per action, the Skate-MSA branches, the partition and reverse operations, trimmed-uniform frame sampling and the joint partitions.

Four panels, one per NTU RGB+D action label, each with a video frame and two bar charts of the mean attention score of Skate-Type-1 to Skate-Type-4. For label 7, sitting down, and label 8, standing up, the Skate-Type-3 or Skate-Type-4 bars are the highest. For label 10, reading, and label 11, writing, the Skate-Type-1 bar is the highest. A diagram on top maps Skate-Type-1 and -2 to local motion with neighboring or distant joints, and Skate-Type-3 and -4 to global motion. — **Skate-Type Importance Scores.** Mean strength of each Skate-Type’s feature correlation map in Skate-MSA: Skate-Type-3 or -4 dominate for ‘sitting down’ (label 7) and ‘standing up’ (8), where the overall motion is pivotal; Skate-Type-1 dominates for ‘reading’ (10) and ‘writing’ (11), which rely on intricate hand movements.

Quantitative results

92.6% on X-Sub60 and 97.0% on X-View60 with the joint modality alone, 98.3% on NW-UCLA; best in every setting of Table 1 except the four-modality ensemble on X-Sub120.

Table 1. Top-1 accuracy of different skeleton-based action recognition methods on the NTU RGB+D, NTU RGB+D 120 and NW-UCLA datasets. The methods of RNN, CNN, GCN, and Transformer types are evaluated based on the number of input frames and the use of ensemble strategies. The best performances are highlighted in bold.

Bold best

Scroll for more columns

Methods	Frames	NTU RGB+D (%)						NTU RGB+D 120 (%)						NW-UCLA (%)
		X-Sub60			X-View60			X-Sub120			X-Set120
		𝔼₁	𝔼₂	𝔼₄	𝔼₁	𝔼₂	𝔼₄	𝔼₁	𝔼₂	𝔼₄	𝔼₁	𝔼₂	𝔼₄
RNN
AGC-LSTM	100	87.5	89.2	–	93.5	95.0	–	–	–	–	–	–	–	93.3
CNN
TA-CNN	64	88.8	–	90.4	93.6	–	94.8	82.4	–	85.4	84.0	–	86.8	96.1
Ske2Grid	100	88.3	–	–	95.7	–	–	82.7	–	–	85.1	–	–	–
GCN
SGN	20	–	89.0	–	–	94.5	–	–	79.2	–	–	81.5	–	–
CTR-GCN	64	89.9	–	92.4	–	–	96.8	84.9	88.7	88.9	–	90.1	90.6	96.5
ST-GCN++	100	89.3	91.4	92.1	95.6	96.7	97.0	83.2	87.0	87.5	85.6	87.5	89.8	–
InfoGCN	64	–	–	92.7	–	–	96.9	85.1	88.5	89.4	86.3	89.7	90.7	96.6
FR-Head	64	90.3	92.3	92.8	95.3	96.4	96.8	85.5	–	89.5	87.3	–	90.9	96.8
Koopman	64	90.2	–	92.9	95.2	–	96.8	85.7	–	90.0	87.4	–	91.3	97.0
LST	64	90.2	–	92.9	95.6	–	97.0	85.5	–	89.9	87.0	–	91.1	97.2
HD-GCN	64	90.6	92.4	93.0	95.7	96.6	97.0	85.7	89.1	89.8	87.3	90.6	91.2	96.9
STC-Net	64	–	92.5	93.0	–	96.7	97.1	–	89.3	89.9	–	90.7	91.3	97.2
Transformer
DSTA-Net	128	–	–	91.5	–	–	96.4	–	–	86.6	–	–	89.0	–
STST	128	–	–	91.9	–	–	96.8	–	–	–	–	–	–	–
FG-STFormer	128	–	–	92.6	–	–	96.7	–	–	89.0	–	–	90.6	97.0
Hyperformer	64	90.7	–	92.9	95.1	–	96.5	86.6	–	89.9	88.0	–	91.3	96.7
SkateFormer	64	92.6	93.0	93.5	97.0	97.4	97.8	87.7	89.4	89.8	89.3	91.0	91.4	98.3

𝔼₁: joint modality only; 𝔼₂: joint + bone; 𝔼₄: joint + bone + joint motion + bone motion, with a separate network for each modality and their outputs ensembled.

Table 2. Comparison with human interaction recognition methods on the NTU-Inter and NTU-Inter 120 datasets. The best performances are highlighted in bold.

Bold best

Scroll for more columns

Methods	NTU-Inter (𝔼₁, %)		NTU-Inter 120 (𝔼₁, %)		Params. (M)	FLOPs (G)	Time (ms)
Methods	X-Sub60	X-View60	X-Sub120	X-Set120	Params. (M)	FLOPs (G)	Time (ms)
Transformer
IGFormer	93.6	96.5	85.4	86.5	–	–	–
SkeleTR	94.9	97.7	87.8	88.3	3.82	7.30	–
ISTA-Net	–	–	90.6	91.7	6.22	68.18	21.71
SkateFormer	97.1	99.3	92.3	93.2	2.02	3.62	11.25

NTU-Inter and NTU-Inter 120 are the 11 and 26 two-person interaction classes of NTU RGB+D and NTU RGB+D 120.

Table 3. Comparative analysis of SkateFormer with other methods by parameters, FLOPs, inference time, and average top-1 accuracy for joint modality.

Bold best

Scroll for more columns

Methods	Params. (M)	FLOPs (G)	Time (ms)	NTU RGB+D (𝔼₁, %)	NTU RGB+D 120 (𝔼₁, %)
GCN
InfoGCN	1.56	3.34	12.97	–	85.7
FR-Head	1.45	3.60	18.49	92.8	86.4
Koopman	5.38	8.76	17.86	92.7	86.6
LST	2.10	3.60	18.85	92.9	86.3
HD-GCN	1.66	3.44	72.81	93.2	86.5
Transformer
DSTA-Net	3.45	16.18	13.80	–	–
Hyperformer	2.71	9.64	18.07	92.9	87.3
SkateFormer	2.03	3.62	11.46	94.8	88.5

Accuracy: average top-1 of the two benchmarks of each dataset, joint modality. Compiled with the publicly available official code of each method.

Table 12. FLOPs and memory usage for attention layers.

Scroll for more columns

Attention Types in attention layer	Total 8 attention layers		First two attention layers
Attention Types in attention layer	FLOPs (M)	Memory (MB)	FLOPs (M)	Memory (MB)
vnjpk + vdjpl + tlocalm + tglobaln	29.12	56.3	15.53	27.0
Naive self-attention	801.79 (×27.53)	1530.0 (×27.18)	603.65 (×38.87)	1152.0 (×42.67)

The 48× reduction stated in the paper is a theoretical figure for the first two SkateFormer blocks, before downsampling, where the effect is largest; measured on the attention layers alone, Skate-MSA needs 38.87× fewer FLOPs there.

Table 9. Top-1 classification accuracy of different skeleton-based action recognition methods on NTU RGB+D dataset. The best performances are highlighted in bold.

Bold best

Scroll for more columns

Methods	Frames	NTU RGB+D (%)
		X-Sub60				X-View60
		J	B	JM	BM	J	B	JM	BM
ST-GCN (*)	100	87.8	88.6	85.8	86.2	95.5	95.0	93.7	92.8
CTR-GCN	64	89.9	90.6	88.1	87.9	–	–	–	–
CTR-GCN (*)	100	89.6	90.0	88.0	87.5	95.6	95.4	94.4	93.6
ST-GCN++	100	89.3	90.1	87.5	87.3	95.6	95.5	94.3	93.8
InfoGCN	64	89.8	90.6	88.9	88.6	95.2	95.5	94.2	93.6
FR-Head	64	90.3	91.1	88.7	87.6	95.3	95.0	93.6	92.6
LST	64	90.2	91.2	88.0	87.8	95.6	95.5	93.7	93.2
HD-GCN	64	90.6	90.9	–	–	95.7	95.1	–	–
SkateFormer	64	92.6	92.1	89.8	89.0	97.0	96.5	95.8	94.7

J: joint, B: bone, JM: joint motion, BM: bone motion. (*): accuracy as reported in ST-GCN++, since the official code and paper give no modality-specific numbers.

Table 10. Top-1 classification accuracy of different skeleton-based action recognition methods on NTU RGB+D 120 dataset. The best performances are highlighted in bold.

Bold best

Scroll for more columns

Methods	Frames	NTU RGB+D 120 (%)
		X-Sub120				X-Set120
		J	B	JM	BM	J	B	JM	BM
ST-GCN (*)	100	82.1	83.7	80.3	80.6	84.5	85.8	82.7	83.0
CTR-GCN	64	84.9	85.7	81.4	81.2	–	87.5	–	–
CTR-GCN (*)	100	84.0	85.9	81.1	82.2	85.9	87.4	84.1	83.9
ST-GCN++	100	83.2	85.6	80.4	81.5	85.6	87.5	84.3	83.0
InfoGCN	64	85.1	87.3	82.1	82.5	86.3	88.5	84.4	84.8
FR-Head	64	85.5	86.8	81.9	82.0	87.3	88.1	84.0	83.9
LST	64	85.5	87.5	82.3	82.4	87.0	88.7	83.9	84.4
HD-GCN	64	85.7	86.7	–	–	87.3	88.4	–	–
SkateFormer	64	87.7	88.2	83.1	82.3	89.3	89.8	85.3	84.1

Ablations Tables 4, 5 and 6

Table 4. Ablation study on the influence of skeletal and temporal relation types (SkateTypes) of Skate-MSA in our SkateFormer. (*: The batch size is set to 8 due to limited GPU memory.)

Bold best

Scroll for more columns

Attention Types	NTU RGB+D (%)		Params. (M)	FLOPs (G)	Time (ms)	Memory (MB)
Attention Types	X-Sub60	X-View60	Params. (M)	FLOPs (G)	Time (ms)	Memory (MB)
Baseline (no attention)	90.7	95.7	2.03	3.59	10.68	80.6
+ Naive self-attention*	90.6 (↓0.1)	95.0 (↓0.7)	2.03	4.39	17.55	1612.1
+ vnjpk + vdjpl	91.8 (↑1.1)	96.4 (↑0.7)	2.03	3.59	11.10	86.9
+ tlocalm + tglobaln	91.9 (↑1.2)	96.6 (↑0.9)	2.03	3.59	11.09	85.5
+ vnjpk + vdjpl + tlocalm + tglobaln	92.6 (↑1.9)	97.0 (↑1.3)	2.03	3.62	11.46	137.6

Each relation type helps on its own, and together they give the best accuracy (+1.9 / +1.3 points over the baseline). With Skate-MSA the model needs 137.6 MB, against 80.6 MB without attention and 1612.1 MB with naive self-attention over all joints and frames (run with batch size 8).

Table 5. Ablation study results on the Skate-Embedding to assess skeletal and temporal embedding impacts.

Bold best

Scroll for more columns

Embedding Methods		NTU RGB+D (%)
Skeletal	Temporal	X-Sub60	X-View60
✗	✗	91.9 (↓0.7)	96.6 (↓0.4)
	Learnable	91.8 (↓0.8)	96.5 (↓0.5)
	Fixed (TE)	91.6 (↓1.0)	96.6 (↓0.4)
Learnable (SE)	✗	92.1 (↓0.5)	96.5 (↓0.5)
	Learnable	91.4 (↓1.2)	96.2 (↓0.8)
	Fixed (TE)	92.6	97.0

Table 6. Ablation study results on the effect of intra-instance and inter-instance augmentations.

Bold best

Scroll for more columns

Intra-instance		Inter-instance	NTU RGB+D (%)
Temporal	Skeletal	Inter-instance	X-Sub60	X-View60
Trimmed			89.8	94.6
Trimmed		✓	90.3 (↑0.5)	94.3 (↓0.3)
Trimmed	✓		92.2 (↑2.4)	96.6 (↑2.0)
Fixed	✓	✓	91.4 (↑1.6)	96.4 (↑1.8)
Uniform	✓	✓	91.6 (↑1.8)	96.8 (↑2.2)
Trimmed	✓	✓	92.6 (↑2.8)	97.0 (↑2.4)

Learnable skeletal features with fixed temporal index features (the Skate-Embedding) work best. Trimmed-uniform random sampling surpasses fixed-stride and uniform random sampling by 1.2 (0.6) and 1.0 (0.2) points on X-Sub60 (X-View60).

Per-class accuracy Table 8

Table 8. Top-1 accuracy based on action labels in NTU RGB+D X-Sub60 evaluation.

Scroll for more columns

Action Label	Baseline (Rank)	+ vnjpk + vdjpl	+ tlocalm + tglobaln	+ vnjpk + vdjpl + tlocalm + tglobaln
drink water	85.77 (47)	85.77 (–0.00)	85.77 (–0.00)	88.32 (↑2.55)
eat meal/snack	76.00 (56)	77.82 (↑1.82)	77.45 (↑1.45)	78.18 (↑2.18)
brushing teeth	90.11 (42)	85.35 (↓4.76)	91.21 (↑1.10)	89.01 (↓1.10)
brushing hair	89.01 (45)	91.21 (↑2.20)	93.77 (↑4.76)	93.41 (↑4.40)
drop	92.00 (39)	94.18 (↑2.18)	92.36 (↑0.36)	92.73 (↑0.73)
pickup	97.09 (18)	97.82 (↑0.73)	97.09 (–0.00)	98.55 (↑1.45)
throw	92.73 (36)	93.45 (↑0.73)	92.00 (↓0.73)	93.82 (↑1.09)
sitting down	98.17 (13)	98.53 (↑0.37)	99.27 (↑1.10)	98.53 (↑0.37)
standing up (from sitting position)	98.90 (7)	98.53 (↓0.37)	99.63 (↑0.73)	98.90 (–0.00)
clapping	83.88 (51)	81.68 (↓2.20)	86.08 (↑2.20)	85.35 (↑1.47)
reading	58.61 (60)	60.81 (↑2.20)	61.54 (↑2.93)	63.37 (↑4.76)
writing	68.38 (57)	68.01 (↓0.37)	66.91 (↓1.47)	71.32 (↑2.94)
tear up paper	95.94 (21)	96.68 (↑0.74)	94.46 (↓1.48)	95.94 (–0.00)
wear jacket	98.18 (12)	98.18 (–0.00)	98.55 (↑0.36)	97.82 (↓0.36)
take off jacket	98.19 (10)	98.91 (↑0.72)	99.64 (↑1.45)	99.64 (↑1.45)
wear a shoe	60.81 (59)	86.81 (↑26.01)	84.62 (↑23.81)	87.55 (↑26.74)
take off a shoe	78.83 (53)	82.48 (↑3.65)	78.47 (↑0.36)	83.94 (↑5.11)
wear on glasses	93.04 (34)	93.41 (↑0.37)	93.77 (↑0.73)	93.41 (↑0.37)
take off glasses	95.62 (23)	95.62 (–0.00)	95.26 (↓0.36)	94.89 (↓0.73)
put on a hat/cap	96.69 (19)	97.43 (↑0.74)	98.16 (↑1.47)	97.79 (↑1.10)
take off a hat/cap	98.90 (7)	98.53 (↓0.37)	98.90 (–0.00)	98.90 (–0.00)
cheer up	93.80 (29)	92.34 (↓1.46)	93.80 (–0.00)	94.89 (↑1.09)
hand waving	93.80 (29)	94.16 (↑0.36)	93.80 (–0.00)	93.07 (↓0.73)
kicking something	94.20 (27)	94.93 (↑0.72)	97.10 (↑2.90)	97.83 (↑3.62)
reach into pocket	84.67 (49)	84.31 (↓0.36)	86.13 (↑1.46)	86.86 (↑2.19)
hopping (one foot jumping)	98.91 (6)	98.91 (–0.00)	98.91 (–0.00)	98.91 (–0.00)
jump up	100.00 (1)	100.00 (–0.00)	100.00 (–0.00)	100.00 (–0.00)
make a phone call/answer phone	90.18 (41)	89.09 (↓1.09)	90.55 (↑0.36)	92.00 (↑1.82)
playing with phone/tablet	76.36 (55)	76.36 (–0.00)	77.09 (↑0.73)	74.91 (↓1.45)
typing on a keyboard	64.36 (58)	72.73 (↑8.36)	74.55 (↑10.18)	77.82 (↑13.45)
pointing to something with finger	79.71 (52)	88.04 (↑8.33)	81.88 (↑2.17)	84.42 (↑4.71)
taking a selfie	92.39 (38)	93.84 (↑1.45)	94.57 (↑2.17)	93.12 (↑0.72)
check time (from watch)	93.12 (33)	91.67 (↓1.45)	92.75 (↓0.36)	92.03 (↓1.09)
rub two hands together	89.13 (43)	89.49 (↑0.36)	91.30 (↑2.17)	92.03 (↑2.90)
nod head/bow	97.46 (14)	98.55 (↑1.09)	98.55 (↑1.09)	99.28 (↑1.81)
shake head	94.55 (26)	95.27 (↑0.73)	96.73 (↑2.18)	96.00 (↑1.45)
wipe face	85.87 (46)	87.32 (↑1.45)	87.68 (↑1.81)	90.94 (↑5.07)
salute	93.48 (31)	93.48 (–0.00)	93.84 (↑0.36)	94.93 (↑1.45)
put the palms together	97.46 (14)	97.46 (–0.00)	96.74 (↓0.72)	96.38 (↓1.09)
cross hands in front (say stop)	96.01 (20)	97.10 (↑1.09)	96.38 (↑0.36)	97.46 (↑1.45)
sneeze/cough	78.62 (54)	82.97 (↑4.35)	83.33 (↑4.71)	85.51 (↑6.88)
staggering	99.28 (4)	99.64 (↑0.36)	100.00 (↑0.72)	99.28 (–0.00)
falling	99.64 (2)	100.00 (↑0.36)	99.64 (–0.00)	100.00 (↑0.36)
touch head (headache)	84.06 (50)	89.13 (↑5.07)	84.06 (–0.00)	88.41 (↑4.35)
touch chest (stomachache/heart pain)	93.84 (28)	94.57 (↑0.72)	93.48 (↓0.36)	96.01 (↑2.17)
touch back (backache)	95.29 (24)	94.57 (↓0.72)	95.29 (–0.00)	95.65 (↑0.36)
touch neck (neckache)	90.22 (40)	90.58 (↑0.36)	92.39 (↑2.17)	93.12 (↑2.90)
nausea or vomiting condition	85.45 (48)	83.64 (↓1.82)	84.00 (↓1.45)	85.45 (–0.00)
use a fan (with hand or paper)/feeling warm	89.09 (44)	90.91 (↑1.82)	89.45 (↑0.36)	94.18 (↑5.09)
punching/slapping other person	92.70 (37)	93.07 (↑0.36)	92.70 (–0.00)	94.16 (↑1.46)
kicking other person	95.65 (22)	96.38 (↑0.72)	95.65 (–0.00)	96.38 (↑0.72)
pushing other person	98.55 (9)	97.83 (↓0.72)	97.46 (↓1.09)	98.19 (↓0.36)
pat on back of other person	93.48 (31)	95.29 (↑1.81)	94.93 (↑1.45)	95.65 (↑2.17)
point finger at the other person	92.75 (35)	94.20 (↑1.45)	94.57 (↑1.81)	94.20 (↑1.45)
hugging other person	99.27 (5)	99.27 (–0.00)	99.64 (↑0.36)	99.64 (↑0.36)
giving something to other person	95.29 (24)	96.74 (↑1.45)	95.65 (↑0.36)	95.65 (↑0.36)
touch other person's pocket	97.45 (17)	97.45 (–0.00)	98.55 (↑1.09)	96.00 (↓1.45)
handshaking	97.46 (14)	97.10 (↓0.36)	97.46 (–0.00)	97.46 (–0.00)
walking towards each other	99.63 (3)	100.00 (↑0.37)	100.00 (↑0.37)	100.00 (↑0.37)
walking apart from each other	98.19 (10)	97.10 (↓1.09)	96.74 (↓1.45)	97.83 (↓0.36)
average	90.65	91.79 (↑1.14)	91.88 (↑1.23)	92.62 (↑1.98)

Compared with the baseline without attention, the full model improves 42 of 60 classes, keeps 8 and slightly lowers 10; ‘wear a shoe’ rises from 60.81% to 87.55%. Most failure cases involve fine finger motions such as reading, writing and typing on a keyboard.

Partition-based transformers compared Table 7

Table 7. Comparison with ours SkateFormer with existing transformer-based methods. S-Attn, T-Attn, T-Conv, and ST-Attn indicate ‘skeletal attention’, ‘temporal attention’, ‘temporal convolution’ and ‘skeletal-temporal attention’, respectively.

Scroll for more columns

Methods	Partition Types	Tokenization for Attention	Attention Types
Human Action Recognition
DSTA-Net	N/A (Reshaping)	No	S-Attn, T-Attn (Sequential)
ST-TR	N/A (Reshaping)	No	S-Attn, T-Attn (Parallel)
IIP-Transformer	S-Type	Yes	S-Attn, T-Attn (Sequential)
STST	N/A (Reshaping)	No	S-Attn, T-Attn (Parallel)
FG-STFormer	S-Type	Yes	S-Attn, T-Attn (Sequential)
Hyperformer	S-Type	Yes	S-Attn, T-Conv (Sequential)
Human Interaction Recognition
IGFormer	S-Type, T-Type	Yes	Joint-group-level ST-Attn
SkeleTR	N/A (Pooling)	Yes	Joint-group-level ST-Attn
ISTA-Net	S-Type, T-Type	Yes	Joint-group-level ST-Attn
Both
SkateFormer	4 Skate-Types	No	Joint-element-level ST-Attn

Partition, attend, reverse

SkateFormer framework. The input X of shape [T, V, C_in] passes three linear projection layers, and the Skate-Embedding of the frame indices t_idx is added. A SkateFormer block holds a self-attention part (layer norm, linear layer, channel-wise split into G-Conv, Skate-MSA and T-Conv branches, concatenation, linear layer, residual) and a feed-forward part (layer norm, FFN, residual). Four stages of two blocks each are separated by stride-2 temporal convolutions, from [T, V, C] to [T/8, V, 2C], followed by a head with pooling and a linear classifier that outputs y-hat. — **The overall framework of SkateFormer.** Linear projections and the Skate-Embedding feed eight SkateFormer blocks in four stages of two, with stride-2 temporal downsampling between stages; each block splits its channels into a graph convolution, a temporal convolution and the Skate-MSA, followed by a feed-forward layer.

01Four Skate-Types. Joints form neighboring partitions (limbs, torso) and distant partitions (same-position joints across them); frames form local segments of consecutive frames and N-strided global axes. Their four combinations are Skate-Type-1 to -4.
02Skate-MSA. Half of the channels go to self-attention, split into four parts that each attend within one Skate-Type partition. In the first blocks this is about 48× less computation than naive self-attention (38.87× measured).
03Skate-Embedding. The outer product of learnable skeletal features and fixed sinusoidal features of the sampled frame indices, trained with trimmed-uniform frame sampling and Bone Length AdaIN, which exchanges bone lengths between subjects.

BibTeX

@inproceedings{do2024skateformer,
  title={Skateformer: skeletal-temporal transformer for human action recognition},
  author={Do, Jeonghyeok and Kim, Munchurl},
  booktitle={European Conference on Computer Vision},
  pages={401--420},
  year={2024},
  organization={Springer}
}