SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves strong unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR–EO observations on tasks that benefit from complementary sensing.

Semantics to queries, pixels to tokens

Semantic queries take the VFM supervision, SAR and EO tokens take masked reconstruction, all in one shared encoder.

Two panels. Previous work: masked SAR tokens pass through a SAR encoder whose output tokens are both decoded by a SAR decoder for the pixel loss and aligned with VFM features of the EO image, one stream with two duties (coupled); raw EO is not used as an input. SAREO-FM (ours): masked SAR tokens, masked raw EO tokens and learnable semantic queries enter one SAREO encoder; the SAR and EO output tokens go to separate SAR and EO decoders for the pixel losses, while only the query outputs are aligned with the VFM features (decoupled), for SAR, EO or SAR plus EO input. — **Coupled versus decoupled semantic supervision.** Top: prior methods align image tokens with EO-derived VFM features, so one token stream serves both alignment and pixel reconstruction. Bottom: SAREO-FM supervises learnable semantic queries and reconstructs SAR and EO from the modality tokens.

Fusion recovers what one sensor misses

Building-damage maps on BRIGHT from pre-event EO and post-event SAR: the fused input recovers destroyed buildings that the single-modality models predict as intact or background.

Input SAREO-FM, SAR+EO (ours) SAREO-FM, one modality Ground truth Baseline

Paper Figure 5: two BRIGHT scenes, each with pre-event EO, post-event SAR, ground truth, SAREO-FM SAR+EO, SAREO-FM EO-only, SAREO-FM SAR-only and SARMAE SAR-only.

Qualitative transfer on BRIGHT. The interactive gallery needs JavaScript; this is the paper figure with the same scenes.

Use the arrows or the ← → keys to change scene · click a tile to compare it with the ground truthSwipe or use the arrows to change scene · tap a tile to compare it with the ground truth

Qualitative, in-corpus examples: BRIGHT also contributes source imagery to SAR-1M, so the scenes illustrate dense transfer rather than a full segmentation benchmark. Paper figure

Label-efficient SAR transfer

With the encoder frozen, SAREO-FM is consistently stronger than SARMAE^* on FUSAR-Ship from 2-shot onward and on SAR-ACD from 5-shot onward.

Two line plots of overall accuracy against shots (1, 2, 5, 10, 20, 40) with frozen encoders, for SARMAE* (gray, dashed), SAREO-FM both (teal) and SAREO-FM sem (ochre). FUSAR-Ship, left: the SAREO-FM curves lie above SARMAE* from 2 shots on, with the widest gap at 5 shots. SAR-ACD, right: SARMAE* is higher at 1 and 2 shots; SAREO-FM sem is higher from 5 shots on. — **Frozen few-shot SAR transfer.** We use fixed encoder features and train only the downstream classifier.

Four SAR-ACD aircraft chips. True A220: SAREO-FM predicts A220, SARMAE predicts ARJ21. True A330: SAREO-FM A330, SARMAE Boeing787. True ARJ21: SAREO-FM ARJ21, SARMAE Boeing737. True Boeing737: SAREO-FM Boeing737, SARMAE Boeing787. — **Qualitative comparison on SAR-ACD.** Representative test images misclassified by SARMAE^* but correctly classified by SAREO-FM.

With only 40 examples per class on the out-of-corpus SAR-ACD, end-to-end fine-tuning with semantic queries raises accuracy from 68.50% to 78.40% (Table 1); the exact frozen few-shot values are in Table 6. SARMAE^*: the official SARMAE checkpoint under the same downstream protocol.

Quantitative results

SAREO-FM is best in five of six controlled SAR fine-tuning settings, surpasses its DINOv3 teacher with EO input, and joint SAR–EO input gives the best result on both paired benchmarks.

Table 1. SAR target classification with end-to-end fine-tuning. The upper block reports published OA (%) under the source-specific protocol of each method and is included only as context. The lower block reports our controlled evaluation with identical data splits and downstream pipelines; mean and standard deviation over three seeds are shown. ^* denotes our re-evaluation of the official SARMAE checkpoint. Bold denotes the best controlled result.

Bold best controlled resultOA (%)

Scroll for more columns

Method	Venue	Backbone	FUSAR-Ship		MSTAR		SAR-ACD
Method	Venue	Backbone	40-shot	30%	40-shot	30%	40-shot	30%
Published results, each under its own source protocol (context only)
ResNet-50	CVPR'16	ResNet-50	–	58.41	–	89.94	–	59.70
Swin Transformer	ICCV'21	Swin-B	–	60.79	–	82.97	–	67.50
BEiT	ICLR'22	ViT-B	59.70	71.13	40.70	69.75	–	79.77
CROMA	NeurIPS'23	ViT-B	83.71	–	–	–	–	88.99
SAR-JEPA	ISPRS JPRS'24	ViT-B	85.80	–	91.60	–	75.50	–
SARATR-X	TIP'25	HiViT-B	87.70	–	98.10	–	76.40	–
LoMaR	WACV'25	ViT-B	82.70	–	77.00	–	67.40	–
SUMMIT	IJAEOG'25	ViT-B	81.50	71.91	63.60	98.39	68.70	84.25
Copernicus-FM	ICCV'25	ViT-B	87.61	–	–	–	–	92.63
CoDe-MAE	arXiv'26	HiViT-B	89.40	–	98.70	–	78.30	–
MaRS	AAAI'26	SwinV2-B	77.70	–	75.50	–	68.40	–
SARMAE	CVPR'26	ViT-B	89.30	92.92	96.70	99.61	–	95.06
SARMAE	CVPR'26	ViT-L	90.86	92.80	97.24	98.92	–	95.63
Controlled evaluation: identical splits and downstream pipelines
Random initialization	–	ViT-B	60.31±1.39	82.11±0.52	34.71±0.09	38.76±8.35	33.14±2.60	39.19±10.27
SARMAE^*	CVPR'26	ViT-B	90.09±0.36	92.82±0.21	94.54±1.72	97.18±0.32	68.50±2.60	95.01±0.85
SAREO-FM (sem)	–	ViT-B	91.05±0.16	93.13±0.14	97.57±0.42	99.13±0.54	78.40±1.24	93.34±0.55
SAREO-FM (both)	–	ViT-B	91.41±0.23	93.12±0.06	96.66±0.31	99.36±0.47	78.00±1.00	92.84±0.68

On the out-of-corpus SAR-ACD, semantic queries raise 40-shot accuracy from 68.50% to 78.40%, a 9.90-point gain; under 30% supervision, SARMAE^* remains 1.67 points stronger. MSTAR and FUSAR-Ship are part of the SAR-1M pretraining corpus.

Table 2. Frozen few-shot EO transfer. We report accuracy (%) for SAR and EO inputs over five support draws. CoDe-MAE values use its separate 10-shot protocol and are included only as context.

Bold bestAccuracy (%)

Scroll for more columns

Method / Feature	SAR				EO
Method / Feature	1-shot	5-shot	10-shot	40-shot	1-shot	5-shot	10-shot	40-shot
DINOv3 (teacher)	38.17±4.61	50.72±2.68	56.01±1.81	64.35±1.56	59.51±3.53	75.96±3.15	83.34±1.42	90.88±0.65
SARMAE^*	37.22±5.71	48.64±3.00	53.97±1.39	64.72±1.12	43.56±6.57	59.57±3.05	66.49±1.97	78.36±1.07
CoDe-MAE	–	–	59.88^†	–	–	–	81.18^†	–
SAREO-FM (both)	39.07±5.98	53.54±2.21	59.36±1.46	69.24±0.96	64.41±5.05	81.09±1.61	86.76±1.24	92.58±0.75

With EO input, SAREO-FM surpasses its frozen DINOv3 teacher by 4.90, 5.13, 3.42 and 1.70 points from 1 to 40 shots; with SAR input it improves over SARMAE^* at every shot count, by 1.85 to 5.39 points.

Table 3. Transfer on paired SAR–EO benchmarks. We report OA (%) on 5-shot So2Sat LCZ42 and micro-AP (%) on 5-shot BigEarthNet-MM.

Bold bestUnderline second bestOA (%) on So2Sat LCZ42 · micro-AP (%) on BigEarthNet-MM

Scroll for more columns

Input	Method	Feature	Benchmark
Input	Method	Feature	So2Sat LCZ42	BigEarthNet-MM
SAR+EO	Random initialization	mod	40.56±2.25	54.29±2.30
EO	DINOv3 (teacher)	mod	54.25±3.87	64.73±1.43
EO	SAREO-FM	mod	58.14±2.90	64.42±1.44
SAR	SARMAE^*	mod	28.96±1.68	57.44±0.69
SAR	SAREO-FM	mod	29.59±2.64	60.47±0.79
SAR+EO	SAREO-FM	both	58.79±4.19	66.70±0.58

On BigEarthNet-MM, joint input reaches 66.70 micro-AP, 2.28 points above the best SAREO-FM unimodal input; on So2Sat LCZ42, adding SAR to EO yields a smaller 0.65-point gain.

Table 4. Cumulative component ablation. All variants use matched pretraining and evaluation protocols. We report 10-shot SAR linear-probing performance. Avg. denotes the arithmetic mean across the two benchmarks, while Δ measures the gain over the preceding configuration.

Bold bestUnderline second bestOA (%)

Scroll for more columns

Pretraining configuration	MSTAR (OA↑)	FUSAR (OA↑)	Avg. (OA↑)	Δ
Random initialization	26.26±1.86	47.86±4.89	37.06	–
Plain MAE	66.16±3.15	76.46±2.52	71.31	+34.25
+ DINOv3 supervision	73.49±2.31	79.90±2.49	76.70	+5.39
+ Decoupled semantic supervision	77.11±2.14	82.55±2.15	79.83	+3.13
+ Raw EO modality (SAREO-FM)	78.41±2.14	84.00±2.88	81.21	+1.38

Decoupling semantic prediction from modality reconstruction adds 3.13 points on average; the complete SAREO-FM improves on plain MAE by 9.90 points.

Exact frozen few-shot SAR results Table 6

Table 6. Exact frozen few-shot SAR classification results corresponding to Fig. 3 of the main paper. We freeze the pretrained encoder and optimize only a linear downstream classifier. We report mean OA (%) and sample standard deviation over five matched support draws. sem uses semantic-query features, whereas both concatenates modality-token and semantic-query features. SARMAE^* denotes the official checkpoint evaluated with the same downstream protocol. Bold and underline denote the best and second-best values within each dataset and shot count, respectively; shaded rows are ours.

Bold bestUnderline second bestOA (%)

Scroll for more columns

Method / Feature	1-shot	2-shot	5-shot	10-shot	20-shot	40-shot
FUSAR-Ship
SARMAE^*	54.19±8.69	64.95±6.93	70.03±3.48	78.18±3.28	83.44±2.03	88.35±0.83
SAREO-FM (both)	55.56±7.50	64.79±3.81	72.84±3.97	80.46±1.81	85.32±1.79	89.81±0.70
SAREO-FM (sem)	55.79±9.07	68.54±4.35	75.74±2.46	81.65±2.91	85.50±1.71	89.56±1.22
SAR-ACD
SARMAE^*	35.09±7.77	39.56±6.52	47.12±3.16	51.68±4.08	60.30±2.86	70.11±1.82
SAREO-FM (both)	33.81±6.30	36.31±6.91	44.78±4.56	51.87±4.12	60.69±2.75	71.35±2.01
SAREO-FM (sem)	33.64±6.95	36.87±5.83	48.10±3.02	53.65±3.19	61.65±3.35	71.72±1.74

Masking modes and active losses Tables 7–8

Table 7. Shared-encoder sequence composition. Training counts use the 75% patch-mask ratio; downstream inference uses all available image patches.

Scroll for more columns

Configuration	SAR	EO	Queries	Total
Independent/shared masking	64	64	256	384
Complete EO dropping	64	0	256	320
Complete SAR dropping	0	64	256	320
Unpaired SAR training	64	0	256	320
SAR-only inference	256	0	256	512
EO-only inference	0	256	256	512
Joint SAR–EO inference	256	256	256	768

Table 8. Student inputs, teacher availability, and active losses. Retained image streams are masked by 75%, and all semantic queries remain active. The loss coefficients are λ_s = 1, λ_e = 1, and λ_align = 0.5.

Scroll for more columns

Sample / mode	Prob.	Student encoder image streams	Teacher input	Active objective
Paired, independent masks	0.35	Visible SAR + visible EO	Clean EO	ℒ_pix^s + ℒ_pix^e + 0.5ℒ_align
Paired, shared mask	0.35	Visible SAR + visible EO	Clean EO	ℒ_pix^s + ℒ_pix^e + 0.5ℒ_align
Paired, complete EO dropping	0.15	Visible SAR only	Clean EO, teacher only	ℒ_pix^s + 0.5ℒ_align
Paired, complete SAR dropping	0.15	Visible EO only	Clean EO	ℒ_pix^e + 0.5ℒ_align
Unpaired SAR	–	Visible SAR only	Unavailable	ℒ_pix^s
Unpaired EO	–	Not sampled in this work	–	–

Downstream datasets and evaluation scope Table 9

Table 9. Downstream datasets and interpretation. OA denotes overall accuracy; micro-AP denotes micro-averaged average precision. The paired EuroSAT protocol uses geospatially matched SAR observations in addition to standard EO imagery; standard EuroSAT itself is EO-only.

Scroll for more columns

Dataset	Input	Classes	Metric	Protocol	SAR-1M relation	Role in this work
MSTAR	SAR	10	OA	1–40-shot / 30%	In corpus	In-corpus target transfer
FUSAR-Ship	SAR	10	OA	1–40-shot / 30%	In corpus	In-corpus ship transfer
SAR-ACD	SAR	5	OA	1–40-shot / 30%	Out of corpus	Primary out-of-corpus SAR test
EuroSAT protocol	EO or SAR	10	OA	1/5/10/40-shot	No out-of-corpus claim	Unimodal scene transfer
NWPU-RESISC45	EO	45	OA	Few-shot / higher supervision	No out-of-corpus claim	EO scene transfer
AID	EO	30	OA	Few-shot / higher supervision	No out-of-corpus claim	EO scene transfer
So2Sat LCZ42	SAR, EO, or joint	17	OA	5-shot, city-disjoint	No out-of-corpus claim	Paired single-label transfer
BigEarthNet-MM	SAR, EO, or joint	19	micro-AP	5-shot, official split	No out-of-corpus claim	Paired multi-label transfer
BRIGHT	Pre-EO + post-SAR	4	Dense labels	Qualitative only	In-corpus source	In-corpus dense diagnostic

Architecture, components and readouts Tables 10–12

Table 10. Architecture summary.

Scroll for more columns

Component	Specification
Input resolution	256×256
Patch size / grid	16×16 / 16×16
SAR / EO tokens	256 per available modality
Shared encoder	ViT-B/16, 12 blocks, D = 768
Attention / MLP	12 heads / ratio 4
Semantic queries	256 learnable, spatially indexed
Position encoding	Shared 2D sine–cosine grid
Stream encoding	Learnable SAR / EO / query embeddings
Teacher	Frozen DINOv3-7B patch features
Semantic projector	Two-layer MLP
Reconstruction	Separate SAR/EO decoders

Table 11. Lifecycle of SAREO-FM components.

Scroll for more columns

Component	Pretraining	Downstream
Shared SAREO encoder	Trainable	Used
Semantic queries	Trainable	Used
SAR/EO patch embeddings	Trainable	Used as available
SAR/EO reconstruction decoders	Trainable	Discarded
Semantic projector	Trainable	Discarded
DINOv3-7B teacher	Frozen	Discarded

Table 12. Image-level readout dimensions.

Scroll for more columns

Available input	mod	sem	both
SAR or EO	768	768	1,536
SAR + EO	1,536	768	2,304

One encoder for SAR, EO, or both

SAREO-FM overview in three panels. Training: SAR and EO inputs are masked; their visible tokens and the learnable semantic queries enter the SAREO encoder; the SAR and EO output tokens go to a SAR decoder and an EO decoder for the SAR and EO pixel losses, and the query outputs are aligned with the features of a frozen vision foundation model applied to the EO input (alignment loss). Inference (downstream): only SAR, only EO, or joint SAR plus EO tokens, always with the semantic queries, pass through the frozen SAREO encoder to the downstream task. Modality masking strategy: cross-modal interaction with (i) independent and (ii) shared masking, and handling missing modality with (iii) EO masking and (iv) SAR masking. — **Overview of SAREO-FM.** The semantic-query outputs are aligned with a frozen VFM and never reach the decoders; the modality-token outputs are decoded for masked reconstruction; after pretraining, the encoder alone serves SAR, EO or joint input.

01Decoupled semantic supervision. 256 learnable semantic queries, one per patch, are aligned patch-wise with frozen DINOv3-7B features and are excluded from the decoders.
02Modality-specific reconstruction. Separate SAR and EO decoders reconstruct the masked patches (75% of each retained modality) from the modality tokens.
03Mixed modality masking. Independent, shared, EO-dropped and SAR-dropped inputs (0.35/0.35/0.15/0.15) train one ViT-B/16 encoder for SAR, EO or both.

BibTeX

@article{do2026sareofm,
  title={SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models},
  author={Do, Jeonghyeok and Kim, Munchurl},
  journal={arXiv preprint arXiv:2610.09317},
  year={2026}
}