LEDOM-2B: Reverse Language Model

⚠️ This is NOT a Hugging Face transformers model. It is a custom decoder-only Transformer trained right-to-left, and it cannot be loaded with AutoModelForCausalLM.from_pretrained(...). You need this repo's loader: https://github.com/your-org/ledom. See "How to use" below.

LEDOM-2B is a purely reverse autoregressive language model from the paper LEDOM: Reverse Language Model (Yin et al., 2025). Where a normal Forward LM (FLM) factorizes P(x) = ∏_t P(x_t | x_<t), LEDOM trains the same decoder-only Transformer on token-reversed documents so it learns the complementary factorization P(x) = ∏_t P(x_t | x_>t) — predicting each token from its future context.

The two main applications:

  1. Generate the past from the future: type a sentence, LEDOM produces text that plausibly precedes it. (Backward storytelling, abductive reasoning, mathematical work-backward derivations.)
  2. Reverse Reward verification: combine a forward LM's P(y|x) with LEDOM's posterior P(x|y) to rerank or beam-search candidate solutions. This implements noisy-channel decoding and yields +6.6 % AIME 2024, +15 % AMC 2023 in the paper, across DeepSeekMath / QwenMath / OpenMath2.

Model details

Parameters ~2 B
Architecture decoder-only Transformer, RMSNorm, RoPE, ReLU² MLP (modified-nanoGPT lineage)
Layers 34
Hidden dim (n_embd) 2176
Attention heads 17 (full attention; n_kv_head=17)
Sequence length 2048
Vocabulary 32 768 (rustbpe / tiktoken-compatible)
Window pattern SSSL (sliding window: 3 short + 1 full per repeat)
Precision BF16 master weights stored as FP32 in checkpoint
Training tokens ≈140 B (66 079 steps × 2 097 152 tokens)
Final val BPB 0.6334
Training data DCLM-Baseline-derived ClimbMix-400B (CommonCrawl)

The depth-34 / d2176 configuration is computed by this repo's standard scaling recipe (model_dim = depth * aspect_ratio with aspect_ratio=64); FLOPs and batch size are auto-derived from the depth.

Files in this repo

File Purpose
model_066079.pt Final model state dict (~10 GB, FP32 master weights)
meta_066079.json Training metadata: model config, hyperparameters, dataloader state
tokenizer.pkl tiktoken Encoding (rustbpe-trained, vocab 32768)
token_bytes.pt Per-token byte counts (used for bits-per-byte evaluation)
README.md This file

Drop these into $LEDOM_BASE_DIR/rlm_checkpoints/LEDOM-2B/ and $LEDOM_BASE_DIR/tokenizer/ respectively (see below).

How to use

1. Install the LEDOM repo

git clone https://github.com/your-org/ledom    # ← replace with the actual URL
cd ledom
uv sync --extra gpu          # CUDA   (or --extra cpu for laptop testing)
source .venv/bin/activate

2. Download the checkpoint

from huggingface_hub import snapshot_download
import os

# Pulls all files into $LEDOM_BASE_DIR/rlm_checkpoints/LEDOM-2B/
local = snapshot_download(
    repo_id="Corning/LEDOM-2B",
    local_dir=os.path.expanduser("~/.cache/ledom/rlm_checkpoints/LEDOM-2B"),
)

Then move the tokenizer files to the location the loader expects:

mkdir -p ~/.cache/ledom/tokenizer
mv ~/.cache/ledom/rlm_checkpoints/LEDOM-2B/tokenizer.pkl  ~/.cache/ledom/tokenizer/
mv ~/.cache/ledom/rlm_checkpoints/LEDOM-2B/token_bytes.pt ~/.cache/ledom/tokenizer/

3. Generate the past from the future (interactive)

LEDOM treats your input as the ending and produces text that comes before it.

python -m scripts.rlm_chat --model-tag LEDOM-2B
# Ending: ...and that is why I never trust raccoons again.
# >>> [it generates a backstory ending in your sentence]

Or programmatically — the trick is that LEDOM is a normal causal Transformer that just happens to have been trained on reversed token streams:

import torch
from ledom.common import compute_init, autodetect_device_type
from ledom.checkpoint_manager import load_model

_, _, _, _, device = compute_init(autodetect_device_type())
model, tokenizer, _ = load_model("rlm", device, phase="eval", model_tag="LEDOM-2B")
bos = tokenizer.get_bos_token_id()

ending = "and that is why I never trust raccoons again."

# Tokenize and reverse — feed the "ending" in reverse order.
tokens = tokenizer.encode(ending)
input_ids = torch.tensor([[bos] + tokens[::-1]], device=device)

# Generate up to N reversed tokens (causal autoregression as usual).
with torch.inference_mode():
    for _ in range(80):
        next_id = model(input_ids)[:, -1, :].argmax(-1, keepdim=True)
        if next_id.item() == bos:
            break
        input_ids = torch.cat([input_ids, next_id], dim=1)

# Reverse the (prompt + generation) back to natural reading order.
all_reversed = input_ids[0, 1:].tolist()       # drop BOS
print(tokenizer.decode(list(reversed(all_reversed))))

4. Use LEDOM as a Reverse Reward verifier

To rerank an FLM's candidate answers with LEDOM's posterior P(x|y):

python -m scripts.reverse_reward \
  --flm-tag <your-FLM>  --rlm-tag LEDOM-2B \
  --benchmark gsm8k --strategy best_of_n \
  --n 32 --lambda 0.4 --temperature 0.7

The bidirectional score is (1-λ)·log P_F(y|x) + λ·log P_R(x|y) (paper Eq. 7). Empirically λ ∈ [0.3, 0.5] works best for math reasoning. Step-wise beam search is available via --strategy beam --k 4 --n 3.

Behaviors to expect

LEDOM inherits its strengths from the reverse factorization and inherits its weaknesses for the same reason. From the paper's evaluation:

  • ✔ Excels at abductive inference, question synthesis from answers, inverse-relation completion, and naturally avoids the "reversal curse".
  • ✔ As a posterior verifier, consistently improves Best-of-N on math reasoning across forward models spanning 42 %–96 % greedy accuracy.
  • ✘ Underperforms forward LMs on forward-causal tasks (incremental code generation, factual recall like NQ-Open / TriviaQA).
  • ⚠ Safety asymmetry: forward-trained safety mechanisms do not transfer to reverse generation. Treat outputs accordingly; do not deploy untuned for user-facing content.

License

MIT (matches the upstream code repository).

Citation

@article{yin2025ledom,
  title={LEDOM: Reverse Language Model},
  author={Yin, Xunjian and Cheng, Sitao and Xie, Yuxi and Hu, Xinyu and Lin, Li and Wang, Xinyi and Pan, Liangming and Wang, William Yang and Wan, Xiaojun},
  journal={arXiv preprint arXiv:2507.01335},
  year={2025}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for Corning/LEDOM-2B