Slayer1B-Poetry
English poetry continuation model derived from SlayerLab/Slayer1B-1.09. In the three recorded comparisons below, regular Slayer continues short openings as descriptive prose; the poetry adapter shifts toward lineated verse, imagery and a more lyrical voice.
The style shift is clearer than any claim of polished literary quality. The adapter can produce awkward phrases, malformed words and very short poems. These samples demonstrate domain adaptation, not a measured human preference win.
Learn to train Slayer: Jupyter tutorials
Use this poetry adaptation as a worked CPT case study, then learn SFT and reward-based RL with the same native Slayer1B parent and LoRA implementation.
| Start here | Lesson |
|---|---|
| 00 · Compare generations | Real parent-versus-poetry samples and a quality evaluation plan; no GPU needed |
| 01 · Continued pretraining | Train on raw verse, inspect next-token targets and compare outputs |
| 02 · Supervised fine-tuning | Train request/poem pairs with response-only loss |
| 03 · Reward-based RL | Sample, score and update with REINFORCE and a frozen-parent KL penalty |
| 04 · Checkpoint averaging | Build a full model from N compatible checkpoints and compare it with each source |
Setup and teaching guide · Training-loop source · Validation evidence. Download a notebook and open it in Jupyter or upload it to Google Colab. Training is opt-in. The SFT and RL lessons are new teaching experiments; the released poetry weights were trained with CPT only. The toy RL reward measures form, not literary quality.
How generation differs from regular Slayer
The following are all three recorded before/after pairs, reproduced in full without edits. Both models received the same opening and used temperature 1.0, top-k 50, repetition penalty 1.1, seed 1234, a 96-new-token maximum, and suppression of EOS for the first 16 new tokens. The parent is the original Slayer1B-1.09 update-45,000 checkpoint; the adapted version adds the selected poetry LoRA. These were the run's existing diagnostic prompts, not a broad or blinded evaluation. A shared seed controls the sampling setup but does not make the two output trajectories equivalent.
Opening: The night
Regular Slayer1B-1.09
The night-crawlers of the genus Trapezias are crustaceans that spend almost all of their life lives in burrows that they excavate. At night the crustaceans feed on insects they can hide under objects and in crevices. Most species are nocturnal, but there is also one day-time dwelling species with an unusual behaviour: it does not burrow at all during the daytime, and wakes up in the morning to res
Slayer1B-Poetry
The night-crowd of the streets
Foamed and frothed to and fro;
And many a wiry streetman in white was shaded
By the flurry: all he felt had been washed with his blood.
And lo, the sun had risen; yet still
His smile seemed sad, with no change in his eyes for sorrow
— Thrust from behind, and backward again,
He heard his old
What changes: The parent follows “night” into an encyclopedia-like passage about crustaceans. The adapter instead produces a street scene, line breaks, alliteration (“Foamed and frothed”) and an emotional register. Quality limit: The scene becomes hard to follow, some phrasing is strained, and the output ends mid-sentence at the token cap.
Opening: On the shore
Regular Slayer1B-1.09
On the shore of a lake or in a river, there are numerous plant communities. There could be grasses on the surface or sedges on the bottom. Mosses and the mushrooms that grow from them add colour to the scene and may attract some insects. If you look closely, there usually is some sort of creature beneath the vegetation too; often they can’t see because it is below the water level, but if your eyes are sufficiently sharp, you will spot an occasional
Slayer1B-Poetry
On the shore of a deep bay.
It's a lonely island;--
The island is small.
What changes: The parent gives explanatory prose about lakeside vegetation. The adapter compresses the setting into three short lines and introduces solitude. Quality limit: The language is plain, “island” repeats, and the poem stops early; line breaks alone do not establish strong poetry.
Opening: A small bird
Regular Slayer1B-1.09
A small bird with a short tail. The red-winged Blackbird is a popular game bird and is found wherever there are people. Often they will be seen in backyards, fields or woodland edge all over the world, but is equally at home in farmland fields, forest margins and shrubby areas. It has small grey/brown body, yellowish eyes on a green black background, yellow wing patches, and a white patch above its chest and a red
Slayer1B-Poetry
A small bird that went forth
And sang from its nest so long
It was not a song of woe,
Nor a warlike tirpe’r the same.
What changes: The parent moves into a descriptive bird entry. The adapter produces a four-line stanza with song and sorrow as motifs. Quality limit: “tirpe’r” is malformed and the final line is unclear. This is a visible shift toward verse with an obvious remaining fluency defect.
What the samples support
- Poetic style: All three adapted outputs use verse-like line breaks; all three parent outputs are descriptive prose.
- Voice and imagery: The adapter introduces emotional framing, sound patterns and lyric motifs, most visibly in “The night.”
- Coherence and fluency: Still uneven. The examples include unclear narrative transitions, repetition and a malformed word. The parent also contains awkward phrasing and factual-looking statements that have not been verified.
- Completeness: Two adapted outputs stop before the cap, while the longest ends mid-sentence. These are continuations, not evidence of reliable complete-poem generation.
- Unmeasured: Meter, rhyme, theme adherence, originality, memorization, and human preference have not been systematically assessed. No claim that the adapter wins a poetry-quality benchmark is made.
For use, supply a poem opening rather than a chat instruction. This adaptation trained on raw poems, with no theme-following instruction tuning or preference optimization. Full sample artifacts: parent, poetry adapter.
Release at a glance
Trained on 17 November 2026 with raw-poem continued pretraining. The frozen parent has 1,073,912,851 parameters; the native LoRA adapter has 22,283,520 trainable parameters. This repository includes the selected adapter, tokenizer, runnable inference code, recipe, provenance and evaluation artifacts. Base weights download separately.
Use
This uses the original Slayer PyTorch architecture and custom LoRA implementation. It is not a Transformers AutoModel or PEFT checkpoint. Base weights are downloaded separately from the pinned public archive on first use (approximately 4.3 GB), and their SHA256 is checked. The pinned native architecture Python source is also downloaded and executed; review common.py and the linked source before running it.
hf download SlayerLab/Slayer1B-Poetry --local-dir Slayer1B-Poetry
cd Slayer1B-Poetry
python -m pip install -r requirements.txt
python generate.py --adapter adapter.pt --opening "The night" --tokens 128
The loader selects CUDA when available and otherwise CPU. The default sampler uses temperature 1.0, top-k 50, repetition penalty 1.1 and seed 1234. Generation uses the last 1,024 tokens as context, matching adaptation training. There is no chat template. CUDA is recommended for practical generation speed.
adapter.pt contains inference weights and provenance with optimizer state removed; it loads with torch.load(..., weights_only=True). adapter_model.safetensors contains the identical adapter tensors in a non-pickle format with native parameter names; the supplied CLI uses adapter.pt. Neither file contains the full base weights. Training-resume optimizer states are not included.
Base and architecture
- Parent: SlayerLab/Slayer1B-1.09, native update 45,000.
- Base archive: SlayerLab/Slayer1B-1.09-checkpoints, revision
33d2faef5ee09de7baa2feb65131e8e9e2c0c008,wagi_45000.pt. - Base SHA256:
e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827. - Native architecture:
train_gpt_ref_r24_2.pyfrom the parent at revision58d8f359c1fdd887f1b7344ec52c5511c111f790. - 20 layers, width 2,048, 16 attention heads, RMSNorm, RoPE, SwiGLU, QK normalization and value residuals; tied embedding/output head; vocabulary 32,768.
- Original base context: 4,096. Poetry adaptation context: 1,024. Longer-context poetry behavior was not evaluated.
- Original tokenizer from Slayer1B-1.08 at
5f854a0cc7160594d54d1788d37a44e8ebabe16c, SHA2560640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd.
Training
LoRA rank 32, alpha 32, dropout 0, applied to 100 block linears: QKV, attention output and all three SwiGLU projections. Frozen BF16 base, FP32 adapter weights and optimizer states, BF16 autocast.
One pass over shuffled nonoverlapping 1,024-token windows: 2,299 optimizer updates and 37,657,600 processed tokens. The saved selected checkpoint is update 2,299. Global batch was 16,384 tokens (H100 recipe: microbatch 16, accumulation 1). The run continued after earlier hardware stages; the H100 elapsed time in result.json is the final process interval, not total end-to-end training time or cost.
AdamW, LR 1e-4, betas 0.9/0.95, epsilon 1e-8, weight decay 0.01, 5% warmup followed by cosine decay to 10% LR, gradient clipping 1.0, seed 20261117. Lowest validation CE on a fixed 65,536-token subset selected the adapter; checks occurred every 200 updates and at completion. Full validation and held-out test were measured after selection. recipe.json comes from the actual saved checkpoint.
Data
English subset of the local Slayer Poetry v0 corpus. Upstream sources include PoeTree 1.0.0, DanFosing/public-domain-poetry, and Wolne Lektury. See DATA_CARD.md for the combined corpus inventory; its token counts use a different inventory tokenizer and must not be confused with the adaptation counts below.
| Split | Documents | Slayer body tokens | With EOS boundaries |
|---|---|---|---|
| Train | 55,699 | 37,546,783 | 37,658,181 |
| Validation | 1,140 | 825,782 | 828,062 |
| Test | 1,521 | 785,964 | 789,006 |
Each document is EOS + poem + EOS; causal attention can cross document boundaries. Incomplete final windows are excluded. Texts passed tokenizer encode/decode round trips. Author-group splits, normalized exact deduplication and conservative near-line deduplication were used. No synthetic poems were added. Historical spellings, translations, long narrative verse and residual variants remain. Pseudonym, translation and collection-containment leakage has not been comprehensively excluded. Raw corpus texts are not distributed in this model repository.
Supporting language-model metrics
Author-run next-token cross entropy and perplexity on English poetry, using the same Slayer tokenizer. Lower perplexity shows better prediction of held-out poetry tokens; it does not measure imagery, emotional impact, coherence or reader preference. The generation comparisons above are the primary illustration of the behavior change.
| Evaluation | Tokens evaluated | CE | Perplexity |
|---|---|---|---|
| Frozen parent, fixed validation subset | 65,536 | 3.413123 | 30.359912 |
| Poetry adapter, same fixed validation subset | 65,536 | 3.277095 | 26.498673 |
| Poetry adapter, full validation windows | 827,392 | 3.289073 | 26.818002 |
| Poetry adapter, held-out test windows | 788,480 | 3.243641 | 25.626855 |
Only the first two rows are a matched before/after comparison. Full-split baseline metrics were not recorded. Evaluated token counts omit incomplete windows. Raw evidence: result.json, baseline.json, metrics.jsonl and run.json. The parent's general-purpose benchmark scores are not results for this adapter; general capability retention was not evaluated.
Intended use and limitations
Research and experimentation with English verse continuation and small-model domain adaptation. Outputs may repeat training passages, contain stereotypes or offensive text, invent facts, produce awkward verse or fail to follow form/theme constraints. Inspect outputs before publication. Do not assume originality, factual accuracy, safety filtering or improved literary quality from perplexity alone. Polish poetry adaptation was not performed.
License and provenance
The parent model declares Apache-2.0; the adapter and accompanying release code are released under Apache-2.0. This does not grant rights to underlying corpus texts or generated passages. Source rights metadata were retained during corpus preparation, but licensing/public-domain status was not independently verified for every work. No blanket dataset license is asserted.
provenance.json records pinned revisions, checksums, selected step and architecture. SHA256SUMS covers release artifacts. The selected original training checkpoint SHA256, before removing optimizer state, was 0b49d5c90e82e694c9b19db856e8260053abfedaae2c4f380be4ac5aa76e55c7.