The Old Way

• You try one experiment
• Wait for results
• Think about what to try next
• Try another experiment
• Repeat... all day long

The AutoResearch Way

• You give a robot your project
• Tell it "try stuff and keep what works"
• Go to sleep
• Wake up to ~100 experiments done
• The robot kept only the winners

experiments run overnight while you sleep

experiments per hour, fully autonomous

CAN Do

• Modify train.py — anything goes
• Change model architecture
• Swap optimizers or hyperparameters
• Adjust batch size & model size
• Restructure the training loop
• Change activation functions

CANNOT Do

• Modify prepare.py (read-only)
• Install new packages
• Change the evaluation harness
• Modify the tokenizer or data pipeline
• Change the time budget (fixed 5 min)
• Blow up VRAM without justification

@dataclass
class GPTConfig:
    sequence_len: int = 2048
    vocab_size:   int = 32768
    n_layer:      int = 12
    n_head:       int = 6
    n_kv_head:    int = 6
    n_embd:       int = 768
    window_pattern: str = "SSSL"

Model dim computed from: depth * ASPECT_RATIO (default 8 * 64 = 512, rounded to HEAD_DIM)

Key Components

• RoPE — Rotary positional embeddings
• RMSNorm — used everywhere (not LayerNorm)
• Flash Attention 3 — via kernels library
• Sliding Window — SSSL pattern (3 short, 1 long)
• Value Embeddings — ResFormer-style
• ReluSquared activation in MLP
• Logit softcap at 15
• Residual lambdas — learnable per-layer

# Window pattern: "SSSL"
# S = short window (seq_len / 2 = 1024)
# L = long window (full seq_len = 2048)

def _compute_window_sizes(self, config):
    pattern = config.window_pattern
    long_window  = config.sequence_len
    short_window = long_window // 2

    # Map pattern chars to window sizes
    char_to_window = {
        "L": (long_window, 0),
        "S": (short_window, 0)
    }

    # Last layer ALWAYS uses full context
    window_sizes[-1] = (long_window, 0)

Why this pattern?

• Short windows are cheaper — less memory, faster
• Most layers only need local context
• Periodic long windows let info flow globally
• Final layer always full context for best output quality

# Alternating layers get value embeds
def has_ve(layer_idx, n_layer):
    return layer_idx % 2 == \
           (n_layer - 1) % 2

# In attention forward:
if ve is not None:
    ve = ve.view(B, T, n_kv_head, head_dim)
    # Input-dependent gate (32 channels)
    gate = 2 * torch.sigmoid(
        self.ve_gate(x[..., :32])
    )
    v = v + gate.unsqueeze(-1) * ve

How it works

• Separate embedding table for values (per layer)
• Gated by first 32 dims of input → input-dependent
• sigmoid(0) = 0.5, scaled by 2 = 1.0 (neutral at init)
• Only on alternating layers (saves params)

class MLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.c_fc = nn.Linear(
            config.n_embd,
            4 * config.n_embd, bias=False
        )
        self.c_proj = nn.Linear(
            4 * config.n_embd,
            config.n_embd, bias=False
        )

    def forward(self, x):
        x = self.c_fc(x)
        x = F.relu(x).square()  # ReluSquared!
        x = self.c_proj(x)
        return x

Design Choices

• No bias on any linear layer
• 4x expansion ratio (standard)
• ReluSquared instead of GELU or SwiGLU

# In GPT.__init__:
self.resid_lambdas = nn.Parameter(
    torch.ones(config.n_layer)    # start at 1.0
)
self.x0_lambdas = nn.Parameter(
    torch.zeros(config.n_layer)   # start at 0.0
)

# In forward pass:
x = norm(x)
x0 = x  # save initial embeddings
for i, block in enumerate(self.transformer.h):
    # Mix residual stream with x0
    x = self.resid_lambdas[i] * x \
      + self.x0_lambdas[i] * x0
    x = block(x, ve, cos_sin, window)

Two scalar families

• resid_lambdas — scales the evolving residual stream (init 1.0)
• x0_lambdas — adds back the initial token embedding (init 0.1 after init_weights)

Tokenizer

• rustbpe — Rust-based BPE training
• GPT-4 style split pattern
• vocab_size = 8192 (intentionally small)
• 4 special tokens reserved
• Wrapped in tiktoken for fast encoding

Dataset

• climbmix-400b-shuffle from HuggingFace
• 6,543 parquet shards
• Default: download 10 shards for training
• Last shard pinned as validation

Dataloader: Best-Fit Packing

# BOS-aligned, no padding
# 100% token utilization

while pos < row_capacity:
  # Find largest doc that fits
  best_idx = find_best_fit(
    doc_buffer, remaining
  )
  if best_idx >= 0:
    # Pack it in
    pack(doc_buffer[best_idx])
  else:
    # Crop shortest doc to fill
    crop_and_fill(shortest)

@torch.no_grad()
def evaluate_bpb(model, tokenizer, bs):
    # Sum per-token cross-entropy (nats)
    # Sum target byte lengths
    # Convert nats/byte -> bits/byte

    total_nats = 0.0
    total_bytes = 0
    for _ in range(steps):
        loss_flat = model(x, y,
            reduction='none').view(-1)
        nbytes = token_bytes[y_flat]
        mask = nbytes > 0
        total_nats += (loss_flat * mask).sum()
        total_bytes += nbytes.sum()

    return total_nats / (log(2) * total_bytes)

Why BPB?

• Vocab-size independent — comparing models with different tokenizers is fair
• Measures compression in bits, normalized by actual byte count
• Special tokens (0 bytes) excluded from both sums
• Evaluated over 40 * 524,288 = ~21M tokens

Time-Based Training

• Fixed TIME_BUDGET = 300s (5 min)
• First 10 steps excluded (JIT compilation warmup)
• Training stops when wall-clock exceeds budget
• Steps per run varies by hardware

Gradient Accumulation

# Total batch = 524K tokens
# Device batch = 128 * 2048
tokens_per_fwdbwd = 128 * 2048
grad_accum = 524288 // tokens_per_fwdbwd
# = 2 micro-steps per optimizer step

LR Schedule

• WARMUP_RATIO = 0.0 — no warmup by default
• WARMDOWN_RATIO = 0.5 — cosine decay for last half
• FINAL_LR_FRAC = 0.0 — decays to zero

Safety Rails

• NaN or loss > 100 → immediate exit(1)
• GC disabled after step 0 (avoids 500ms stalls)
• GC runs every 5000 steps
• bf16 autocast for all forward/backward

Why FA3?

• IO-aware: minimizes HBM reads/writes
• Fused kernel: no materialized attention matrix
• Native sliding window support
• Loaded dynamically via kernels package

QK-Norm Before Attention

# After RoPE, before FA3:
q, k = norm(q), norm(k)
# RMSNorm on head dimension
# Stabilizes attention logits

Branch Strategy

# Each run gets a dated branch
git checkout -b autoresearch/mar5

# Good result? Branch advances
git commit -m "increase LR to 0.04"

# Bad result? Revert
git reset --hard HEAD~1

The branch becomes a linear history of improvements. Every commit on the branch represents a positive delta.

results.tsv Format

commit  val_bpb  memory_gb  status  description
a1b2c3d  0.9979  44.0  keep     baseline
b2c3d4e  0.9932  44.2  keep     increase LR
c3d4e5f  1.0050  44.0  discard  switch to GeLU
d4e5f6g  0.0000  0.0   crash    double width

Minimal Dependency Stack

Hardware

• Designed for single NVIDIA GPU
• Tested on H100
• Peak FLOPS reference: 989.5 TF (bf16)
• ~45 GB VRAM at default config