Vizuara AI Labs · RL from AI feedback (v2)

Gemma 2B RLAIF

Gemma 2 2B (legal instruction model) improved against a learned reward model (best-of-N, QLoRA).

2.6B

parameters

4,000

pref pairs

QLoRA

method

N=4

best-of

Validation metrics along the Gemma 2 2B lineage

Each perplexity is measured on that stage's own validation set, so read the trend as 'how well the model fits its own stage's data', not as one curve on one dataset. DPO and RLAIF optimize preferences rather than likelihood, so they log preference margin and reward instead of perplexity. Click a stage to open that model.

Base

n/a

Google's weights, not trained by us

→

QA SFT

ppl 4.26

QA val

→

Instruct

ppl 1.90

instruction val

→

DPO

QLoRA-DPO

preference-trained, no ppl

RLAIF

loss 0.63→0.27

best-of-4 SFT, no ppl

→

RAFT on DPO

ppl 1.22

RAFT val

RAFT on RLAIF

ppl 1.25

RAFT val

instruction or question

optional: text to work on (attached as TEXT)

ready

The response will appear here.

What this is RLAIF v2

The Gemma 2 2B instruction model aligned with RLAIF: a Bradley-Terry reward model trained on the pairs, then best-of-N reward-weighted fine-tuning, on 4,000 AI-feedback preference pairs spanning closed-book QA and instruction-following failure modes (wrong figures, invented citations, broken format constraints, ignored instructions), prompts held out of every SFT set. Lineage: base → QA SFT → instruction SFT → RLAIF v2.

Served scale-to-zero on Modal, so the first request may take ~20–60s while the model wakes.

HuggingFace → SLM Arena 21 → 125M base → 500M base →