Sampreeth Avvari
Resume
Read the story

Architecture

Trains Llama 3 to write counterarguments.A reward model judges each one.

Replies from Reddit's ChangeMyView that earned a delta, paired with replies that did not, train two models from one 4 bit Llama 3 8B: a writer through supervised fine tuning and a judge that scores replies. GRPO then lets the writer practice against the judge, and an eval script scores held out premises against the untrained base.

1,000 premises

per GRPO run, four sampled replies each, up to 256 tokens a reply

0 prompt tokens

in the SFT loss. Only the winning reply is scored

5,000 replies

in the first judge's set, 2,511 that earned a delta and 2,489 that did not

4 bit

Llama 3 8B under every stage, with LoRA rank 64 on top

New York --:-- simulated

A premise through training
Llama RLHF Replies from Reddit's ChangeMyView that earned a delta, paired with replies that did not, train two models from one 4 bit Llama 3 8B: a writer through supervised fine tuning and a judge that scores replies. GRPO then lets the writer practice against the judge, and an eval script scores held out premises against the untrained base. Llama 3 8B 4 bit NF4, Unsloth Weights and Biases every run logged ChangeMyView replies with deltas Preference pairs chosen vs rejected SFT LoRA r64, reply only GRPO 4 replies per premise Eval script BLEU, ROUGE-L, reward Reward model Llama 3 8B, one score Held out premises 100 per eval run deltas winners adapter replies weights scores chosen vs rejected premises metrics NYU GPU cluster

Hover a part to trace what it touches. Click it for its notes.

  • Built by me
  • Managed by Google
  • Outside system
  • Main data path
  • On demand
  • Credentials cross

When it breaks

What each failure does, and what it does not do.

  1. A job dies mid run

    SFT saves a checkpoint every 225 steps and the reward model every 400, so a killed job restarts from the last one.

    Sampreeth/Unsloth/SFT_unsloth.py:135

  2. A score per token

    Loaded through Unsloth, the reward model can answer with a score per token. The reward function keeps the first, so each reply gets one number.

    Sampreeth/Unsloth/grpo_unsloth.py:219

  3. A premise runs long

    Only the first 470 words of a premise go in, so prompt and reply fit in 768 tokens.

    Sampreeth/Unsloth/SFT_unsloth.py:74

What I tuned

Small choices that keep it fast and cheap.

  1. 4 bit base, LoRA on top

    NF4 with double quantization keeps an 8B model trainable, and rank stabilized LoRA at rank 64 does the learning.

    Sampreeth/Unsloth/SFT_unsloth.py:50

  2. Four samples per premise

    GRPO compares four replies of up to 256 tokens from a 512 token prompt, so the judge ranks within a group.

    Sampreeth/Unsloth/grpo_unsloth.py:273

  3. Sampling for eval

    Temperature 0.7, top p 0.9, a 1.06 repetition penalty, no repeated 4 grams and at least 180 new tokens.

    Sampreeth/Unsloth/SFT_unsloth.py:184

Guards

Rules the code enforces, not rules people remember.

  1. Loss on the reply only

    Prompt tokens get the ignore label, so the model learns to answer and never to echo the premise.

    Sampreeth/Unsloth/SFT_unsloth.py:95

  2. Reward path smoke test

    Before GRPO trains, the reward function scores a dummy reply and must print one float.

    Sampreeth/Unsloth/grpo_unsloth.py:240

  3. One base for every stage

    SFT, the reward model and GRPO load the same 4 bit Llama 3 8B, so scores never drift from a mismatched start.

    Sampreeth/Unsloth/grpo_unsloth.py:49