autoregressive language modelling

When we talk about generative language modelling—— speaking of which, I highly recommend for you to check out my previous post on my experiments with Reverse Engineering (RE) with LLMs, where I went thru some handwaving on the theory of the modern approach w/ transformer-based LLMs—— the commonly-used approach in the current state of this space is that we usually do things autoregressively.

This is because the model that we use in this space, generative language modelling, are decoder-only models. Where they’re only using the decoder component of a traditional transformer encoder-decoder architecture (where, if you’re familiar w/ that infamous “Attention Is All You Need” [1] paper? Yeah, that’s what they proposed in there).

The reasoning here is that, the decoder is really the only generative part in a model like this, the encoder itself really just provides some level of “residuality” which connects the whole latent w/ the current generational process, but both can, in a way, be used to model the same processes… Mmm… man my brain b wacky today… we’re getting sidetracked, anyway, we’ll come back to this down below, this is a bit of a spoilers.

Anywho, “autoregressively” as in? Well, to make it quick for you, here’s a demo I had before (do ignore the sample lmoa), try hovering over each word:

When we do language modelling with self-attention autoregressively, we apply what’s called masked self-attention. This variant of self-attention is basically the same as the same old self-attention that we’re already used to, but the attention computation is only done on itself and the prior inputs in the sequence. You can see this phenomenon by hovering over each word in the sequence above.

That “itself-to-itself-and-everyone-before” is because of the fact that causal language modelling usually has to do its modelling over a cause-effect assumption, hence “causal”, where the next token is caused by prior inputs in the causality sequence. Kinda why we can somewhat assume a $P(X_i)$ is $P(X_i|X_{i-1})$ in markovian modelling (read: Markov Property), where $X_i$ inherently depends on the $X_{i-1}$, which means that, really, to get to an $X_i$, one can, for the most part, reduce it down to that chaining approach in a Markov chain.

(…although this is really just a convention tbh, as this is really just the most intuitive/tractable way of viewing it, and we’ll go over why not in a bit down below)

To do a prediction, therefore, one would get the output logits of the next token by computing the score of a “hypothetical” token with respect to the rest of the inputs which has been processed, which is then shoved into an FC to get the final logits. I.e., say, given the following tokenized input (with the assumption of a basic word-by-word tokenization method):

Positions:    [0,    1,    2,    3,     4,    5,    6]
Tokens:      ["I", "love", "femboys", "->", "J'", "aime", "les"]

To predict some token at position 7, we run a self-attn relationship computation to tokens 0-6 (and ofc, themselves for each).

After which, as you can see up there, it’s (the latest latent state) shoved into an FC head at size of the vocab, which then gets logits, then softmax (or some other normalization shid you got) to get the probdist. …So smth like:

{
    # Where "Tokens": Prob
    "Femboys": 0.5,
    "Girls": 0.3,
    "Guys": 0.1,
    "Humans": 0.05,
    "Cats": 0.01,
    # ... think like, some tens to hundreds of thousands more down here lmao
}

(Ofc, assuming we’ve alrd mapped the IDs back to the tokens themselves, yadda yadda)

…so both ways?

Yis! GO BOTH WAYS. That, is indeed, the “fix” for that “flaw” in the modelling. This, in fact, is why we do encoder-decoder. Encoders, do a full self-attention computation, which then gets shoved into the decoder as part of a sort of… “residual” latent connection via the cross-attn.

This ensures that to model that forward dependency, we don’t have to first fold and collapse it down. This, in turn, allows for a lower level of dilution in the causality of the sequence, at the cost of having worse asymptotic complexity.

… hmmm, so bad bad, yeah? Mhm, I mean, think abt it, just to get there, we’d have to shove three attention mechanisms, one from the encoder, one from the cross attn, and one from the decoder. In fact, this whole thing is the reason why this scheme would never have really worked out for generative modelling, processing is just, too much for it to work out.

… confus, y not autoregressive, then, again?

Well, for one, autoregressive modelling, does, again, work. I’m not saying that you can’t model linguistics autoregressively, it’s just that this model has some inherent “flaws” that it just, can’t address, without assuming “transforms” done on the input (i.e., that “folding”).

We need a method which considers the whole of the input and the “output” sequence, that is, we sort of… predict, the whole future window at some $X_n$ with $n$ tokens, in order to allow for it to hold that forward dependency. But how? How would one allow the prediction of future tokens, … basically at the same time, for the entire window…?

… so what am “I” proposing? Well, let’s see… let’s talk abt diffusion models. 💀

diffusion language modelling? (DLM)

Anyway, well, here’s how we can view this. Think about it like this, say given the input of

Because he was feelin ADHD af,

We wanna predict the whole of Because he was feelin ADHD af, Astolfo went to scroll reels all night, buuut still keep that forward dependency relation which we went over up there.

Here, we can view the “entirety” of the $x$ as the input + some allocated “budget” which is at least at the size of window which we want to predict with the bidirectional dependency.

… seems straightforward enough, so why not shove a U-Net in there to do “denoising” on a language input, then? Well, mainly, two problems:

1. The usual way we do things for language modelling, where we assign some arbitrary units we call “tokens” to process things, is inherently discretized. What do I mean?

   Well, look at it simply like how tokens can’t just, change in-between states, as in, well, say I had femboy as a predicted thing, it can’t just, be in-between of femboy and female, assuming, say, like, they’re ID’d at 0 and 1, add 0.5 to an input of 0, take the middle ground of some… 0.5, femole or smth, ykno. It’s gotta be one or the other token. In contrast, in image or audio or other continuous domains, we can occupy a space between the numbers, hence the “continuity”, this allows the flexibility when we modify the inputs during diffusion.

2. What’d “noise” even BE in language modelling? What, typos? LMOA, yeah no, imho, there really isn’t a great intuitive way of interpreting it. The reason why noise inherently works in the image domain, is that noise, in and of itself genuinely does model the “movement” of the data distribution in images (and audio), there isn’t really a good way of interpreting noise in this domain that goes “bidirectionally”.

   Tying to the previous one, “noise” like typos model the destruction of input, but it inherently assumes that the “destruction” are therefore then modelled at the token-level, here, you can’t really start from gibberish then typo your way into coherency when your space doesn’t 1-to-1 align to your tokenization space. (femboy = fem + boy, but female = fe + male, can’t modify fem into a female from a “typo”) But this is a discussion for another day, I think there’s definitely an argument to be had here tbf, but I’m tired, my team’s tired, needa collect ts tonight, so I digress.

… So solution? Well, one way is to just, ignore the second problem, allow for token-level “noise” schedule, and just, reframe the “noise” schedule as being applied across the entire sequence, i.e., where only some number of “noised” tokens within the budget window is then “decoded”, creating “noise” in the input sequence, yet still staying within that discretized constraint.

This, finally, in fact, is what LlaDA [2], the model we’re working with, has chosen to do. Wat? See the following demo.

In order to allow for the described behaviour above, during pretraining, we “mask” some number of tokens, then, we train the model to understand which tokens to “unmask” given some input with masked tokens.

During inference, the “unmasking” process here, is really just that the allocated “budget” tokens are all masked tokens, where the ones actually processed for decoding are the ones “unmasked”, some top $k$ prediction logits (where they’re most certain) are unmasked, where the rest are “re-masked” for future timesteps.

To be clear, I am skipping a couple caveats here, you can understand the generational process better if you check out the following code I derived from their demo:

def generate_with_llada(
    model,
    tokenizer,
    prompt,
    gen_length=64,
    steps=32,
    temperature=0.5,
    cfg_scale=0.0,
    block_length=32,
    remasking='low_confidence',
    constraints=None,
    device='cuda'
):
    """
    Generate text using a trained LLaDA model.
 
    Args:
        model: The trained model
        tokenizer: The model's tokenizer
        prompt: Input text prompt
        gen_length: Length of text to generate
        steps: Number of denoising steps
        temperature: Sampling temperature (0.0 = deterministic)
        cfg_scale: Classifier-free guidance scale
        block_length: Block length for semi-autoregressive generation
        remasking: Remasking strategy ('low_confidence' or 'random')
        constraints: Dict of {position: word} constraints
        device: Device to run generation on
 
    Returns:
        Generated text
    """
    MASK_ID = 126336  # LLaDA's mask token ID
 
    # Process constraints
    if constraints is None:
        constraints = {}
 
    # Convert string constraints to token IDs
    processed_constraints = {}
    for pos, word in constraints.items():
        tokens = tokenizer.encode(" " + word, add_special_tokens=False)
        for i, token_id in enumerate(tokens):
            processed_constraints[pos + i] = token_id
 
    # Encode input prompt
    input_ids = tokenizer.encode(prompt, return_tensors='pt').to(device)
    prompt_length = input_ids.shape[1]
 
    # Initialize sequence with masks for the response
    x = torch.full((1, prompt_length + gen_length), MASK_ID, dtype=torch.long).to(device)
    x[:, :prompt_length] = input_ids.clone()
 
    # Apply initial constraints
    for pos, token_id in processed_constraints.items():
        absolute_pos = prompt_length + pos
        if absolute_pos < x.shape[1]:
            x[:, absolute_pos] = token_id
 
    # Mark prompt positions for CFG
    prompt_index = (x != MASK_ID)
 
    # Calculate blocks
    block_length = min(block_length, gen_length)
    num_blocks = (gen_length + block_length - 1) // block_length
    steps_per_block = max(1, steps // num_blocks)
 
    # Process each block
    for num_block in range(num_blocks):
        block_start = prompt_length + num_block * block_length
        block_end = min(prompt_length + (num_block + 1) * block_length, x.shape[1])
 
        block_mask_index = (x[:, block_start:block_end] == MASK_ID)
        if not block_mask_index.any():
            continue
 
        # Calculate tokens to unmask per step
        mask_count = block_mask_index.sum(dim=1, keepdim=True)
        base_tokens = mask_count // steps_per_block
        remainder = mask_count % steps_per_block
        num_transfer_tokens = torch.zeros(mask_count.size(0), steps_per_block, device=device, dtype=torch.int64) + base_tokens
        num_transfer_tokens[:, :remainder] += 1
 
        # Process each step
        for i in range(steps_per_block):
            mask_index = (x == MASK_ID)
            if not mask_index.any():
                break
 
            # Apply CFG if enabled
            if cfg_scale > 0.0:
                un_x = x.clone()
                un_x[prompt_index] = MASK_ID
                x_ = torch.cat([x, un_x], dim=0)
                logits = model(x_).logits
                logits, un_logits = torch.chunk(logits, 2, dim=0)
                logits = un_logits + (cfg_scale + 1) * (logits - un_logits)
            else:
                logits = model(x).logits
 
            # Apply temperature and get predictions
            if temperature > 0:
                logits = logits.to(torch.float64)
                noise = torch.rand_like(logits, dtype=torch.float64)
                gumbel_noise = (-torch.log(noise)) ** temperature
                logits = logits.exp() / gumbel_noise
 
            x0 = torch.argmax(logits, dim=-1)
 
            # Calculate confidence scores
            if remasking == 'low_confidence':
                p = F.softmax(logits.to(torch.float64), dim=-1)
                x0_p = torch.gather(p, -1, x0.unsqueeze(-1)).squeeze(-1)
            else:  # random
                x0_p = torch.rand(x0.shape, device=device)
 
            # Don't consider positions beyond current block
            x0_p[:, block_end:] = float('-inf')
 
            # Apply predictions to masked positions
            x0 = torch.where(mask_index, x0, x)
            confidence = torch.where(mask_index, x0_p, torch.tensor(float('-inf')).to(device))
 
            # Select tokens to unmask
            transfer_index = torch.zeros_like(x0, dtype=torch.bool)
            for j in range(confidence.shape[0]):
                block_confidence = confidence[j, block_start:block_end]
                if i < steps_per_block - 1:
                    _, indices = torch.topk(block_confidence,
                                         k=min(num_transfer_tokens[j, i].item(), block_confidence.numel()))
                    transfer_index[j, indices + block_start] = True
                else:
                    transfer_index[j, block_start:block_end] = mask_index[j, block_start:block_end]
 
            # Update sequence
            x = torch.where(transfer_index, x0, x)
 
            # Maintain constraints
            for pos, token_id in processed_constraints.items():
                absolute_pos = prompt_length + pos
                if absolute_pos < x.shape[1]:
                    x[:, absolute_pos] = token_id
 
    # Decode final text
    generated = tokenizer.decode(x[0, prompt_length:], skip_special_tokens=True)
    return generated

If you look at the above, you’ll notice that, this… isn’t “parallelized”… entirely. I.e.? Well, see how there’s block_size in there? Yeah, what is that? Well, you see, in theory, we’d want to process the entire sequence’s attention in parallel, correct?

… Well, problem. $O(N^2)$ bb, it’s still there.

If we were to compute the entire sequence’s attn for the masked budget as well, that’s the whole thing all, at, once. Bad, really bad. It’s like each step you take is the whole sequence shoved in there again lol.

(Ofc you can somewhat KV cache this I’m p sure, but their implementation, since it’s just proving that DLMs work, is not compat w/ KV caching, not to mention that I think if you were to think abt it, the attn could end up invalidating itself since the input window is changed (? I.e., once you’ve chosen some k tokens, it’s like, a diff sequence? Handwavy ig, but ygwim, right?) but I’m spiralling.)

Soooooo, instead, we process the budget window in blocks. These “blocks”, are what’s processed in “parallel”. In effect, this turns LlaDA into a “semi-autoregressive” model, since the “denoising” (full self-attn) occurs not on the entire allocated budget, but just for each block.

Anyway, with the above, this means that LlaDA, essentially, in practice, basically just functions as an encoder model, where the self-attn processed is a full sequence-to-sequence self-attn computation (regardless of the block-wise processing, we’re still doing full self-attn for each block, yknow). This is what allows that forward dependency.

… You may have a liiiiittle interesting question in the back of your mind now, “wait, so we’ll see that the tokens are decoded in parallel?” GOOD QUESTION! Well, not necessarily.

Mmmm, well here’s the thing, when we say that the markovian way of doing $P(X_i|X_{i-1})$ (i.e., collapsing the autoregression into it) is “flawed”, it doesn’t necessarily imply that it, therefore, just won’t work, for language modelling. It does work, and we talk from left-to-right, there’s nothing… inherently wrong, with that. The problem, arises from the fact that conceptually, we do not think, left-to-right (or whatever directionality your language goes).

Thoughts— and this gets a bit conceptually handwavy— don’t have a direction. It’s a high-dimensionality space. Concepts can be “reduced” into words, but you’re otherwise going to lose some degree of semantical fidelity when you collapse the latent into tangibility.

Think of the following image.

You can caption it as something like:

'''
The image depicts a cartoon scene of a dog sitting on a red chair in a room filled with flames. The dog is wearing a hat and has a speech bubble that says "This is fine". A small table with a yellow mug on it is situated next to the dog. The room is filled with flames, creating a chaotic and intense atmosphere. The dog's expression appears to be one of confusion or bewilderment.
'''
# (captioned w/ moondream btw)

But you won’t be able to capture the degree of “yellowness” which the mug has, or what shape the dog was, or how “red” that chair was. You can only describe them in discrete descriptors. This is the inherent limitation, not in language modelling, but language itself. You must collapse conceptuality into tangibility when reducing the latent into describable words.

The “flaw” in autoregressive modelling, therefore, is not in the fact that we’re structuring outputs left-to-right. In fact, this is fine, this is literally how it’s supposed to be done, your tangible outputs must match the domain’s space. We literally write left-to-right (or, er, again, wherever tf it goes).

It’s just that, we don’t think left-to-right. We don’t create context only on the left side, it can be on the right as well, just that maybe, sometimes, we need to bridge to it, maybe it’s a bit too fuzzy and we need to refine the left-hand side first before getting the right on more solid ground, maybe it’s pretty solid already but you just haven’t “infilled” the rest of the text. Regardless, the right-hand side is there, it’s just, it might not be visible to the other side, your audience.

Now, how does this tie into how we do it for this? Well, think of it like this, you shove your “thinking”, into the masked tokens. While the masked tokens are still “masked”, in practice, they do serve a purpose.

That is, a way to probabilistically store that “thinking” of yours way ahead. IRL, it’s more often than not, that, you don’t, like, write or talk, in a way that, you put some “waypoint” words somewhere in the middle or at the end of the text and write your way there. You write somewhere, to a conceptual waypoint. It’s still there, it’s just not tangible yet.

Soooo, LlaDA, while it does have the capability of doing the above (and it does, given certain inputs, sometimes!), in practice, you’ll see it still do whatever is most convenient to work with in the language’s domain.

A nice possible upside to doing modelling like this, is that the loss of forward dependency in autoregressive models have basically led models to fail during Chain-of-Thoughts (CoT) [3] once it’s committed to a single wrong step in the flow to an answer (as every $x_i$ depends on $x_{i-1}$), can be inhibited, without having to introduce an intermediary state to “interrupt” the model’s thinking process (a la modern thinking models like Deepseek’s R1 [4]), or having to aggregate (a la Self-Consistency [5]), it inherently holds an understanding of what it’ll do in the future and can inhibit itself as it decodes the window.

LoRA?

Yes, LoRA. Wtf is LoRA? LoRA stands for Low-Rank Adaptation [18], which, okay, there’s a lot of theory to go over here, but I had from last semester’s final project (yeah, the one I mentioned I submitted for publishing from last post, it’s at minor revisions stage btw, gona edit this post too if it gets accepted in the future) that explains this pretty well, so I’m just gonna copy off that.

Low-Rank Adaptation (LoRA) was a method introduced by Hu et al. (2022) of Microsoft, which proposes to… well, let’s start with the diagram actually, kinda hard to intuit without a visual:

Let’s start with the basics, when we do backprop on weights, we have to do a backprop on the weights matrix, right? Now, the problem with this, is that, if we have a LOT of weights, this quickly gets very, very, impractical. What Hu et al. proposes is that, we can approximate this backprop change to the matrix with a low-rank decomposition matrix.

Wtf is that? Idk how to explain it exactly, but basically we try to approximate/represent some larger matrix with smaller ones, α and β (we’ll just refer to them as A and B to make it easier on me lol), at size rank r.

Let me show you a quick example, say we have this $\Delta$ on the weights when we calculate it by the usual backprop method:

[[1, 2, 3, 4],
 [2, 4, 6, 8],
 [3, 6, 9, 12],
 [4, 8, 12, 16]]

This, if you notice, is basically just, [1, 2, 3, 4] when $\otimes$ (outer product) to

[[1],
 [2],
 [3],
 [4]]

right?

So that’s what a low-rank approximation looks like for r=1. Where, A = [1, 2, 3, 4] and B =

[[1],
 [2],
 [3],
 [4]]

We optimize A and B such that when $A \otimes B$, we get as close as an approximation as possible to the actual backpropped $\Delta$ on the weights.

(P.s., there is another thing that I’m gonna skip here, which is LoRA rank-stabilization, but for the sake of my sanity, and maybe even yours, you can just, read up on Kalajdzievski, D. (2023) from Tenyx lol)

(There are also some assumptions there, since we’re working with self-attn, such it’s dxd for the $W$, it’d generalize to a dxk matrix, but the gist of it is in there fwiw.)

Here, LoRA can be used to target an approx change on any component really, but here we took the simplest QKV target on their respective projection layers.

LoRA+

Same deal, I need to collect this by tonight bruh :kms: so we’re gonna copy off the notebook again,

Okay, whew, now that you’re up to speed to LoRA, we’re moving to LoRA+ :kms: This was introduced by Hayou et al. (2024), and I’m not gonna go over the maths again for the sake of my sanity :plsgodno: so but, basically, this is what we went over up there, but for LoRA+, we track two different learning rates for A and B, where B’s LR is λ times that of A, this, according to their results, got faster convergence rates compared to the base LoRA.

There you go! You’re basically up to speed with our methodology now lmao.

PiSSA

I’m gonna kms— Okay, so PiSSA [19], to make this quicker for the both of us and keep my sanity, the way you can view this is that…

Let’s see, in initializing the LoRA weights for both of A and B, there really isn’t that much guidelines as to how one’d go abt doing it. Usually, as per the usual w/ weight init, we set up via gaussian over A and zeros on B, tho, according to PiSSA, this is actually not necessary.

Look at it like this, if we already have some number of pretrained parameters we can work off of, why not obtain some approximation of where we’re starting from? Here, PiSSA proposes the use of Singular Value Decomposition (SVD) to get a matrix which essentially is a reduction of the initial $W$ matrix.

Here, to do the decomposition, there are different ways, but generally speaking, as per any reduction technique, it gets pretty slow as your input scales up. Here, you can basically restrict the number of iterations which it uses to reduce the initial matrix, applied via a fast SVD method.

data curation

So to do data curation, we sampled off the Cendol Collections [7] dataset, via the following procedure:

1. Grab the subsets of data from Cendol Collection which contains Indonesian only, this includes the following subsets: indo_puisi, wikihow, wikipedia_id, safety_prompt, identity_prompt, and dolly.

   The rationale behind this was to accelerate the convergence rate, considering that later on we’ll be using LoRA to add additional parameters that won’t be able to learn significantly different domains from the base domain (otherwise it’d have to very much overfit), as such Indonesian would be closer to English than, say, Maduranese or Javanese.

   This results in a $\approx 50\%$ reduction in the sample size, so $\approx 6 \text{ million}$ samples from the initial 12 million provided from Cendol Collection.

2. Sample from those subsets for a $3\%$ sample via stratified sampling on template_names, getting us some $\approx196 \text{ thousand}$ samples.

   The rationale behind this was to effectively represent each subset in the next step, as each template_names here represent a tighter scope within each subset (i.e., we wanna ensure a wider variability by allowing a more diverse sample set, hence allowing template_names as there are more of those).

3. KMeans cluster the instances by embedding them with IndoBERT-p1-cased at K=10, grabbing for each cluster 1,000 samples and augmenting from other clusters if the current cluster has insufficient instances, resulting in a sample size totalling at 10,000 instances.

   By doing this, we ensure that for any semantical clusters, there’s at least a degree of representation within the training set, this ensures good diversity and higher representation even at a lower scale set.

data prepro

To make it work with LlaDA, we shoved it thru some prepro to ensure that the formatting works out, this is especially because to do the tuning, this is essentially just Supervised Fine Tuning (SFT), and looking at the paper [2], it looks pretty standard, so we shoved it thru the same prepro as the one we had for the ML final project.

#@markdown Util functions
def convert_to_conversations_batched(examples):
    conversations = [
        [
            {"role": "user", "content": input_text},
            {"role": "assistant", "content": output_text}
        ]
        for input_text, output_text in zip(examples["input"], examples["output"])
    ]
 
    return {"conversations": conversations}
 
def check_valid_conversations(examples):
    # examples is a dict with batched features
    return [all(msg["content"] is not None for msg in conv)
            for conv in examples["conversations"]]

{'dataset_name': 'nusa_t2t_v2',
 'subset_name': 'wikipedia_id',
 'prompt_id': 'cd34b736-1ce0-48c1-b9ed-eab9342a419f',
 'template_name': 'wikipedia_subsection_4',
 'dataset_key': 'train',
 'input': 'Gw mau tau Komposisi dari HD 40307 g',
 'output': 'Penemu kode Hugh Jones, dari University of Hertfordshire di Inggris, menduga: "Orbit yang lebih panjang dari planet baru berarti bahwa iklim dan atmosfernya mungkin tepat untuk mendukung kehidupan."\n\n..."'}

[{'content': 'Gw mau tau Komposisi dari HD 40307 g', 'role': 'user'},
 {'content': 'Penemu kode Hugh Jones, dari University of Hertfordshire di Inggris, menduga: "Orbit yang lebih panjang dari planet baru berarti bahwa iklim dan atmosfernya mungkin tepat untuk mendukung kehidupan."\n\n..."',
 'role': 'assistant'}]

[{'content': 'Gw mau tau Komposisi dari HD 40307 g', 'role': 'user'},
 {'content': 'Penemu kode Hugh Jones, dari University of Hertfordshire di Inggris, menduga: "Orbit yang lebih panjang dari planet baru berarti bahwa iklim dan atmosfernya mungkin tepat untuk mendukung kehidupan."\n\n..."',
 'role': 'assistant'}]

<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 July 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nGw mau tau Komposisi dari HD 40307 g<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nPenemu kode Hugh Jones, dari University of Hertfordshire di Inggris, menduga: "Orbit yang lebih panjang dari planet baru berarti bahwa iklim dan atmosfernya mungkin tepat untuk mendukung kehidupan."\n\n...<|eot_id|>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 July 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nGw mau tau Komposisi dari HD 40307 g<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nPenemu kode Hugh Jones, dari University of Hertfordshire di Inggris, menduga: "Orbit yang lebih panjang dari planet baru berarti bahwa iklim dan atmosfernya mungkin tepat untuk mendukung kehidupan."\n\n...<|eot_id|>

tensor([128000, 128000, 128006, 9125, 128007, 271,  38766, 1303,  33025,  2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696,  25, 220, 1627, 5887, 220, 2366, 19, 271, 128009, 128006,    882, 128007, 271, 51, 337, 647,  86088,  79451, ...], device='cuda:0')

4. Train on Responses Only

This is pretty intuitive, basically, our training focuses on the augmentation on how the model responds to user input, therefore, we have to “mask” out parts of the prompt that’s irrelevant to that (since we don’t need the model knowing how to make inputs lol). We mask it out with -100 on the irrelevant portions, which basically turns (assuming tokenized) this

tensor([128000, 128000, 128006, 9125, 128007, 271,  38766, 1303,  33025,  2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696,  25, 220, 1627, 5887, 220, 2366, 19, 271, 128009, 128006,    882, 128007, 271, 51, 337, 647,  86088,  79451, ...], device='cuda:0')

to

tensor([-100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,
          -100,   -100,   -100,   -100, ..., -100,   -100,   -100,   -100,
          -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,
          -100,   -100,   -100,   -100,   -100,   -100,    271,  84583,   2857,
         74976,  74529,   5230,    718,  55679,    869,    258,  31475,   1208,
         30728,    220,    966,  92033,    220,  ...], device='cuda:0')

this will later then be interpreted by the model as an attention mask, and anything that’s -100 is basically excluded from its training, so smth like (where 0 is the masked out portion and not trained on.)

tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ..., 1, 1, 1, 1, 1, 1, ..., 0, 0, 0, 0, ...], device='cuda:0')

This is also to follow the SFT schema provided by the LlaDA authors, where the “unmasking” is only done on the response. This makes sense, since the response is the budget window which we allocate for the model, so training for the prompt’s loss would make no sense, the model doesn’t need to understand how the prompt was constructed. This is different from pretraining, as during pretraining, we want to ensure it has a general sense of understanding regarding language modelling.

evaluation

In order to evaluate the performance of the model before and after the application of the adapters, we used 4 metrics, ROUGE, BLEU, METEOR, and F1.

Here:

1. ROUGE is used to evaluate the recall capabilities of the model, this is measured via the overlap between the reference and the generated texts. Wud be most useful wen we wanna consider using this for RAG, for example. ROUGE has three different types, -1, -2 and -L depending on what’s used to determine the overlap.
2. BLEU is used to determine the quality of the cross-lingual transfer, this is based on how well the grams match up and the brevity of the generated text. Such, if the model gets high BLEU, that means it’s learned the language’s syntax and verbosity better.
3. METEOR is really just a modernized version of BLEU, gets a harmonic mean of precision and recall of the gram , so it’s like a compromise between the other two up there.
4. F1 here is calculated via embedding w/ all-MiniLM-L6-v2, where for something to be determined to be true, we grab the cosine dist from the reference’s embedding for the generated text, and if it passes the threshold of 0.5, it’s determined to be true.

For all of them, we assumed a 0.7 temperature on the sampling over each probdist, and 32 timestep diffusion.

conclusions

As we can see up there, PEFT does work in the usecase of cross-lingual transfer learning of a generative LLM. However, as per the discussion section, there does need to be some degree of adjusting one’s expectations when it comes down to the resulting performance which one could get out of the training regimen.

The application of unrelated external adapter weights into a model often incurs some level of domain loss, this is due to the fact that the newly added weights won’t have connections which have weights already adapted with the rest of the network. If the rest of the network alrd had some level of preconveived “notions” which can’t be changed with the added parameters, the addition of the parameters may only inhibit the general performance of the model, and we believe that some of the abnormalities observed within our results reflected at least part of that.

In practice, we do have to consider also the fact that if a model barely had any data to transfer with to begin with, the small number of added trainable parameters from the LoRA often won’t be able to shift the underlying lingustic understanding distribution far enough to properly transfer its knowledge over.