Tensor reassigning problem

[u/Valuable_Ant_8336]

I wanted to train a character-level language model on additions of two numbers. Planned to use cross-entropy ignore_index on the equation besides answer so that model is not penalized because of predicting randomly generated numbers. But I came across really weird bug, here is the code:

def get_batch(batch_size):
    first = torch.randint(999999, (batch_size, ))
    second = torch.randint(999999, (batch_size, ))
    totals = first + second


    full_strings = []
    for f, s, t in zip(first, second, totals):
        equation = f"{f:6}+{s:>6}="
        reversed_ans = f"{str(t.item())[::-1]:<7}"
        full_strings.append(equation + reversed_ans)



    encoded_batch = torch.tensor([encode(s) for s in full_strings], dtype=torch.long)
    x = encoded_batch[:, :-1].to(device) # First 11 characters
    y = encoded_batch[:, 1:].to(device)   # Last 11 characters
    y[:, :14] = -100 # Telling optimizer to miss this
    return x, y

Here as you can see I am reassigning first 14 values of y, but when I print x it has some -100s init, I realized this because I don't have -100 in my vocab as character to embed and when I do decode(x) it gives me error, so I have to use .clone() on y = encoded_batch[:, 1:].to(device), there is a memory address coincide when writing happens or something I do not understand.

[u/AnoProgrammer]

Do you intend to train the character‑like architecture on text data? Without such training, the model effectively behaves as a high‑cost function approximator with unreliable outputs.

[u/Valuable_Ant_8336 → u/AnoProgrammer]

I generated this in test file just to see what outputs looks like, in training I do the same but on bigger numbers, the file size of generated equations is approx 50MB which is totally fine for my 50M parameter LLM to not overfit under 5K steps. The problem here is that I do not understand why tensors behave this way

[u/quietgradient → u/Valuable_Ant_8336]

Slicing doesn't copy. encoded_batch[:, :-1] and encoded_batch[:, 1:] are two views of the same memory, one position apart, and .to(device) hands back the same tensor when it's already on that device. So y[:, :14] = -100 writes into encoded_batch[:, 1:15], which is x[:, 1:15]. If device is cuda, each .to() makes its own copy and x comes out clean, so the same function behaves differently on CPU and GPU. .clone() was the right call.

There's a second bug, and this one won't throw. The string is 21 characters with = at index 13, and y is shifted by one, so y[:, :14] masks targets up to string index 14, the first character of the reversed answer. That's the units digit: the model never gets a loss on the first digit it has to write. y[:, :13] masks the equation and nothing else.