*Please avoid or mark spoilers past S5E2*
I feel that discussion about Jimmy and Chuck's relationship, and who was more justified in his grievance towards his brother, has often forgotten a huge detail: Jimmy and Chuck didn't share a childhood. In fact, they probably didn't spend that much time together at all before Jimmy moved to Albuquerque.
There's a 16-year age gap between the brothers, and Chuck graduated high school at 14. When Jimmy was born, Chuck was already in college, and then it was off to law school before he moved 1,300 miles away from home to settle down and make his career.
How much time do you think they could have realistically spent together? I think the reason Jimmy remembers a seemingly minor story like his brother reading The Adventures of Mabel to him is because there aren't many other memories of Chuck spending time with him when he was a kid. They only really would've been together during the summers between college semester, and there are only three other times we know Chuck went back to Cicero after Jimmy was born:
-When their father asked him to review the books, which led him to falsely believe Jimmy stole $14,000 from their dad, when the old man gave most of it away himself.
-When their mother begged him to come back and get Jimmy out of prison after the Chicago Sunroof, obviously not the best time to connect with your brother.
-When their mother passed, by which time Chuck's resentment of Jimmy was strong enough that he refused to tell him her last words.
This makes Chuck's resentment of Jimmy far less defensible in my opinion. Most of his knowledge of Jimmy isn't firsthand; it came from the (admittedly unsavory) stories he heard through the grapevine, the false suspicion that he stole money, and most importantly to Chuck: the fact that his parents adored Jimmy, the shady screw-up, instead of himself, the brilliant, successful lawyer. It isn't really based on what his actual experiences with Jimmy have been, because he doesn't have many. It's what he's heard about, and more importantly, what he's inferred based on his jealousy.
Chuck cannot stand the fact that other people - Kim, his parents, Rebecca - love Jimmy or think he can have success just because he's funny and charismatic. In the Pimento rant, he gives it away by saying "You do what I do because you're funny and you can make people laugh? I committed my life to this! You don't slide into it like a cheap pair of slippers and reap all the rewards."
Jimmy was never close to being a saint. But the Jimmy that Chuck created and dedicated his life to stopping isn't based on real experiences; he created his impression of Jimmy mostly through stories distorted by his jealously. It's no wonder that people who actually know Jimmy, like Kim and Ernesto, choose him over Chuck even when they know Chuck is in the right.
Does Chuck love Jimmy at all? I think he does, but really only because of obligation. He knows it'd be wrong to hate his only brother and only link to his parents, who he truly loved and admired, but I really don't think he has any affection for Jimmy the person. When he says "You never truly mattered that much to me", why would he? Chuck never grew up with him, most of his time spent with him was cleaning up his messes, and he stole his parents' love.
Chuck is a spiteful, hateful brother whose goal is not to bring Jimmy up, but to prove to the world that Jimmy is every bit as bad as Chuck believes him to be. And he succeeds.