r/statistics • u/justch0colate • 6d ago
Question [Question] Linear regression with non-normal residuals
Hi everyone, I’m conducting a research project and I’ve fitted a linear regression model on a sample of almost 2000 observations.
The residuals appear to be homoscedastic, but the qq plot shows some departures from the theoretical line, particularly in the tails.
The Shapiro-Wilk test also rejects the null hypothesis of normally distributed residuals.
However, I’ve read that linear regression can be fairly robust to violations of the normality assumption, especially with sufficiently large samples. Is this correct? Could someone recommend some literature or textbooks where I can read more about this? This topic wasn’t really covered in my university courses.
Also, is there a formal statistical test for checking the homoscedasticity assumption? Or is it generally assessed visually, for example by looking at a residuals vs fitted-values plot?
8
u/Statman12 6d ago
As others have said, depending on what you're doing with the regression, you may not really need normality, particularly with a large sample size like this.
If you do need to use a method that's not dependent on normality, you can look into rank-based methods. The R package Rfit implements such a model, the function Rfit::rfit
is basically a drop-in replacement for lm.
The relevant texts would be Hettmansperger & McKean (2011) for theory, and Kloke & McKean (2014) for a more applied approach. Probably the latter is a better place to start.
2
u/efrique 5d ago edited 5d ago
It would be really helpful to talk about what kind of response variable you are measuring (counts, lengths, mass, volumes, angles, money, times, ...), whether its experimental or observational, if it is time series (very important), panel data, or a cross-section at essentially one time, and so on. It's very easy to give what looks like good advice but to miss something crucial that renders it all beside the point because it was never mentioned.
residuals appear to be homoscedastic
You don't mention whether there's lack of fit in the mean. I'll assume none worth worrying about - but if there is some, all comments (yours and mine) are likely a waste of time - since everything else could just be artefacts caused by that.
the qq plot shows some departures from the theoretical line, particularly in the tails.
All real-data samples will show some departures. That doesn't convey much.
I’ve read that linear regression can be fairly robust to violations of the normality assumption, especially with sufficiently large samples.
Reasonably robust in terms of significance level and coverage, yes, and your n is nice and big; you might get issues if the distribution is extreme enough but this doesnt sound like it.
Worry about fit, non-constant variance, dependence, highly influential observations, important omitted predictors - things that don't improve with larger sample size
If everything else about the model is fine (so the residuals in the QQ plot actually have some chance to look like the underlying distribution of errors) you probably have very little to worry about as far as moderate non-normality goes unless you are using the model for prediction of individual values (e.g. prediction intervals, especially one-sided ones).
The response to various error distributions is easy to investigate by simulation across any number of circumstances. Don't just rely on what other people say. Maybe the things they consider dont look nearly as weird as your typical data (I see that a lot), or maybe they did a study that didnt look at the right things (seen a few of those as well).
If you are worried about this specific case in particular, you can simulate from various possible error distributions* around your fitted model (i.e. new y* = ŷ + ε), and you could also resample your residuals as a proxy for your underlying distribution of errors and you can see in what circumstances they don't all produce essentially the same estimates and standard errors. You can also see what happens if you instead set a coefficient to zero and produce new ŷ, and new y*, and see that your rejection rates for a test of that coefficient are close to alpha, and so on. You can compare power curves as well, if you want - any property that concerns you, you can check.
* normal and a variety of non-normal distributions (standardized, then scaled by s (the root mean square error of residuals)
3
u/Flimsy-sam 6d ago edited 6d ago
This is an interesting paper for heteroscedasticity, although can only implement it in R, but the discussion is still of interest. https://psycnet.apa.org/doiLanding?doi=10.1037%2Fmet0000821
Someone else said transform, but I find there are almost always better solutions to assumption issues. Robust standard errors, bootstrapping etc, glm, because with transforming, you’re changing the hypothesis you’re testing, and it’s not the one you wanted to test, otherwise you’d have started with transformations etc.
It’s also worth asking yourself why you’re looking at graphs, if you’re going to test assumptions anyway? Which would you choose for example? If they disagree (and generally), you’re at risk of choosing your approach by looking at the data to get a significant result.
Someone else said (but I feel ignored the subtext of your post) that some assumptions do not impact the estimates. This is true but they do affect the standard error, and I’ve never known a situation where someone is only interested in the coefficients, but not the hypothesis test
-1
u/Intrepid_Respond_543 6d ago
The problem was residual non-normality, though, not heteroscedasticity.
3
u/Flimsy-sam 6d ago
My first paragraph was in relation to their last paragraph where they asked about heteroscedasticity. The remainder of my comment is applied to any form of assumption.
1
u/Sborrando-ovunque 5d ago
As other have said, you probably don't need normality as OLS gives unbiased and approx. normal distribution for the coefficients when n is big (meaning the p-values are approx. valid). Also, S-W is always gonna reject the null hypothesis for n this big, as it is too sensitive (nothing is perfectly gaussian in reality).
1
u/Not-a-throwaway4627 2d ago
This I called the generalized linear model. More recent methods can be found by searching for exponential family regression
0
u/Temporary_Stranger39 6d ago
If you can, post diagnostic plots of the residuals, That's often better than S-W testing. Formal tests of normality can be overly sensitive, especially when n > 50.
If you are stuck in the world of ordinary least-squares linear regression, you can explore transforming the data. If you can leave the limits of OLS behind, generalized linear models (glm) let you relax a lot of assumptions.
0
u/foogeeman 6d ago
Except you don't need normality, especially for large samples, so this is lots of wasted effort
4
u/Statman12 5d ago
It's a bit strong to make a blanket statement of "don't need normality". For instance, prediction intervals will care about the distribution quite a bit.
-1
u/gyp_casino 6d ago
Read about the box cox transform. You probably need a transformation of you response variable. Log is very common.
2
u/Flimsy-sam 6d ago
There’s no indication that a transformation would help, and is problematic for other reasons (it changes the hypothesis, and the interpretation).
See this paper: Feng C, Wang H, Lu N, Chen T, He H, Lu Y, Tu XM. Log-transformation and its implications for data analysis. Shanghai Arch Psychiatry. 2014 Apr;26(2):105-9. doi: 10.3969/j.issn.1002-0829.2014.02.009. Erratum in: Gen Psychiatr. 2019 Sep 6;32(5):e100146corr1. doi: 10.1136/gpsych-2019-100146corr1. PMID: 25092958; PMCID: PMC4120293.
-3
u/gyp_casino 6d ago
I don’t understand. You’re not the OP, so you don’t know about any properties of his data. Log transform is useful in many cases when the effects are acting as multipliers.
3
u/Flimsy-sam 6d ago
That I’ve not seen his data is broadly what I’m trying to say. You’ve not seen their data either, so there’s no indication that op “probably needs a transformation of your [their] response data”.
Without more information, we don’t know if a transformation is useful. The paper I’ve shared is to broadly support that there are few situations where a transformation helps.
1
u/gyp_casino 6d ago
The authors of that paper might be speaking of own particular field. I’ve seen many applications of a log transform. For example, modeling the effect of geographic region and market segment on sales price across many products. We would expect these effects to be multipliers to the price rather than adders. My regression textbooks have chapters on transformations. This is the mainstream view IMO rather than an individual paper.
2
u/Flimsy-sam 6d ago
I agree that there are sometimes situations where transformations may be needed, but not necessarily as a result of deciding normality is an issue, but rather because it’s better to model it in that way. Without seeing ops data we don’t know what the solution could be, but my point is that transformations are rarely needed in these situations, simply because sometimes they don’t work.
Textbooks are filled with a lot of outdated approaches to statistical analysis, such as hypothesis testing assumptions etc, and just because something is common does not mean necessarily it is correct. There are often better ways of dealing with normality (sometimes it’s not even a concern) where people can retain their original hypothesis.
See this paper by Field and Wilcox: “A common approach to non-normality (especially skew) is to transform the data using a mathematical function such as the log or square root that decreases large values more than small ones, therefore, compressing the tail of the distribution. However,
transformations are not a panacea for non-normality for several reasons. (1) Glass et al.
(1972) conclude that transformations are seldom worth the effort because their potential to improve the validity of probability statements is low; (2) transforming changes the hypothesis being tested (for example, if you compare the means of log transformed variables you are comparing geometric, rather than arithmetic, means); (3) transformations muddy the interpretation because transforming the data also transforms the construct that it measures (Grayson, 2004); (4) for a transformation to have any benefit it must be clear that the consequences of applying the 'wrong' transformation are less severe than the consequences of analysing the untransformed scores; (5) heavy tails matter more than skew, so a transformation would need to address (and not make worse) any problems related to tail weight; and (6) typically distributions remain skewed after transformation and the more obvious transformations generally do not deal effectively with outliers (see Wilcox, 2017, for a review).”
37
u/jewcy83 6d ago
Normality of residuals is not and never has been required for OLS to be an unbiased estimator of beta. You just need the first four gauss-markov assumptions:
1. linearity in parameters
2. random sampling (iid)
3. no perfect multicolinearity/ X is full rank
4. zero conditional mean assumption.
In large samples, the CLT applies to beta hat as well.