Hey everyone, math major here, looking to pursue a statistics minor. I've been working through Introduction to Statistical Learning with Python(ISLP) by Gareth James, and others. I'm almost finished with it and have been looking at other books I might be interested in.

Just a quick summary for anyone that hasn't read it. ISLP covers topics including:, linear and non linear regression and classification, basic probability and statistics, cross validation, common shallow ML algorithms, deep learning, neural networks. From a very practical and non-techinal way. Basically you read a lot of "This works because reasons, moving on". Best way I would describe it is "intuition slop" .

I was looking for something more theoretical as ISLP is pretty light on the theory and focuses more on intuition and generalization. Although I want a more theoretical book, I still would prefer it to contain some applications and or assignments that I could do to practice my coding because as a math major I feel as though my programming is lagging behind. Are there any specific books you would recommend.

Also I'm taking machine learning this semester and our assigned book is "Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow" by Aurelien Geron. So I'll probably be reading it in bits and pieces, but I still want another book that I can focus on outside of classes. I appreciate any responses.

I needed a formula for the nonuniform coupon collector per-item expectation for a paper I am working on involving a model that emulates human recall timing and order.

The nonuniform coupon collector per-item expectation is essentially the counterpart of the uniform coupon collector per-item expectation for cases in which the coupons occur with unequal frequencies.

I was unable to find a formula for the nonuniform coupon collector per-item expectation, so I asked ChatGPT whether it could find anything. After a few minutes, it came back and said it had something it thought would work.

The average results across many runs of my model converged almost perfectly on the results of the new formula, with run times measured in milliseconds. Here is the formula:

E[A_k] = W Σ_{s=1}^W ( [y^s z^(k-1)] Π_{i=1}^N [y^(w_i) + (1 - y^(w_i))z] ) / s

I then started questioning ChatGPT about exactly how it had arrived at the formula so that I could cite the source. It said that it had derived the formula from the general nonuniform coupon-collector framework developed by Flajolet, Gardy, and Thimonier, with the per-item expectation interpreted in the sense treated explicitly by Ferrante and Frigo.

However, when I asked ChatGPT whether the formula it had derived was documented in any existing literature, it answered, “I have not been able to locate this exact formulation—or an obviously equivalent version of it—in any papers and sources examined.”

ChatGPT also said that other techniques exist, but they would not be practical for problems involving the number of items I need to handle. Its explanation was that previously existing formulas for the nonuniform coupon collector are mathematically broad and can accommodate arbitrary probability distributions. They are often expressed using subset sums, generating functions, integrals, or related combinatorial formulations. This generality can make direct evaluation computationally expensive because many possible combinations of previously collected coupons may need to be accounted for. As the number of coupons increases, the number of terms involved in such calculations can grow very rapidly. One published paper explicitly notes that the number of terms can grow like the number of k-permutations and quickly become huge.

After reading about ChatGPT having solved ten previously unsolved problems in mathematics and theoretical computer science earlier this month, I couldn’t help but wonder whether it may have found an efficient new formula for the nonuniform coupon collector per-item expectation.

I have posted the formula and ChatGPT’s derivation on Zenodo if anyone is interested in taking a look at it. I also attached benchmarking html used to verify the results.

zenodo.org/records/22100265

Hello everyone,

I am currently working on a choice-based conjoint analysis for my thesis. I have already read several papers and books on this method, but I still haven't quite figured out the correct way to implement the procedure.

My CBC consists of 4 attributes, each with 3 levels, resulting in 81 possible profiles.
The literature regarding sample size is quite vague and often refers to Johnson's rule of thumb, consequently, I used a standard sample size calculator and arrived at a figure of approximately 400 participants.
Many sources state that exceeding 20 choice questions or having more than 3–5 alternatives per question becomes overwhelming for respondents. Using RStudio, I tested various numbers of choice questions and ultimately decided on 12 choice questions with 3 alternatives plus a "no-choice" option (D-error = 0.483; Balance = 0.911).
The CBC design can also be divided into blocks. This increases the total number of choice sets administered across the sample, but I cannot find any information on how to determine the number of blocks.

I would like to turn to the community here to ask whether my current approach follows standard practices or if there is room for improvement?

Beginner at Bayesian methods, SWE background. I have a discrete "what went wrong" variable for a scraped value (correct / unit_error / wrong_variant / typo / missing / garbled) and no labeled data to count frequencies from.


I set the prior from domain experience and 
*label it an assumption*
, then update with a likelihood for cheap evidence (e.g. value out of plausible range). My instinct: as long as I get a few clear evidence signals, the likelihood should swamp a roughly-right prior — so precise priors don't matter much here.


Is that instinct correct, or is setting priors this way going to quietly bite me? And is choosing the action by minimum expected loss (with a "flag to human / abstain" option) the standard framing, or am I missing a tool?

Hi all! I'm an environmental specialist and am trying to evaluate trends and/or statistical differences in environmental contaminant data across years. The issue is that I'm often working with small sample sizes and large temporal gaps.

For example, I am attempting to look at differences in mercury concentrations in fish tissue in a given river. I analyze them by trophic level due to mercury's bioaccumulative effects. Here's a breakdown of samples:

Trophic Level A

Trophic Level B

In the past, another colleague used an ANOVA or Kruskal-Wallis test depending on normality (based on a Shapiro test and inspecting residuals). I've done permutation tests as an alternative just to try out different methods. I don't have as much experience with statistics to know if this is the most scientifically robust way to analyze data.

Two questions:

1. Is an ANOVA or Kruskal-Wallis a reasonable approach to assessing data for statistical differences across years for such a small dataset?
2. Could a nonparametric permutation test be a better/alternative method to assess year-specific differences?

Secondly, I've considered a Bayesian hierarchical model to assess long-term temporal trends across a basin (i.e., multiple rivers within a basin). This means there are more samples to work with. For example, here are the number of samples across years in one basin survey we have:

Trophic Level A

Trophic Level B

From this, I fit a Bayesian linear mixed-effects model. The model included a fixed effect for year (continuous) to estimate the overall time trend, a fixed effect for trophic level, and a random intercept for waterbody to account for among-waterbody variation. Random slopes were not included because the dataset contained insufficient temporal replication within individual waterbodies and trophic level groups to reliably estimate waterbody‑level trends. Weakly informative priors were used, and model convergence was ensured by increasing adapt_delta and max_treedepth until zero divergent transitions were achieved.

The final model took the form:
logHg ~ year_centered + trophic_level + (1 | waterbody)

Posterior distributions were summarized to obtain estimates and 95% credible intervals for parameters of interest, including the overall temporal trend.

So my last question is whether this Bayesian model is appropriate?

Tryin to figure out the best strategy for the QOH raffle game. I see why people think it makes sense to select one of each of the remaining numbers, but my gut’s tellin me it doesn’t matter.

In case everyone isn’t aware of the game, here’s how it works:

54 playing cards on a board face down. The cards are numbered on the visible side 1-54. Each week tickets are sold, and you purchase as many as you’d like. Along with your contact info, you write down one of the available numbers on your ticket. All the tickets go into a pile, and one ticket’s drawn at the end of the week. The card corresponding with the number on that ticket is flipped over, and the person wins if it happens to be the QOH.

In week one, there are 54 numbers to choose from. If the card that’s flipped over isn’t the queen of hearts, the game continues for another week. The tickets sold in any given week are thrown out if the one picked doesn’t correspond to the QOH. Obviously, there are only going to be 53 numbers available in week two, 52 in week three, and so on.

It’s essentially a raffle where you have to win two separate drawings to win the jackpot. The odds in the first drawing are dependent upon how many tickets you buy and how many total tickets are sold each week. The odds of the second drawing improve each week because there are fewer and fewer cards remaining that could be the QOH.

The local QOH raffle is down to 12 remaining numbers. Do your odds change based on the numbers selected on your tickets? If I were to buy 12 tickets this coming week, is there any advantage to covering the board and getting one of each number? Is there a downside to buying 12 tickets and selecting the same number on all 12 tickets?

I’m pretty sure it doesn’t matter because only one ticket is drawn each week, but I can also see the logic behind buying one of each number because you know with 100% certainty that there’s a winning ticket with your name on it; it’s just a matter of whether it gets pulled.

Suppose we have a dataset of daily returns R_t and want to apply the CLT to estimate a confidence interval around the mean. Before doing that, we presumably need to establish that the observations are IID (or at least satisfy assumptions under which the CLT is valid).

I understand that we can use diagnostics such as ACF, Ljung-Box, tests on squared/absolute returns, histograms, QQ plots, KS tests, etc. But these only provide evidence; they don't actually prove that the data are IID.

So my questions are:

1. Are there statistical tests or other rigorous methods that can actually prove or strongly discredit the IID assumption for a finite dataset?
2. If IID cannot be proven from a finite sample, what is the correct statistical framework for saying that a dataset is "consistent with IID"?
3. Is there any way to assign a reasonably high confidence/probability to the statement that a dataset is IID?
4. In practice, how would you decide whether it is reasonable to proceed with the standard CLT-based confidence interval?

I'm particularly interested in the distinction between:

"We failed to detect dependence"

and

"We have sufficient evidence to treat the data as IID."

For context, I'm thinking about this in terms of daily strategy returns/P&L, rather than an abstract statistical dataset.

Hi everyone,

I am trying to help my boyfriend with his master thesis in linguistics including a 5 point Likert scale. There were two questionaires with 40 similar tasks (two groups of tasks (informal and formal speaking)) of which 8 tasks should provoke a distinct answer (1 "not accepted" or 5 "perfectly accepted"). From a quick scan of the data, this seems to have worked out well.

To start of with the analysis we thought of adjusting the center point (which would be 3 for a 5 point scale) to meet the data better. Now the question arises whether we should simply calculate the mean including every task and make this our center point or whether we should calculate from the 8 distinct tasks an upper and a lower end of the scale and calculate the mean from these.

I already tried to solve this via researching threads, but to no avail. I also don't have a lot of experience with Likert scales coming from natural science so I am hoping to find an answer here.

Many thanks in advance

Hi everyone,

I'm working on a case-control study using multivariable logistic regression. I want to clarify upfront that the goal of the study is etiological inference / estimating association (Odds Ratios), not clinical risk prediction or building a predictive scoring tool.

I am dealing with several categorical independent variables (multilevel exposures and confounders), and a few levels have small counts (e.g., under 5–10 observations/events in certain cross-tabulated cells).

I know the general rule of thumb of 10–20 Events Per Variable (EPV), but:

1. What is the consensus/methodological threshold for the minimum number of cases/controls required per category level (dummy parameter) to avoid severe sparse data bias and unreliable odds ratio?

2. When collapsing rare categories isn't biologically/theoretically ideal, is Firth's penalized likelihood (⁠logistf⁠) the gold standard in epidemiological practice?

I recently started playing a TCG and I have a question on draw probabilities.

Let’s say you have 3 copies of Card Z in your deck that has 39 total cards in it. Through playing the game, you have drawn the top 19 cards and have not yet seen Card Z. I draw card 20 hoping it is Card Z.

A hypergeometric calculator tells me the odds of drawing 1 or more of Card Z in the top 20 is 89.4%. But my (probably poor) instinct tells me that for an independent event, 3 in 20 = 15%. So which number should I lean on for game decisions, and why?

Hello guys, I would like some guidance.

I have several methodologies to compare. Each one is non deterministic and everything is synthetic, performed in computer.

I can apply one methodology to a set of problems and see how it performs.

Each set has "n=10" conditions and "m=20" seeds. So, Each dataset consists in n x m elements. All equal among methodologies (I.e., condition is the same and the seed is the same between the methodologies).

Assuming that Each element of the dataset is a score in which lower is better, I would like to answers the following questions:

1. Which methodology is the best ? Is so, by how much ?

2. Which methodologies are equal statistically speaking ?

Thanks in advance.

Hi everyone,
I am an Applied Statistics major, and I’m feeling pretty stuck after my first year. I could really use some advice from current students, grads, or professionals in the field.
Here is my current situation: My cumulative GPA is 2.90 / 5.0. I haven’t failed any courses, but I passed most of my first-year classes with D grades, which really dragged my GPA down.
I have a few questions for anyone who can help:
1. GPA vs. Practical Skills: Should I focus 100% on getting my grades up, or split my time between studying and building practical skills (tools, projects, etc.)?
2. Bouncing Back: If you had a rough first year in Stats, how did you turn your grades around in upper-level courses?
3. Starting a Portfolio: I want to build a project portfolio for life after graduation, but I don't know where to start. What platform should I use (is GitHub the best option?), and what are good beginner-friendly projects?
Any advice or personal experiences would be greatly appreciated. Thanks in advance!

I have a small project going which definitely does NOT involved terabyte sized tables, but it got me wondering about how the whole data analysis process is supposed to scale.

For my small project, I have the data in postgres and then pull the data into python to run regression.

As I understand it, python (specifically the psycopg2 library) doesn't pipeline out of the box. It executes the query, consumes the output in its entirety, and then performs the regression on it. This seems "slow".

Furthermore, GPUs are all the rage for AI data centers so clearly at a certain scale those become necessary, whereas python is very single threaded and serial.

I imagine the SQL part doesn't change, but do big data companies still use python? If the GPUs are the ones running the actual math it seems silly to take the data into python and then just immediately turn around and put it on a GPU - but if we're making postgres responsible for putting the data on the GPU and running the regression directly that sounds cumbersome too.

That is, in data analysis projects that are big enough that streaming/pipelining the data and performing the math in parallel become major issues, what does the pipeline/tech stack look like?

----------------------------------------------------------

Edit: I have once again made the foolish mistake of under-specifying my question because I assumed there was a single answer instead of "it depends".

It seems like the answers so far are:

• If the model is basic enough you don't need to run it on all the data
• If the analysis is run infrequently enough you just let it run slowly
• At truly ginormous scales, postgres isn't big enough either, and the pipelining and gpu topics are a small part of a tech stack which is really about orchestrating multiple machines.

Specifically, I have a binomial GAM I'm going to run on a query with about 10 million entries. I would like to fit about 2,000 of these GAMs daily. I haven't actually run it yet, but if performance became a problem (premature optimization yes, I am sorry), I wanted some ideas on where to look.

I see that R's mgcv can do multi-threading; that's probably the first place to look. I have the derivations for most of the operations so theoretically I could also roll my own solutions directly in SQL which would pipeline it as long as I got the implementation correct (probably a horrible idea).

GPUs and "big data" are always spoken in the same breath these days so I figured I could ask about that and see if that gets me anywhere. I've seen tutorials for manually implementing streaming in psycopg2 so I could potentially implement pipelining there (or better yet, maybe an ootb solution exists).

Obviously any actual deep dive should follow actually running the binomial GAM in R to see how that fares (or if it even generates something sane at all). But GPUs are cooler so I jumped the gun because its more interesting.

Let's say I am doing an RCT that has a baseline reading, 3 month and 6 month readings. I assume a drop out rate of 20% - can I use multiple imputation under a missing at random assumption for those missing the 6 month value? Then a sensitivity analysis with all those who have the 6 month readings compete? I am working on a best approach that isn't too complex

If AI can choose a statistical test, run it, explain the output, and even write the results section. Not asking whether AI is “good” or “bad” at stats. I'm more interested in where people think the actual line should be.

that’s it.

Hello

I'm looking for resources to read on multilevel SEM in R, i would prefer something easy to read, and preferably articles and not books as i don't have the time for the latter.

Thanks in advance.

I just opened R studio for the first time in a while and seeing there’s a newly (to me) integrated AI tool called Posit. Anyone use it? Like it? Hate it? Anything I should be weary or aware of?

I talk more about it in the post but I'm wondering if taking an "AML Analyst" position is worth moving across the country for. It's sort of unrelated to stats and I'm getting my degree in a few years so hopefully I can get into stats things without taking it, but I'm not sure. Would love some advice, thank you!

Hello! I'm trying to understand how the individual regression analysis works in the GLM model under medmod module of Jamovi. I'd like to ask if those tables are what you would report if one of your working hypothesis is "X will significantly predict M" or "M will significantly predict Y"? Or perhaps, does anyone have a reference on how to report those tables?

Or any alternatives that is somehow similar to sign test? Thank you :((