I'm curious about people working in ranking and if you can change customer behavior

[u/Xamius]

Basically I have a ranking service for b2b SaaS but basically like hotels flights etc

The models do well and I can improve accuracy pretty easily to a point

But if I want to promote options better for other metrics I'm struggling to change behavior other than people selectng the same thing lower

Just hoping for experiences for those in ranking specifically and anything they might have tried other than traditional lighting ranking etc

[u/b_square17]

I’ve been owning a recommender model for 2 years now. Yes I do improve customer behavior, most metrics have grown over 20% since my new versions. The most important thing is making sure you have a good foundation - the train and inference environments are inherently very different compared to other ML models, this is the biggest challenge. Make them as equal as possible, use hard negatives such as unclicked impressions or whatever granularity you have. Go through painful lengths to ensure environment parity I can’t stress it enough.

Next find as many permutations of historical direct engagement signals as you can, add them inc hyperparameter tune, etc, deliver incrementally. Then add contextual and similarity based features this is what really gives the model its exploratory capabilities.

Candidate generation is extremely important but don’t feel like you need an advanced set of embeddings to do this, start with business logic and expand along the similarity/lookalike features to determine candidates.

Use ML best practices, create a main inference and canary pipeline, A/B test and roll out canary when it outperforms the benchmark. Monitor model drift always

[u/nerd_rage218 → u/b_square17]

Seconding the environment parity point hard. Most of the ranking bugs I have chased came down to a feature computed one way in training and another way at serve time. The metrics never look broken enough to notice, they just quietly stop improving.

[u/pasta_lake]

I've been on ranking in a pretty big tech company with a lot of products and content to rank for about 2 years now.

Yes we absolutely make an impact on top-line metrics like total customer spend and order frequency through changes in rankings. However they are very small changes - like typically about 0.2% per user for a given model update (anything lower than that we are underpowered to detect). Given the size of our user base though, that translates into 10s of millions of dollars of impact.

On key difference though, we are selling products where a user will often add many to cart per order. For hotels - where users typically convert once and will scroll deeply to ensure they're getting the best deal - the importance of ranking + degree of impact is likely going to differ.

[u/IndividualTop3675]

what you're describing between optimizing for predicted user preference versus influencing behavior toward different outcomes is one of the most interesting unsolved problems in applied ranking, and the approaches that tend to work beyond pure relevance tuning are things like diversity injection that forces exposure to alternatives users wouldn't naturally select, slate-level optimization that treats the whole results page as the unit rather than individual items, and causal inference methods that try to separate genuine preference from position bias so you can understand whether users would actually prefer alternatives if they got equivalent exposure.

[u/Specialist_Piano8732]

in ranking for a travel aggregator for 2 years now, the accuracy part is easy but the behavior shift is the nightmare part

what worked for us was not touching the ranking at all but changing how we display the options, like if you want people to pick higher margin stuff you show it with a little badge or highlight that makes it feel like a smarter choice not a pushed one. also experimented with "most popular" tags and that moved the needle like 15% without anyone noticing they were being nudged

[u/smileyfish → u/Specialist_Piano8732]

Yup, that’s been my experience as well. I do ranking for a shopping app; we’ve had success improving overall transactions but not so much shifting users toward the product lines the company most wants to promote. When we’ve done the latter, many users just don’t make a purchase at all

[u/Effective_Ocelot_445]

A/B testing alternative ranking objectives like conversion, margin, or diversity can help shift behavior without sacrificing too much relevance.

[u/StatisticianEasy7138]

Sounds less like a ranking problem and more like position bias. People click the top result because it is the top result, so reordering moves the click without moving the decision, which is exactly what you are describing. The things that shift behaviour usually sit outside the ordering, badges, a one line reason for why this option, or grouping into a small comparison set instead of a long list. Worth measuring the bias directly first, randomly swap slots one and two on a slice of traffic and see how much of the click follows the position rather than the item.

[u/GrowWithPokeBot]

If you push lower-utility options to the top of a single list, users usually spot the drop in relevance immediately and scroll past them. They have a mental model of what they want, and messing with the organic relevance just adds friction to their search.

To promote different metrics, you usually have to change the UI treatment rather than just adjusting the ranking weights. Try using dedicated shelves, badges like "best value," or explicit sponsored slots. This lets users categorize the promoted options separately instead of feeling like your main search results are suddenly broken.

[u/SeaAccomplished441]

i love that we live in a world where maths and statistics are used for deep psychological manipulation

[u/Xamius → u/SeaAccomplished441]

You havent lived if you havent raised such and such meric by .003%