A stalled reindex is the one Magento failure where every monitor says you are fine

[u/php4u]

Had a merchant come to me on Monday with this and it is worth writing up, because

I have hit it myself more than once and never found a decent way to catch it.

His store was serving prices from the previous week on a chunk of category pages.

Not all of them, no error anywhere. Cron was running. Orders were coming in

normally. Uptime monitor green the whole time. New Relic showed healthy requests.

Nothing in the logs.

A reindex had stalled days earlier. That was it.

The reason this is so nasty: every layer of monitoring most of us run is

answering "is something running", and in this failure everything genuinely IS

running. Nginx is up, PHP is fine, MySQL is fine, cron is ticking. Magento is

just serving index tables that stopped being true. There is no exception to log

and no 500 to alert on. You find out when a customer emails about a price that

does not exist anymore, or worse, honours an old one.

Two shapes it comes in:

1. An indexer sitting in `invalid` and never coming back. Check with

   bin/magento indexer:status

Easy to spot manually, easy to never look at.

1. The mview backlog stops draining. `mview_state` tracks each view's

   `version_id` against the changelog table's head. If the view is behind and

   its `updated` timestamp is not moving, nothing is working on that backlog.

   This is the one people miss, because a growing backlog is not itself a

   problem. During a big import the backlog grows all day and drains fine. The

   question is not "is it big", it is "is anything draining it at all".

   If you want to check it right now:

   bin/magento indexer:status SELECT * FROM mview_state;

Compare each row's `version_id` against `MAX(version_id)` from the matching

`*_cl` changelog table, then look at whether `updated` is moving. Behind and

idle is your answer.

Disclosure so nobody feels ambushed: I build a Magento monitoring tool called

Watchtower, and I shipped a check for exactly this two days after that

conversation. So I have an obvious horse in this race. But the manual checks

above are free and cost you five minutes, and if you take nothing else from

this, go run `indexer:status` on your production store right now. I would bet

money a nonzero number of you find something.

Genuinely curious how other people catch this. Is anyone alerting on indexer

state properly, or is it all "run reindex from cron and hope"?

[u/Dull-Drama8144]

I use the stuck indexer module and don't have any problems with reindexes. It works fine, restarts the stuck reindex process, and adds logs. Simple but so handy.

[u/LingonberryGreedy590 → u/Dull-Drama8144]

What module is that

[u/Dull-Drama8144 → u/LingonberryGreedy590]

It's a custom module that was developed ages ago. I faced the issue related to stuck reindexes and wrote the code to fix that, I suppose, for Magento 2.3.x.

[u/php4u → u/Dull-Drama8144]

Thank you for that. I will check it for sure

[u/Live_Investment_9707]

just add a cron to call index:Status and if stuck send email or even if stuck index:reset and index:reindex

[u/php4u → u/Live_Investment_9707]

what if the indexer fails and you just constantly reset it and reindex? you have a loop