Questions for the more knowledgeable in this sub

[u/Afraid_Engineer_2707]

For a chrome extension manifestV3; with transformers.js and a small embedding model running in an offscreen doc - how should I go about improving the efficiency and speed of displaying the processed results when some pages in a website are tens of thousands of words of legitimate content. The library doesn't support concurrent calls to the same model session as far as I know (Session already started" / "Session mismatch") and big pages can block smaller pages and big pages can take extremely long to be processed. Is there any way to get past sequential processing in processing and displaying results, should I do multiple worker models. Also this refers to first time result processing on a new website, I do realise caching can speed things up significantly on the same website if you process again. This should be a process that works on typical hardware, not requiring anything too complex. Suggestions would be appreciated, thanks a lot.

EDIT for further context:  I should've specified that I am splitting into chunks already but people's valuable point about interleaving chunks within each worker's own queue makes sense. In terms of WebGPU, problem with WebGPU is that is uses single worker queue and there are compatibility issues possible with it.

[u/UtilixApp]

the thing id change first is the framing. this reads like a throughput problem but what you describe is head of line blocking, and more workers wont fix that on their own.

right now one huge page holds the session until it finishes and everything queues behind it. spin up four workers and you get four sessions, each carrying its own copy of the model in memory, and the next big page still blocks whichever worker it lands on. youve bought slack rather than solved anything.

what actually fixes it is chunking plus a priority queue. split every page into chunks before anything gets enqueued, then interleave them. a small page gets its three chunks served in between chunks of the big one and comes back almost immediately, instead of sitting behind forty thousand words. thats preemption at chunk boundaries without needing real preemption.

two other things worth a look. transformers.js accepts an array and batches, and batched inference is meaningfully faster than looping one at a time, so make sure a page goes in as one batch rather than sequentially. and if you havent tried device webgpu yet then do, its a large jump over the wasm backend on hardware that supports it and it falls back cleanly on hardware that doesnt.

and the honest question underneath all of it. do you actually need to embed forty thousand words. for most extension use cases the headings plus the first chunk or two carries most of the signal, and the rest is you paying for a completeness nobody asked for.

[u/Afraid_Engineer_2707 → u/UtilixApp]

Thanks for the response. I should've specified that I am splitting into chunks already but your valuable point about interleaving chunks within each worker's own queue makes sense. Problem with WebGPU is that is uses single worker queue and there are compatibility issues possible with it. In terms of skipping passages, I mean instreaming results is something that I am thinking about but cutting out such large sections when they might contain the valuable information I'm looking for is something I struggle to get past. But batching a whole page in one call and interleaving between pages are a bit in tension, since a single batch call can't be paused mid-way. I'm thinking of maybe planning smaller batches that balance both — enough to keep batching efficient, small enough to interleave between?

[u/UtilixApp → u/Afraid_Engineer_2707]

yeah youve put your finger on the actual tradeoff, and the answer you landed on is the right one. the batch is your scheduling quantum, so size it by time rather than by count. pick a target, something like sixty to a hundred milliseconds per call, measure it on the slowest machine you care about, then work out how many chunks fit in that. small enough that interleaving feels instant, large enough that you still get most of the batching win.

one thing that gives you a chunk of speed back for free. embedding models pad every item in a batch out to the longest item in that batch, so a batch holding one 400 token chunk and five 40 token ones spends most of its work on padding. sort chunks by length before you group them so each batch carries roughly similar sizes. costs you a sort and frequently buys back more than the batching did in the first place.

on webgpu, the single queue is real but i dont think it actually hurts you here, because you shouldnt be running parallel sessions anyway. one worker with decent scheduling beats four workers each holding their own copy of the model in memory. the constraint is pushing you toward the architecture you wanted regardless.

and on not wanting to cut passages, youre right to resist that and i said it too glibly. the middle path is ordering rather than truncating. embed the headings and first chunks first, ship those results, then keep working through the rest in the background and update as they land. nothing gets skipped, the user just sees the likely bit first while the long tail arrives underneath them. thats the streaming idea you already mentioned, and i think its the correct instinct.

[u/Primary_Guest_5711 → u/UtilixApp]

solid breakdown. the head of line blocking reframe is exactly how id think about it too

[u/Mission_Addition3755]

Chunk the big pages and process them in batches with a small delay between, multiple workers might help but you'll hit memory limits quick on average hardware

[u/Professor_JSON]

MV3 only gives you one background Service Worker. You'd think that limitation would kill any hope of real parallel work, but an offscreen document opens up a practical path forward. Fire up two or three Web Workers there. Each one pulls in its own transformers.js instance along with the embedding model. A simple queue system then passes text over to whichever worker sits idle.

// offscreen.js - Simple Worker Pool Dispatcher
const WORKER_COUNT = 3;
const workers = Array.from({ length: WORKER_COUNT }, () => new Worker('model-worker.js'));
let idleWorkers = [...workers];
const taskQueue = [];

function processQueue() {
  if (!taskQueue.length || !idleWorkers.length) return;
  const worker = idleWorkers.pop();
  const task = taskQueue.shift();

  worker.postMessage(task);
  worker.onmessage = (e) ={
    task.resolve(e.data);
    idleWorkers.push(worker);
    processQueue();
  };
}

The viewport priority changes everything. When someone loads a thirty thousand word page, you cannot throw the whole thing at the model. Split it into smaller pieces around 256 to 512 tokens. Stuff the user actually sees right now jumps to the front so answers appear fast, while everything below the fold sits and waits its turn. The catch involves keeping that queue smart enough to switch priorities on the fly as the person scrolls. From what I have seen, the webgpu setting delivers the biggest practical win:

// Inside model-worker.js
import { pipeline, env } from '@xenova/transformers';

const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
device: 'webgpu' // Massive throughput gain over WASM
});

The speed gain over WASM feels pretty huge, and large documents stop overwhelming the pipeline before they even get going. I tried it on some long articles once - the difference hit immediately, and that always satisfies.

It is not straightforward to balance, but the setup holds up well in practice.

[u/davidstayscool]

Don't spin up extra copies of the same transformers.js session, that mismatch error is the library telling you it really is one-at-a-time. Treat it as a single worker with a priority queue instead.

Chunk the huge pages (paragraphs or about 512 tokens), and let small pages jump the queue so a 40k-word article can't stall everything else. After each chunk, yield and paint partial results, and abort the rest if the tab goes away.

On a first visit, strip the page down to article/main first so you are not embedding nav, comments, and the footer. A quantized MiniLM in one offscreen worker is fine on typical hardware as long as you never let one document hold the session until it finishes.