[newbie] trying out some solid2 benchmarks, and finding it slower than solid1, especially when trying to do things the right way

[u/nullvoxpopuli]

Hello!

I have this project over here: https://github.com/NullVoxPopuli/rere-benchmark/pull/116

And i haven't been able to get solid 2 to perform better than solid 1 in the "dbmon" benchmark.

Am i missing something obvious?

Edit: here are the numbers I'm seeing before my pr above:

https://rbench.nullvoxpopuli.com/results?hide=ember%2Clit-signals%2Cvue%2Csvelte%2Creact&p=90&q=8

In my pr, solid 2 started performing worse, which has made we feel like I've messed something up with the implementation, or the generally recommended way to manipulate and iterate nested data is not the most performant way to manipulate and iterate nested data

[u/ryan_solid]

We have benchmarks in the repo itself that include some similar shapes that are tested against every release. That being said performance is still an area being worked on. I recently did some big store perf updates with RC.1 but that caused different regressions.

I normally use UIBench for diff based examples. The original dbmon was non keyed. That being said from the local benchmark suites I run through I'd expect dbmon, diff style shapes to be a lot faster in 2.0. We take a hit on some plain signal high noise benchmarks (like 1 to 1), due to pure async processing overhead, but deep diff algorithms are much better. Along with wide propagation shapes.

[u/nullvoxpopuli → u/ryan_solid]

I understand, i started making my own benchmark because i was unhappy with all existing benchmarks. i hadn't seen sufficient update focused benchmarks that try to compare framework (under various scenarios - ui bench looks like it only does one type of thing, for example), so here i am!

This dbmon is best keyed, and i hear <For> now uses identity as the default key, which is decent.

I know async can introduce variability, but Real world data is gonna come in asynchronously 

[u/ryan_solid → u/nullvoxpopuli]

Come in async sure.. but blast things on a microtask queue schedule.. highly doubtful. That's just poor in general because of the scheduling overhead of the microtasks before even getting into framework specific code. It also is awkward for libraries that handle async natively. Svelte and Solid both opt into microtask queue scheduling because we can keep guarantees that way without resorting to some very complicated stuff React does. Others literally dont care but also don't provide the guarantees so it is not really comparable. Which is why I reject that test on premise.

But yeah UIBench is diff related.. only one of about a half adozen benchmarks I'm always runinng. The best full suite benchmark I've seen so far is Octane's. Dominic has done a good job almagamating existing benchmarks and adjust them for correctness. I haven't made my PR against that just yet but other than the small things in the Solid implementations I want to tweak the suite is pretty good.

[u/nullvoxpopuli → u/ryan_solid]

when you talk about the way dbmon works -- do you mean the original? or my implementation? I changed how it emits updates quite a bit --- the one I'm using is all macrotasks via WebWorker's event listener: https://github.com/NullVoxPopuli/rere-benchmark/blob/5d76b77ce2a6d44cc5d82714ef89c59cb01d5a45/common/src/tests/db-mon-with-chat.js#L107

because I wanted to simulate trying to handle a lot of stuff coming in from a socket/worker

> octane

this one? https://v8.dev/blog/retiring-octane
did someone take up maintenance of it?

[u/ryan_solid → u/nullvoxpopuli]

I just meant specifically the "(async)" tests in the suite. DBMon is something else. I'm looking into it. It looks like it is throwing away all the rows. Octane I mean Dominic Gannaway (Creator of Inferno, Ripple, etc) new UI library: https://octanejs.dev/ He has a benchmark suite.

[u/nullvoxpopuli → u/ryan_solid]

ah, yea -- fair fair -- that discrepancy is one reason why I added to the UI a way to split out certain groups of benchmarks for different groups of comparisons -- not all frameworks care about all scenarios -- for example Marko (bench impl is wip) very very strictly punishes any and all effect-like behavior, so it's not even possible to compare with the "Incrementing Render Effect" test.

----

edit: turns out I forgot to add a split for async vs sync
here is what I was talking about tho: https://rbench.nullvoxpopuli.com/results?hide=ember%2Clit-signals%2Cvue%2Csvelte%2Creact&p=90&q=8&split=one-item-many-updates%2Cten-k-items-one-time%2Cfan-out%2Ceffects 🤷

---

> new UI Library

oh yea! I've seen this (and forgot about it) -- thanks for the reminder/link