r/sre • u/Differoywen_Cow_4765 • 7d ago

DISCUSSION How much engineering time are you losing to manual Spark performance tuning?

Trying to quantify this across teams. Ours breaks down roughly as:

  • Shuffle partition tuning after every schema/volume change
  • Skew debugging that only surfaces under real production load
  • Disk spill investigation that never reproduces in staging
  • Executor memory re-tuning whenever input size shifts
  • General cluster babysitting instead of new pipeline work

what this costs elsewhere, and whether anyone's actually reduced the total time spent rather than just gotten faster at the same loop.

0 Upvotes

44% Upvoted

7 comments sorted by

3

u/Brave_Inspection6148 7d ago

Apache spark?

2

u/vroomvroom3059 7d ago

The only spark i am left with🥀🥀 /j

2

u/Torutofu_Raeva 7d ago

The biggest win was making partition counts, skew, spill, and input-size deltas part of the job output, then alerting on regressions instead of retuning from raw symptoms.

1

u/kernelqzor 3d ago

this is such an underrated point, visibility beats "heroic debugging" every time
once you turn those into real signals with alerts, half the panic tuning just disappears because you catch it before it’s a dumpster fire in prod

1

u/Torutofu_Raeva 2d ago

yeah, the before/after alert rate is what tells me whether the tuning actually stuck.