Spark, or a single node,
in Microsoft Fabric?
Stop paying for Spark you don't need. In Fabric, a notebook can run two ways — on a Spark cluster, or on a single node in pure Python (DuckDB or Polars). Spark's the obvious default, so it's easy to reach for a cluster without realising the single-node path is often faster and far cheaper on Capacity Units. Answer five quick questions and we'll point you at the right one — the minnow (single node) or the trawler (Spark).
The difference: a Spark session spins up a driver plus executors — a startup tax and more Capacity Units. A single-node notebook is one machine: leaner and quicker to start, but bounded by that one node.
It's a starting-point heuristic, not a benchmark. It points you the right way — always benchmark your real workload before you bet the farm.
Answer five questions
Everything's computed in your browser — no data is sent anywhere. Change any answer and recompute as often as you like.
No black box
These are the directional heuristics behind the recommendation — rules of thumb, not laws. Your real workload can always surprise you, so treat the result as a starting point and benchmark before committing real Capacity Units.
- Under ~10 GB → single node, almost always. A cluster is pure overhead at this size, whatever the shape of the work.
- ~10–100 GB → single node usually wins. DuckDB spills to disk out of the box (Polars only via its lazy/streaming engine); it depends on the query.
- ~100 GB – 1 TB → it depends. You're near a single node's practical ceiling — this is exactly where you should benchmark.
- >~1 TB, massive shuffles, or genuinely distributed ML → Spark. Spreading work across executors is what distribution is for.
- Short, frequent or interactive jobs → single node. No cluster-startup tax — a few seconds on a starter pool, minutes on a cold custom pool, every run.
- Heavy existing PySpark → a migration-cost call, not a technical one. It only ever tips a genuine coin-flip; it never flips a real single-node win.
- Streaming → a different question entirely. In Fabric that's Real-Time Intelligence or Spark Structured Streaming — not a batch notebook.
When Spark genuinely wins — and when it's overkill
When is a single-node Python notebook the better choice in Microsoft Fabric?
For working sets up to roughly 100 GB, SQL-shaped transforms (filters, joins, group-bys, aggregations), and short or frequent jobs. A single-node engine like DuckDB or Polars avoids Spark's cluster-startup tax and executor overhead, so it's often faster and uses far fewer Capacity Units. It's a heuristic — benchmark your real workload.
When does Spark genuinely earn its keep in Microsoft Fabric?
When the working set is beyond roughly 1 TB, when the job does massive full-dataset shuffles or sorts, or for genuinely distributed ML training where the data or model won't fit on one node. That's what distribution across executors is built for.
Is a single node always cheaper than Spark in Fabric?
No. Capacity Units are roughly vCores multiplied by time. On genuinely big or shuffle-heavy jobs, Spark's parallelism can finish far faster, and a shorter run at a higher CU-per-hour can cost fewer CU overall — which is exactly why this advisor sends those workloads to Spark. Fabric doesn't expose per-query CU cleanly, so treat cost guidance as directional.
What about streaming or continuous ingestion in Fabric?
Streaming isn't really a single-node-versus-Spark-notebook decision. In Microsoft Fabric that's usually Real-Time Intelligence (Eventstream into Eventhouse/KQL), or Spark Structured Streaming — not a batch Python notebook. This advisor covers batch and interactive compute.
Want evidence, not a rule of thumb?
This advisor is a heuristic. The next tool in the series — an importable diagnostic Fabric notebook — will profile your actual data and query and give you an evidence-based verdict. It'll default to the cheap path (profile, single-node trial, EXPLAIN) so it won't burn Capacity Units just to answer the question, with a full Spark-vs-single-node A/B only if you opt in.
Tell us what you'd want from it →Want your Fabric platform to right-size itself?
We build the data platforms and pipelines that pick the right engine for each job — and keep your Capacity Unit bill honest. That's literally what we do, for builders and makers who'd rather ship than tune clusters.
Talk to us →