Allow `#each` block to handle arbitrary non array like iterables

[joeally]

Describe the problem

Consider the following REPL:

<script>
	const myMap = new Map();
	myMap.set('key1', 'value1');
	myMap.set('key2', 'value2');
	myMap.set('key3', 'value3');
	for(const [k, v] of myMap.entries()){
		// Javascript lets us iterate through a set so I don't see why svelte shouldn't
		console.log(k, v);
	}
</script>
<!-- This is not allowed because mySet does not have `.length` -->
{#each myMap.entries() as [k, v]}
	<h1>Key: {k}, value: {v}</h1>
{/each}

Obviously one could trivially achieve this by wrapping myMap in Array.from or spreading into an array but it would be nice if svelte could handle iterables in addition to ArrayLike objects.

Describe the proposed solution

The problem is caused by the fact that svelte generates the code which loops over the each value using its .length property. This can be seen below:

// from top of create_fragment
let each_1_anchor;
let each_value = /*myMap*/ ctx[0].entries;
let each_blocks = [];

for (let i = 0; i < each_value.length; i += 1) {
	each_blocks[i] = create_each_block(get_each_context(ctx, each_value, i));
}

and

// from the p function returned by create_fragment
for (i = 0; i < each_value.length; i += 1) {
	const child_ctx = get_each_context(ctx, each_value, i);
	if (each_blocks[i]) {
		each_blocks[i].p(child_ctx, dirty);
	} else {
		each_blocks[i] = create_each_block(child_ctx);
		each_blocks[i].c();
		each_blocks[i].m(each_1_anchor.parentNode, each_1_anchor);	
	}
}

As you can see it assumes that each_value (in this case myMap) has a length property. It seems to me that we could easily generate the following code which is equivalent:

// the top of create_fragment
let each_1_anchor;
let each_value = /*myMap*/ ctx[0].entries();
let each_blocks = [];
for (const item of each_value) {
	each_blocks.append(create_each_block(get_each_context(ctx, item)));
}

// the loop in the p function generated by create_fragment
let each_value_i = 0;
for (const item of each_value) {
	const child_ctx = get_each_context(ctx, item);
	if (each_blocks[each_value_i]) {
		each_blocks[each_value_i].p(child_ctx, dirty);
	} else {
		each_blocks[each_value_i] = create_each_block(child_ctx);
		each_blocks[each_value_i].c();
		each_blocks[i].m(each_1_anchor.parentNode, each_1_anchor);	
	}
}

With get_each_context rewritten to be:

function get_each_context(ctx, item) {
	const child_ctx = ctx.slice();
	child_ctx[1] = item;
	return child_ctx;
}

It's worth noting that the for...of syntax isn't supported by IE11 but the svelte compiler appears to recommend the use of the spread operator which also isn't supported in IE11 so perhaps this isn't an issue.

Alternatives considered

The alternative here is to leave it as it is and tell people to use Array.from and or [...mySet]. It isn't the end of the world.

But given that it seems so trivial to support arbitrary iterables and that javascript can loop over iterables I can't see why svelte wouldn't implement this. Is it because you want any infinite loops to be in user code rather than generated code?

Importance

nice to have

[Pika-Pool]

Rich Harris has recently talked about supporting this too: https://youtu.be/dB_YjuAMH3o?t=4m18s

[stordahl]

Would love to see this added to the roadmap for v4

[brandonmcconnell]

@joeally Just confirming— once implemented, this wouldn't even require using .entries(), right?

I'd think this could be as simple as—

<script>
  const myMap = new Map();
  myMap.set('key1', 'value1');
  myMap.set('key2', 'value2');
  myMap.set('key3', 'value3');
</script>

{#each myMap as [k, v]}
  <h1>Key: {k}, value: {v}</h1>
{/each}

[stordahl]

@brandonmcconnell @joeally What Brandon has provided is the API I would love to see. IMO Svelte should obfuscate the implementation details so the template reads the same whether you're iterating over a map or an array.

[dummdidumm]

What will likely happen is that the each block calls Array.from under the hood

[Conduitry]

Yeah, we need to buffer the whole iterable into a single array anyway to implement the keyed-each minimal edit stuff, so we might as well just use Array.from() on whatever's send to the {#each}. As a bonus, this works with { length: x } object as well, so we don't need special handling to avoid breaking that idiom we've promoted in the past.

[brandonmcconnell]

Doesn't that defeat the performance benefit of Set and Map, though, when dealing with massive data sets?

In native JS, we can iterate over sets and maps using for..of and retain those perf benefits without needing to port their contents to an array. It would be ideal to keep those benefits.

const myMap = new Map();
myMap.set('key1', 'value1');
myMap.set('key2', 'value2');
myMap.set('key3', 'value3');

I ran some benchmarks just to demonstrate this difference, and Map iteration is faster across all (by over 50%):

	JSBench.me results	Perflink results	JSBEN.CH results
for..of(Map)	fastest ✅	fastest ✅	fastest ✅
for..of(Array.from(Map))	62.03% slower 🔻	~60% slower 🔻	62.06% slower 🔻
for..of(Array.from(Map.entries()))	63.42% slower 🔻🔻	~62% slower 🔻🔻	63.63% slower 🔻🔻
	link to test	link to test	link to test

---

The fact alone that taking advantage of the perf benefits is so attainable and not something other frameworks have done yet is a huge opportunity for Svelte, and I think we'd be falling short to not take a key stake in big (and attainable) performance wins like this.

I think for..of loops require an iterable anyway, so by swapping to a for..of loop here, you'd naturally be supporting both Maps and Sets and other iterable data structures.

[Conduitry]

The DOM is slow, and I am almost certain that losing the ability to do minimal DOM updates would far outweigh what we'd gain by only iterating through the array once with for of. Using for of internally would also mean that we could no longer directly support iterating over the array-like object { length: 42 }, which is something we've encouraged people to do in the past when all they really care about is the number of times something appears in the DOM. We would probably want to have something like 'length' in foo ? foo : Array.from(foo) so we don't lose performance for things that already are arrays or other array-likes.

[brandonmcconnell]

I agree that a solution like 'length' in foo ? foo : Array.from(foo) would be ideal, though I think we could still preserve the performance benefits of Maps and Sets with minimal effort.

What "ability to do minimal DOM updates" would be lost by switching to use for..of loops?

If we use a lightweight abstraction like the iterateOver function below, we can keep all the benefits you mentioned and also support all iterables (e.g. maps, sets, generators, arrays, etc.) and array-likes.

Implementation

— see on TS Playground

function isIterable<T>(obj: Iterable<T> | ArrayLike<T>): obj is Iterable<T> {
  return typeof (obj as Iterable<T>)[Symbol.iterator] === 'function';
}

function iterateOver<T>(obj: Iterable<T> | ArrayLike<T>, expose: Function) {
  if (isIterable(obj) && !Array.isArray(obj)) {
    /* non-array iterables (array iteration using a traditional `for` loop is more performant) */
    let i = 0;
    for (const elem of obj) {
      expose(elem, i++);
    }
  } else {
    /* arrays and array-likes */
    const indexedObj: ArrayLike<T> = (
      'length' in obj && typeof obj.length === 'number'
        ? (obj as { length: number })
        : Array.from(obj)
    );
    for (let i = 0; i < indexedObj.length ?? 0; i += 1) {
      expose(indexedObj[i], i);
    }
  }
}

In ☝🏼 this example, I am using an imaginary "expose" function, but that would just be replaced by whatever logic or helper function you use to slot in the HTML for each {#each} block iteration.

Notes on why I used the !Array.isArray(obj) check to push array iterables to the else block

The !Array.isArray(obj) check I added is optional, though I did some additional performance testing and found that arrays—specifically—perform better using traditional for loops rather than for..of loops.

However, that is not the case for other iterables (e.g. maps, sets, generators, etc.), which perform significantly better when iterating using a for..of loop. So with that in mind, I am using the !Array.isArray(obj) check to push arrays into the second block, which iterates over them using a traditional for loop instead.

Tests

— see all tests on TS Playground (same link as above)

Here are some tests for that function, each of which works as expected:

• Array-like example { length: 3 }

  const arrayLikeExample = { length: 3 };
  
  iterateOver(arrayLikeExample, (_: undefined, i: number) => console.log(_, i));
    // -> undefined 0
    // -> undefined 1
    // -> undefined 2

• Array example ['value1', 'value2', 'value3']

  const arrayExample = ['value1', 'value2', 'value3'];
  
  iterateOver(arrayExample, (val: string, i: number) => console.log(val, i));
    // -> 'value1' 0
    // -> 'value2' 1
    // -> 'value3' 2

• Set example {'value1', 'value2', 'value3'}

  const setExample = new Set();
  setExample.add('value1');
  setExample.add('value2');
  setExample.add('value3');
  
  iterateOver(setExample, (val: string, i: number) => console.log(val, i));
    // -> 'value1' 0
    // -> 'value2' 1
    // -> 'value3' 2

• Map example {'key1' => 'value1', 'key2' => 'value2', 'key3' => 'value3'}

  const mapExample = new Map();
  mapExample.set('key1', 'value1');
  mapExample.set('key2', 'value2');
  mapExample.set('key3', 'value3');
  
  iterateOver(mapExample, ([key, val]: [string, string], i: number) => console.log(key, val, i));
    // -> 'key1' 'value1' 0
    // -> 'key2' 'value2' 1
    // -> 'key3' 'value3' 2

• Generator example 'value1' => 'value2' => 'value3'

  function* generatorExample(): IterableIterator<string> {
    let current = 1;
    while (current <= 3) {
      yield `value${current}`;
      current++;
    }
  }
  
  iterateOver(generatorExample(), (val: string, i: number) => console.log(val, i));
    // -> 'value1' 0
    // -> 'value2' 1
    // -> 'value3' 2

[brandonmcconnell]

And while it arguably reads easier with the isIterable type guard outside iterateOver, you could move that same logic inside iterateOver with relative ease, which I'm guessing may be easier for Svelte since IIRC Svelte doesn't use TS under the hood but JSDoc style typing:

function iterateOver<T>(obj: Iterable<T> | ArrayLike<T>, expose: Function) {
  if (typeof (obj as Iterable<T>)[Symbol.iterator] === 'function' && !Array.isArray(obj)) {
    /* non-array iterables (array iteration using a traditional `for` loop is more performant) */
    let i = 0;
    for (const elem of (obj as Iterable<T>)) {
      expose(elem, i++);
    }
  } else {
    /* arrays and array-likes */
    const indexedObj: ArrayLike<T> = (
      'length' in obj && typeof obj.length === 'number'
        ? (obj as { length: number })
        : Array.from(obj)
    );
    for (let i = 0; i < indexedObj.length ?? 0; i += 1) {
      expose(indexedObj[i], i);
    }
  }
}

— see on TS Playground

Here's a JSDoc version 👇🏼

/**
 * Iterates over an iterable object or array-like object and exposes each element to a callback function.
 * @param {Iterable<T> | ArrayLike<T>} obj - The iterable or array-like object to iterate over.
 * @param {Function} expose - The callback function to call for each element.
 * @returns {void}
 * @template T
 */
function iterateOver(obj, expose) {
  if (typeof obj[Symbol.iterator] === 'function' && !Array.isArray(obj)) {
    /** @type {Iterable<T>} */
    const iterableObj = obj;
    /* non-array iterables (array iteration using a traditional `for` loop is more performant) */
    let i = 0;
    for (const elem of iterableObj) {
      expose(elem, i++);
    }
  } else {
    /** @type {ArrayLike<T>} */
    const indexedObj = (
      'length' in obj && typeof obj.length === 'number'
        ? obj
        : Array.from(obj)
    );
    /* arrays and array-likes */
    for (let i = 0; i < (indexedObj.length ?? 0); i += 1) {
      expose(indexedObj[i], i);
    }
  }
}

[joeally]

@brandonmcconnell - I'd prefer the user to have to call .entries(). IMO the {#each} should work with any iterable (see: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols).

{#each} should not have a special case for Map and Set.

Yes you're right. I didn't know that Map itself can be iterated over.

[brandonmcconnell]

@joeally Exactly. The Map and Set are not special cases. Every iterable can be iterated over using the same for..of loop style. The only exception is the "array-like" case, which is easy enough to support performantly, and one additional but optional exception is the plain array type.

Array types are iterable in the same way that Map, Set, and generators are, but they actually iterate more performantly if iterated over using a plain for loop, so I use a !Array.isArray(obj) check to push array types into the second block alongside the array-like type.

We can loop over someMap.entries() and someSet.values() alternatively, but generally speaking, it's actually more performant to just loop over maps and sets directly without using entries() and values(), which is also simpler syntax, so win-win. 🎉