Revert fisker's memory regression

[devinrhode2]

I am happy with the git work, but still need to do some manual sanity checking, and probably run test suite

@fisker you are totally welcome to change anything here. For example, "use xo to lint xo was lost as collateral damage, but we can restore that. Likewise I think the "ignores" fix you had in the same commit could also be restored.

Here, I was just trying to do the cleanest possible revert.

This could be considered like a v0.45.1 but since it's also reverting fisker's fixes, it could also be considered like a v0.43.1. It's a mutant. Probably 0.43.1 would make the most sense. (If review goes well, and sindre wants to publish it, of course)

Closes #599 but also re-opens #583

[devinrhode2]

@fisker in theory, you (or anyone!) could take another stab at your original fixes for #583 like so:

1. Get these commits locally
2. Squash all 5 together
3. Revert the fully-squashed commit
   3.1 Checkout fresh branch off of main
4. (?) Cherry-pick that revert commit
   Then finally re-revert that revert, undo, and edit away
   (I'm sure you could figure this out of course, just sharing to maybe save you time messing with git)

[devinrhode2]

taking the really brain-dead approach and also reverting the "remove fs-extra" commit: #619