Avoid repeated copying in clean_tokens.

[nnethercote]

clean_tokens takes a vector of tokens and removes some. The decision
of which tokens to remove is somewhat complicated, so it can't use
retain or filter, but instead does everyting itself. This includes
calling remove on every individual removed token. Unfortunately, this
results in quadratic-ish behaviour when the number of tokens removed is
large, due to all the shuffling down of tokens after each removed token.
The closer the removed tokens are to the start of the vector, the worse
it is.

This commit switches to a two pass approach. The first pass records
which tokens should be removed, using an auxiliary vector of bools. The
second pass does the removal, using Vec::retain. When running
rustdoc (which uses minifier) on helloworld, this reduces memcpy
traffic from this:

297,675,803 bytes in 869,856 blocks, avg size 342.21 bytes

to this:

179,253,407 bytes in 868,510 blocks, avg size 206.39 bytes

[GuillaumeGomez]

Very interesting approach! Thanks!

[nnethercote]

Thank you for the fast response. Would you be able to make a new release so we can take advantage of the speedup in rustdoc?

[GuillaumeGomez]

Already on it: #100. ;)