Reconsider the excluded dir anywhere in the hierarchy patches

[rwstauner]

#345 excluded files with a "t" directory in their hierarchy from getting the "indexed" attribute

attempting to solve #340.
Then #367 built upon that to add "inc" and "local" to solve metacpan/metacpan-web#1411 and metacpan/metacpan-web#1239
However that logic would exclude legitimate modules like local::lib.
We can only exclude that directory when it's in the dist root.
For anything else we should be checking the CPAN::Meta object that is already built (with default dirs built into it).

<haarg> i'd rather have too much stuff indexed from the perl tarball than doing the wrong thing everywhere else
<oalders> rose is already working on special casing perl indexing
<rwstauner> yeah
<oalders> she could add this to her list
<haarg> also, pause indexes stuff like ext/Foo/t/blah.pm
<haarg> so i'm not really sure there's a problem to solve
<rwstauner> it's all Getty's fault
<oalders> Getty pointed it out because we link modules in source code
<oalders> and he got bounced over to that module somehow
<rwstauner> :)
<haarg> pretty sure the only reason pause didn't index that MyModule.pm file is because it doesn't have a version number
<haarg> and MyModule is already indexed with version 1 in another dist
<haarg> if those modules shouldn't be indexed, they should be added to the META.json file perl has, which already includes a bunch of exclusions
<haarg> i'm not sure why metacpan picked the MyModule out of perl, that part is a bug
<rwstauner> oh yeah, i forgot, perl does have a huge no_index structure
<oalders> we should add some of this conversation to that ticket so we don't have the same conversation in 8 weeks

[haarg]

I think we should only be excluding t, inc, and local from the top level. This matches what PAUSE does.

[rwstauner]

I agree.
I liked the way you put it in irc:

i'd rather have too much stuff indexed from the perl tarball than doing the wrong thing everywhere else

[rwstauner]

4f034ea

[rwstauner]

I added a test to confirm that the next time you'd release local::lib we wouldn't index it,
so I removed the other code to fix it. The two issues about inc/ and local/ are still fixed (by 4f034ea).
I'll reopen the MyModule one.