GS: Merge assembly files in GS

[TellowKrinkle]

Replaces 6 copies of the same file ([x86, x64] * [SSE4, AVX, AVX2]) with one slightly more complicated one. I think it'll be easier to maintain this way, but I'm curious as to what other people think

Pros:

• Only have to edit one file when changing the SW renderer
• x64 now supports everything x86 does

Cons:

• Naming the registers with variables can make it less obvious that some registers are the same. (e.g. in a bunch of spots x86 uses one register where x64 uses two, which requires two variables to be able to be different in x64, but makes it less obvious that they're actually the same in x86. An alternative would be to use more split code instead of trying to keep everything on one code path, though I personally would rather not if possible)
• Lots of macros used to generate boilerplate in xbyak_smart.h as well as to abstract over common differences between our use of assembly in x86 vs x64. I personally don't mind, but I know macros can make things extra hard to understand so please at least look and decide if you think it's OK

Outstanding Issues:

• While SSE and AVX's comments of C code were all the same, AVX2 is based off of different C code and therefore a lot of the comments are different. A lot of the code with comments from two different C sources, however, is now the same piece of asm for both, so I'm not sure how we want to deal with that. At the moment I haven't transferred over the AVX2 comments at all.
• I did a terrible job of naming things, especially when they were referring to PS2 hardware things, as I still don't know much about how that actually works (yay for blindly transcribing things). In particular, _rip_local_d is supposed to refer to either m_local.d8 or m_local.d4 (depending on AVX register size), but after realizing m_local also has a d field, that's clearly not a great name. I don't, however, have any clue what would be as I don't know why it's called d in the first place.
• Windows x64 is still broken, having issues with the display of a bunch of the bios (e.g. the "Sony Computer Entertainment" comes up as a bunch of white squares). I tried a bunch of things to make win64 and macOS 64 closer (e.g. switching all the ifdefs and adding movs to transfer the inputs to the other platform's calling convention) but I both couldn't get Windows x64 to work or macOS x64 to break, so I'm thinking the issue might be somewhere else in GSdx (Windows x64 just crashes in the current version of GSdx so it's never been tested). If anyone has any ideas as to where this issue might be please let me know Fixed

Also, since this is a full rewrite of the SW renderer's assembly routines, lots of testing is needed

[TellowKrinkle]

Wait are we building GSdx in C++03 mode? That's the only way to make this error... https://gcc.godbolt.org/z/c8oj9W

[TellowKrinkle]

Never mind this is the issue https://gcc.godbolt.org/z/a4rvYz

[rspretty]

Works almost perfect with my win8.1 x64 machine! Great work!
Tested both x32 and x64 cmake build.
I've found minor graphics gritch in SOTC(NTSC-J). Color of water surface gets pruple.
Sorry for this post becomes just a report, ASM part of the code is far more difficult than I can read.

[TellowKrinkle]

GSdump please

[rspretty]

Thank you.
This is a dump of above scene.
https://sidenet.ddo.jp/pcsx2dumps/mergeasm-sotc-purple-texture.gs.gz
Same scene dump running on master branch.
https://sidenet.ddo.jp/pcsx2dumps/master-sotc-same-scene.gs.gz

[TellowKrinkle]

@rspretty please check and confirm that the issue is fixed

[rspretty]

Confirmed!
Works like a charm. Awesome!!

[Ziemas]

Yup, this is working fine for me as well now.

[JordanTheToaster]

Checked through my game collection on 64 bit software AVX2 Windows and all seems good to me no issues present on any of the games I played.

[Ziemas]

I found out this doesn't build with GCC 11.1.0.

/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp: In lambda function:
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:1774:105: error: ‘i’ is not a constant expression
 1774 |                                 return ptr[_g_const + offsetof(GSScanlineConstantData, m_log2_coef_128b[i])];
      |                                                                                                         ^
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:1774:64: warning: ‘offsetof’ within non-standard-layout type ‘GSScanlineConstantData’ is conditionally-supported [-Winvalid-offsetof]
 1774 |                                 return ptr[_g_const + offsetof(GSScanlineConstantData, m_log2_coef_128b[i])];
      |                                                                ^
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:1776:105: error: ‘i’ is not a constant expression
 1776 |                                 return ptr[_g_const + offsetof(GSScanlineConstantData, m_log2_coef_256b[i])];
      |                                                                                                         ^
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:1776:64: warning: ‘offsetof’ within non-standard-layout type ‘GSScanlineConstantData’ is conditionally-supported [-Winvalid-offsetof]
 1776 |                                 return ptr[_g_const + offsetof(GSScanlineConstantData, m_log2_coef_256b[i])];
      |                                                                ^
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp: In member function ‘void GSDrawScanlineCodeGenerator2::ReadTexelImplLoadTexLOD(int, int)’:
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:3366:62: error: ‘lod’ is not a constant expression
 3366 |         Address lod_addr = m_sel.lcm ? _rip_global(lod.i.u32[lod]) : _rip_local(temp.lod.i.u32[lod]);
      |                                                              ^~~
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:3366:40: note: in expansion of macro ‘_rip_global’
 3366 |         Address lod_addr = m_sel.lcm ? _rip_global(lod.i.u32[lod]) : _rip_local(temp.lod.i.u32[lod]);
      |                                        ^~~~~~~~~~~
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:3366:96: error: ‘lod’ is not a constant expression
 3366 |         Address lod_addr = m_sel.lcm ? _rip_global(lod.i.u32[lod]) : _rip_local(temp.lod.i.u32[lod]);
      |                                                                                                ^~~
/home/ziemas/Development/pcsx2/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.all.cpp:3366:70: note: in expansion of macro ‘_rip_local’
 3366 |         Address lod_addr = m_sel.lcm ? _rip_global(lod.i.u32[lod]) : _rip_local(temp.lod.i.u32[lod]);

[RedPanda4552]

Tested on Dragon Quest 8, Gran Turismo 4, Ratchet and Clank, NCAA 08, and Star Wars Battlefront 2. No issues, working on 64 bit.

[gtgamer468]

I tested Gran Turismo 3, Gran Turismo 4 Prologue, Midnight Club 3, Tekken 5, Sonic Heroes and NASCAR '09 and all of them are working as intended in x64 SW mode.

[refractionpcsx2]

Tested about 10 games of various graphical effects + BIOS, all looks good :)

[JordanTheToaster]

Tested 12 games including bass master fishing, looks good to me.

[TellowKrinkle]

I found out this doesn't build with GCC 11.1.0.

Try now

[Ziemas]

I found out this doesn't build with GCC 11.1.0.

Try now

Yup, works now.

[lightningterror]

Copyright headers are outdated compared to the new ones we use in GS.

[lightningterror]

Copyright headers are still inconsistent, use these instead
https://github.com/PCSX2/pcsx2/blob/master/pcsx2/GS/Renderers/SW/GSDrawScanlineCodeGenerator.cpp#L1

[lightningterror]

I see some new warnings in gcc, maybe would be nice to take care of them.
https://github.com/PCSX2/pcsx2/runs/4100022508?check_suite_focus=true#step:9:566

[TellowKrinkle]

I see some new warnings in gcc, maybe would be nice to take care of them

Fixed, but test and look for new memory leaks

Hopefully no one was relying on GSAlignedClass having a virtual destructor, I made the new destructor protected to catch the obvious issue (someone trying to delete a GSAlignedClass<32>*), but it won't catch a subclass relying that declaration to implicitly make its own destructor virtual (example). I checked the obvious two (GSRenderer and GSDevice) but there could be others. So far it seems like no one else was aware that GSAlignedClass had a virtual destructor, as everyone has been virtual-ing the destructors of their subclasses...

[lightningterror]

Time for the testers to shine now.

[JordanTheToaster]

Tested on 13 games and showed no issues in any of them. Memory behavior was the same as master and I have found no leaks in my time testing it.