Current Technologies & Roadmap

[flaser]

This thread is for discussing and setting long term technical goals the project can strive for.
While the term technology is a bit vague, I couldn't come up with a better name to refer to both frameworks, libraries and applications that are employed in CR.

---

Current project technologies:

• Application framework:
  • .NET Framework 4.8 --> .NET (AKA .NET Core)
• GUI framework:
  • Tao.OpenGl --> OpenTK
  • Windows Forms
• Utility Libraries
  • Nuget:
    • IronPython 2.7.4
      (Note: version, 3.4.1 is for Python 3 scripts, CR Plugins use Python 2)
    • Microsoft-WindowsAPICodePack-Shell
    • MySQLConnector
    • Sharpcompress
    • SharpZipLib
  • Unmanaged (ComicRack\Resources)
    • 7z (7-zip)
    • libdjvulibre
    • libjpeg
    • libtiff
    • libwep
    • libz
  • Unmanaged (ComicRackEngine\DLL)
    • sharpPDF

Some of these are not ideal, as they're either single platform or considered obsolete.

---

Proposed technologies:

• GUI framework:
  • These may not happen due to the quantity of work needed
  • XAML: Avalonia
  • XAML/Blazor: MAUI
• Persistence
  • SQLite
• Diagnostics
  • ILogger

---

Evaluation Criteria

If you want to add a feasible choice, consider these:

• MUST: Is there a viable way to introduce or transition without a complete rewrite?
• MUST: Is the library/framework mature? Is there sufficient documentation?
• MUST: Will it support the existing feature set?
• SHOULD: Is there a way to gradually introduce it?
• SHOULD: Is it available on Nuget?

[maforget]

• OpenTK is already being done, but I no nothing about OpenGL. Check HW Accel #23, there is already a fork with some work done.
  • I would better use an already done framework and be done with it, maybe something like ImageMagisk? What is important is the speed, just open an super high resolution like 7000 x 10000 and it's a slugfest, haven't pinpointed the reason yet, maybe memory allocation is very low.
• UI stuff like Avalonia/MAUI will probably never happen. The UI as so many layer inside each other, it's really impressive what they were able to do with just winforms. But this would be a complete rewrite of all the UI elements and since they work usually by a different model MVVM, so probably a rewrite of the backend. Also I hate WPF style stuff, haven't check Avalonia really, but from what I heard it's just WPF. I never remember what is it I need to use, that DataTemplate or ItemTemplateContainer.
• .NET is basically a given, I think most of what was taken it back is SoapHttpClientProtocol, which is already removed, and probably the BinaryFormatter. Like said in this thread Steps to modernize the code #10, I think the binaryFormatter is used for storing the thumbnails. This seems like a big problem is it is, if we can't keep compatibility. Something else that would prevent adoption of it, is if it breaks plugin, that would be a line then and we would be staying on 4.8 to preserve the existent plugins.

[flaser]

Like said in this thread #10, I think the binaryFormatter is used for storing the thumbnails. This seems like a big problem is it is, if we can't keep compatibility.

If it's only the thumbnail cache, I think at some point people have to take the hit and regenerate theirs.

Does it also include stuff like the cover fore file-less comics?
If so, I see your point, but then I think what we need is a means to migrate the data to another format that CR-CE could invoke if it finds an old cache.

Something else that would prevent adoption of it, is if it breaks plugin, that would be a line then and we would be staying on 4.8 to preserve the existent plugins.

I did a tiny bit of digging, seems like Iron Python did add .NET 6 support:
https://github.com/IronLanguages/ironpython2/releases/tag/ipy-2.7.12

[maforget]

Yes I believe it includes fileless comics, haven't check just saw something about thumbnails really. We could always include a converter app or something similar.

I know that IronPython supports .NET 6, so in theory should work with 8, but some plugin might be using .NET code also. Maybe they wouldn't work anymore. We just need to test it really.

[Totengeist]

I was also thinking about creating a converter. We could give users an option in the installer and/or have ComicRack detect old thumbnails on first run.

The cache files seem pretty simple. They're serialized CacheItem objects (cYo.Common\Collections\Cache.cs), which contain:

• lock information
• last accessed timestamp
• data (the valid JPG image, in this case)
• a linked list, the purpose of which I don't understand yet

There might be a few ways we could detect a BinaryFormatter serialized cache item, use the file modification date as "last accessed" and just load it whole as the cache data (IrfanView recognizes them as JPGs, so another library might also). Then, once the expiration has been reached, clear the file and re-cache it with the new serializer.

[maforget]

I checked on old project of mine that had a clone extension. And it used the binary formatter also.

[Totengeist]

After further testing, XmlSerializer and JsonSerializer require all classes to be serialized to have parameterless constructors. While it will create cache.idx fine, it will not read an existing one and recreates the entire cache. I've tried to follow this all the way down, but I still get errors about "no parameterless constructor." I'll continue looking into it.

[maforget]

I found this post on stackoverflow, with a Clone utility that uses MemberwiseClone.

Link to Github.

Probably the easiest, because you won't have any choice than to use Reflection for that.

[maforget]

I've tried to update the Clone function with a nuget packages called DeepCloner. It worked in the Debug config, but once a Release was done, ti crashed when trying Copy Data. So I've had to revert it.

[maforget]

I've posted an update that starts removing BinaryFormatter. To update the clone utility, I've used DataContractSerializer, it seems to work well. The CloneUtility isn't used in a lot of place. Mainly Copy/Paste Data and when Exporting it clones the ExportSettings.

For the cache, I was able to save it in XML, but it was annoying, since it wasn't just parameterless constructors, but requiring to add the key type to the serializer as extra types. Which in turn needed to also fetch the derived type, because the declared generic type K was only an ImageKey, but used the derived type. Then types needed to be set as public and add setters. Some I also needed to remove the readonly modifier. So it would probably have been easier to use @Totengeist branch.

I've kept the original cache.idx and added a new file cache.idx.xml, so it will use the new file if it exists but also keep saving the old file. So in case of errors just deleting the cache.idx.xml will load the original file. I wasn't able to test the Files folder, because it seems to be only used by Web Comics, which I don't have. So more testing that the index file is correctly created is required.

I did try to use the experimental NRBF Decoder published by Microsoft to read existing payloads saved with BinaryFormatter. But all I got was an error message saying that "Record Type is not supported by design". My though is that because the cache is a List (List<CacheItem>) instead of a single Type, the package can't read it. It seems to understand only primitive types. Also there is almost no documentation or samples and haven't even found code on Github despite 130K download on nuget.

At some point reading the existing cache files will not work anymore, in the meantime this will migrate users. Also it's not like the cache is absolutely necessary. Custom Thumbnails are also saved directly in the DB, so even removing it completely should only affect users for a short while when the cache is being rebuilt.

[Salvora]

Just putting what is on my mind here. I did check the code but found it hard to read I guess due to de-compiling. I will need more time to follow up some utilities and functions. I would really like to see an official roadmap on the wiki page after concluding the discussions here.

Code Quality
1- Adding comments for readability.
2- A separate stable channel with an installer (maybe with an auto-update feature)
3- A separate beta channel with an installer (maybe with an auto-update feature)
4- An alpha channel where the commits are pushed frequently.

Basic Functions & Modernization:
1- Moving to OpenTK
2- Moving to .NET (This will open up more venues like headless or Linux installations but the first step is moving to .NET)
3- Moving to IronPython 3.4 (This will break all the plugins and scripts since it is using Python 3)
4- Better sql structure for queries to speed up the sql queries. (I have very superficial knowledge for sql data structures but I am sure this can be further optimized)
5- Start-up time (I find that for big libraries, the CR starts up very slowly due to loading all the data and querying the database)
6- Modernized UI (I know this is a long shot but does not hurt to put it here)
7- Better multi-core and memory utilization for speed (This is very vague, any input to speed up the project is welcome)
8- Using newer faster common libraries like libjpeg-turbo or libpng for speed etc.

Bugs & Problems:
1- Windows Filepath length limit. (This should have been resolved after moving to .NET Framework v4.8 but did not have time to confirm.)
2- Filepath encoding (There is a problem when deleting or manipulating some files with special Unicode characters. You need to change the name of the file then delete or manipulate)
3- I find thumbnail caching not working very well. The covers are getting scanned almost every time I open a list despite the fact that I set a huge number in the caching settings.

New Features
1- Unified settings (Moving comicrack.ini to UI to unify all the settings)
2- Unified database ==> Currently CR creates multiple database files if you are using SQL (comicdb.xml and SQL). These can be unified to create a single database file.
3- Moving to JSON from XML for ComicInfo.xml. Just an idea to ditch the XML due to complexity and readability and use JSON for the ComicInfo. There are some speeds to be gained especially for large files.
4- Animated gif, png etc support for the reader.
5- A built-in function to check the integrity of the cbz/cbr/7z files.

[maforget]

Some of these will never happen:

Moving to IronPython 3.4 (This will break all the plugins and scripts since it is using Python 3)

• I see no point in breaking all the plugins, especially since they will probably never be updated, since long abandoned, but still very useful. These are core to why ComicRack is still useful, without those I see no point. This for me is a breaking change, anything that prevents existing plugins from working would be a no-no. I do have plans for a better plugin system New Plugin System #3.

Moving to JSON from XML for ComicInfo.xml. Just an idea to ditch the XML due to complexity and readability and use JSON for the ComicInfo. There are some speeds to be gained especially for large files.

• The ComicInfo.xml is already a standard in almost all comic program, it would bring nothing to break that standard, for a little readability. And I doubt performance would be better to change 1 serializer for another.

Not certain

A separate stable channel with an installer (maybe with an auto-update feature)
A separate beta channel with an installer (maybe with an auto-update feature)
An alpha channel where the commits are pushed frequently.

• This seems all a bit much, just a release and nightly seems enough for me.

Unified settings (Moving comicrack.ini to UI to unify all the settings)

• I see the comicrack.ini file as extra settings (It's called ExtendedSettings in code) and although I see your point, I believe it would overwhelm users with too much options, that are mostly not to be touched. Some could be brought to the settings like changing thumbnails Caption (it could be hooked with the new Virtual Tags feature) or maybe some SQL settings.

Unified database ==> Currently CR creates multiple database files if you are using SQL (comicdb.xml and SQL). These can be unified to create a single database file.

• The way I see it, the XML database is a backup in case a problem with the SQL arises.

Tempering expectation

Moving to .NET (This will open up more venues like headless or Linux installations but the first step is moving to .NET)

• I am pretty sure that we could change it right now, but I am in no rush to change it. New language features and better library support is nice. But with all the Windows API code that this program exists a Linux version is probably a far away dream, if just that a dream. Headless server install #22 For headless server, there are still a lot of unknown about usage/setting and I don't see why this would need the new .NET and can't be done with .NET Framework. Either we move to it now or wait for it. I like the fact that .NET Framework is already provided with Windows and doesn't need to extra download or bundling of extra files with the program.

Moving to OpenTK

• Already starting doing some changes, but we would need to do some extra research on OpenGL/OpenTK. So for now it's pretty much on the back burner unless it would radically speed up the program.

Better sql structure for queries to speed up the sql queries. (I have very superficial knowledge for sql data structures but I am sure this can be further optimized)

• The problem I see with better SQL queries is that as far as the rest of the program is concern this is just another Implementation of an interface providing the data. So it should be invisible to the program, the library itself should not know what is the source of the data. Although it shouldn't be a problem with Entity Framework building it, it's the rest of the program I ain't certain of how it would work.
• I Haven't checked the code for smart lists really I don't know how this would translate to SQL queries. Like someone mentioned smart lists are basically SQL. queries, but I don't know if the existing system would work correctly thorough it.
• All that would require to wait for .NET, if we are to use Entity Framework, I would just do it with the Core version, no need to do it now to have to redo it later for the new version.
• To be fair, I don't use SQL and so frankly don't really care in that case. That would need someone else working on that part.

Start-up time (I find that for big libraries, the CR starts up very slowly due to loading all the data and querying the database)

• With the current database in XML, the whole database is pretty much loaded into memory. I don't even know if an Async version exists of the deserializer.

Modernized UI (I know this is a long shot but does not hurt to put it here)

• It seems like a nice idea, and it is. But like you said this is a long shot. Going to something like Avalonia, would require a pretty big overall. Needing to split the backend code with the UI. With the complexity of the UI, it is a big task.

Nice ideas:

Animated gif, png etc support for the reader.

• I think the problem with animated gif may be the viewer and not the reading of the file itself.

A built-in function to check the integrity of the cbz/cbr/7z files.

• Checking image integrity would be nice.

Better multi-core and memory utilization for speed (This is very vague, any input to speed up the project is welcome)

• I agree that there is some sluggishness that could be better, the program doesn't use any Async/Await. This can be problematic because it starts ballooning async/await everywhere. It does have a robust Queue/Thread Manager that I am apprehensive to start touching. Any hints of what that might help are welcome.

Using newer faster common libraries like libjpeg-turbo or libpng for speed etc.

• Sure if there are any library that can be updated just point it out.

My OWN priority

• Right now there is an annoying hard crash related to resizing thumbnails. I've tracked the original source of the code and redid the code a couple of times because I figured it was related to decompiling. I've decompiled these lines with 3 programs and each one gives me a different line. I don't believe it was happening before in the Original version. I benchmarked this with a multiple of library for resizing, and this way is multitude faster than most. It does use unsafe code and pointers, so the error is hard to track. I've also decorated the method so that error is caught, and it seems to work but once the error is happening it does it repeatedly and hard crash the program (even if it should be caught). Not really certain why.

ComicRackCE/cYo.Common/Drawing/ImageProcessing.cs

Lines 1019 to 1024 in 4856eec

	for (int i = 0; i < minPixelSize; i++, dst++, p1++, p2++, p3++, p4++)
	{
	byte b = (byte)((double)dx2 * (double)*p1 + (double)dx1 * (double)*p2);
	byte b2 = (byte)((double)dx2 * (double)*p3 + (double)dx1 * (double)*p4);
	*dst = (byte)((double)dy2 * (double)b + (double)dy1 * (double)b2);
	}

Check this commits:
678dad6 1a25c1b efb7789 ea97437

• Find the reason why some very high resolution files seem to stall the program. This is probably related to OpenGL and the low video memory allocation. I would need to research all this OpenGL because my knowledge is really zero.
• I really want to find a nice way to implement Dark Mode. There is somekind of Windows API that might be in the works that would use the system theme. (Again more Windows API).
• Someone mentioned on reddit unlimited stacks and talked about cYo about it and he said he had programmed himself in a corner with that. I haven't checked if would be possible­.
• Tweak any memory numbers/hardware limitations to better use new hardware.

Conclusion

There are a lot of discussion, but what would really help is some PR for some of these. I am just a hobbist without really any formal programming education. I rebooted this because I didn't want it to die, and already did my own hacks of the program. But a lot of what some people wants are pretty overwhelming in scope. It seems that a lot of ideas are thrown, but not a lot of how it would be done, does it benefit the user base? Is it too much for some user? Would they even use it, or is it just 2 people?

For now I suggest we concentrate to small compartmentalized changes, like updating libraries, porting new ones, fixing bugs, adding some small features. Also I really don't want this project to become a full time job, and I do have other past time I would like to have time for, so more surgical changes (at least for my part). Changes that benefit most user first, something stupid like a dark mode would have a big impact.

[iguyking]

As a long time fan and paying customer of CR, thanks for getting this going again. I appreciate every thing you've done so far.

Therefore, whatever you are willing to do, that's all that will be done. If folks want a SQL system or update to Avalonia, they can make an PR. If they don't, you owe them nothing. What you've done so far is 100x better than it used to be.

My vote is you work on whatever bothers you, makes you excited, makes your life better and that's it.

Thanks again and if we ever meet I'll buy you some food and a drink.

[maforget]

I've updated the projects files to support the newer SDK style, along with changes to support the newer language functions. This is probably the limit of what we can do. I don't know if it will be possible to move to .NET 8+.

While trying to remove the need for BinaryFormatter, i've updated the Clone Utility to use a nuget package called DeepCloner, that seems to work but crashed the release build. So I've had to revert the changes for now.

The remote library is using a function called Windows Communication Foundation that is deprecated, that will need to be either scraped or redone using more modern alternative (that I've not looked what they are).

Edit: It seems that the replacement for WCF is CoreWCF.

[Totengeist]

While trying to remove the need for BinaryFormatter, i've updated the Clone Utility to use a nuget package called DeepCloner, that seems to work but crashed the release build. So I've had to revert the changes for now.

Sadly, I haven't had time to return to this. I had to give up because none of the solutions I tried worked properly. Either they would crash, require a large rewrite, or would just reset the database. I hope DeepCloner will end up working in the end.

I'm getting a bit more comfortable with C# on other projects, so I'm hoping I can be of more help soon!