Discuss: database replication server #2442
Description
Due to the current issues with the parquet export, a user on Discord came up with an idea for sharing the Sourcify database in a different way:
Was there a system figured out for updating a remote database without downloading the whole parquet dump again? [...]
Or could a log replay be implemented? Everytime a parquet dump is done start a new log of database actions? then keep a manifest of them and replay any since you grabbed the dump?
or an asynchronous replication server - that wouldn't slow down the main server at all, and then we could run further asynchronous databases off that.
We've spoken about this a while ago and there is the edge case of a partially verified contract becomming fully verified - I noticed there is a cleanup script for it in the admin functions folder now.
THere are a few types of replications servers - the type I suggested is 1-way - it just copies the master. Anyone could then maintain a copy that is always up to date. I suggested an additional one - so that the main server only needs the exra workload of sending commands to a secondary. Then some scheme could be set up where people could subscribe to the secondary and let it take any additional load. I'm not sure you can just subscribe though - it would probably need people to request access from the team and have a team member manually add it.
we could just leave the partia and full in thee and instruct people to upsert.
i do like the idea of an always uptodate local server through - not one up to 24 hours old.