Description
To be able to support any kind data update, we need to emulate or achieve the transaction concept, to avoid queries on dirty or outdated data. Also, we need to support cache invalidation and index regeneration on updates.
I was thinking in two approaches to do it:
Approach 1:
A copy of the data is managed by gitbase. A listener on the original folder checks if some file changed, generating gitbase events per repository. Using these events we can invalidate partition cache or regenerate partition indexes if needed.
The data will not be accessible until all the indexes are regenerated and all the caches are invalidated (transactions). All the queries executed on index generation will use the old index until the new one is ready.
Indexes could be regenerated in two ways:
- Regenerate the whole partition.
- Regenerate only affected rows.
This will depend on how precise are our events (just this file has been added or deleted, or check which objects changed depending on the updated or created packfile).
Pros:
- Easier to implement that Approach 2
- Less intrusive than Approach 2
- We can just sync a specific refspec and ignore the rest (only x,y and z references, per example).
Cons:
- Disk space usage
- How to sync status when gitbase is powered off and then on?
Approach 2:
Gitbase is also a git server. Repository data is handled by gitbase. On each git push to the server, gitbase generates several events to update indexes or invalidate caches. The rest is working exactly as in Approach 1.
Pros:
- Less disk space usage than Approach 1
- If gitbase is off, data cannot change
Cons:
- Intrusive, you need to use gitbase as a git server
- we need to handle and keep all the data, no refspec filtering possible