Skip to content

Fix/1.6.0 #96

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
PierreRambaud merged 13 commits into from
May 6, 2025
Merged

Fix/1.6.0 #96

PierreRambaud merged 13 commits into from
May 6, 2025

Conversation

@Sudrien
Copy link
Collaborator

@Sudrien Sudrien commented Apr 17, 2025

Other projects are to the point I can finally address the issues I've been seeing

Hex code for sha256 of /versions is not expected, base64 is.
-- it's the correct checksum in the wrong format
-- I can't use base64 as part of the filename, so I'll convert when serving

Easy fix. Committed.

gemirro index --update is not showing full file lists, only recent changes

I might have to reorder calls to do this efficiently.

@Sudrien
Copy link
Collaborator Author

Sudrien commented Apr 21, 2025

Gem install from rubygems.org

  • get /versions
  • get /info/package
  • get /quick/Marshal.4.8/package-version-platform.gemspec.rz
  • get info/package dependancies
  • get gem
  • get latest_specs.4.8.gz
  • get rubygems-update

Bundle install from ruby gems.org

  • get /versions
  • get /info/package list

SO.

We only need
/versions, /info and /latest_specs.4.8.gz for syncing. We probably should maintain rubygems-update

The proper order for —update

  1. Pull latest_specs.4.8.gz => public/
  2. Pull newest ruby gems-update gem if not present => public/gems
  3. Pull source (rubygems.org) /versions and /info/package => public/rubygems.org/versions etc

[this is the index —update variant]
4. If there are any gemspecs without gems, remove gem specs. => public/quick … Add their names to the list.
5. If there are any gems without gemspecs, create gemspecs. => public/quick … Add their names to the list.
6. Create or delete info/package according to list

[this is the non-update variant]
4. Delete all gemspecs
5. Generate a gemspec per gem
6. Create info/package for all gem specs

  1. Create public/local/versions and public/local/names according to gemspecs (not gems)
    8a. If set to download missing (aka mirror), merge versions files into public
    8b. If set NOT to download, public/local/versions copied to public/versions

…ruby gems gemspec.rz mirrored? “all requests will check if gems are detected, and download them if necessary” leads to potential significant slowdown. Sidekiq or cron should be options for these checks.

This job is essentially running update and index —update at a regular interval. As such, It would also catch any new user-uploaded gem versions.

PierreRambaud
PierreRambaud reviewed Apr 27, 2025
View reviewed changes
lib/gemirro/cli/init.rb Outdated
['specs.4.8', 'latest_specs.4.8', 'prerelease_specs.4.8'].each do |s|
File.utime(Time.at(0), Time.at(0), File.join(directory, 'public', s))
end
# ['specs.4.8', 'latest_specs.4.8', 'prerelease_specs.4.8'].each do |s|
Copy link
Owner

@PierreRambaud PierreRambaud Apr 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Collaborator Author

@Sudrien Sudrien Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, individual gemspec.rz are used, but all the marshaled stuff is depreciated / used for fallback

@PierreRambaud
Copy link
Owner

PierreRambaud commented Apr 27, 2025

Gem install from rubygems.org

  • get /versions
  • get /info/package
  • get /quick/Marshal.4.8/package-version-platform.gemspec.rz
  • get info/package dependancies
  • get gem
  • get latest_specs.4.8.gz
  • get rubygems-update

Bundle install from ruby gems.org

  • get /versions
  • get /info/package list

SO.

We only need /versions, /info and /latest_specs.4.8.gz for syncing. We probably should maintain rubygems-update

The proper order for —update

  1. Pull latest_specs.4.8.gz => public/
  2. Pull newest ruby gems-update gem if not present => public/gems
  3. Pull source (rubygems.org) /versions and /info/package => public/rubygems.org/versions etc

[this is the index —update variant] 4. If there are any gemspecs without gems, remove gem specs. => public/quick … Add their names to the list. 5. If there are any gems without gemspecs, create gemspecs. => public/quick … Add their names to the list. 6. Create or delete info/package according to list

[this is the non-update variant] 4. Delete all gemspecs 5. Generate a gemspec per gem 6. Create info/package for all gem specs

  1. Create public/local/versions and public/local/names according to gemspecs (not gems)
    8a. If set to download missing (aka mirror), merge versions files into public
    8b. If set NOT to download, public/local/versions copied to public/versions

…ruby gems gemspec.rz mirrored? “all requests will check if gems are detected, and download them if necessary” leads to potential significant slowdown. Sidekiq or cron should be options for these checks.

This job is essentially running update and index —update at a regular interval. As such, It would also catch any new user-uploaded gem versions.

If you're able to fix / update tests, I think the behavior is cool (and also maybe the best one for your needs ^^)

@Sudrien
Copy link
Collaborator Author

Sudrien commented Apr 29, 2025

If you're able to fix / update tests, I think the behavior is cool (and also maybe the best one for your needs ^^)

@PierreRambaud I started live tests today (some very large self-hosted binary gems, gigabytes worth), will continue to fix rspec tests as time allows

@PierreRambaud
Copy link
Owner

PierreRambaud commented Apr 29, 2025

Ping me when you'll need some reviews, I'll be happy to help

@Sudrien
Copy link
Collaborator Author

Sudrien commented May 6, 2025

@PierreRambaud Ok, Ready for your review.

@PierreRambaud PierreRambaud merged commit 65fac32 into master May 6, 2025
8 checks passed
@PierreRambaud
Copy link
Owner

PierreRambaud commented May 6, 2025

Thank you 🙏

@PierreRambaud PierreRambaud deleted the fix/1.6.0 branch May 6, 2025 20:18
@PierreRambaud
Copy link
Owner

PierreRambaud commented May 6, 2025

I'm gonna publish a new release tomorrow if it's ok for you :)

@PierreRambaud
Copy link
Owner

PierreRambaud commented May 10, 2025

Unfortunately, I wasn't able to do it in time, I'll do it next week 🙏

@Sudrien
Copy link
Collaborator Author

Sudrien commented May 11, 2025

No objections, whenever you're ready

@PierreRambaud
Copy link
Owner

PierreRambaud commented May 13, 2025

No objections, whenever you're ready

2.0.0 is released 🎉 Thanks for all the work you did!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PierreRambaud PierreRambaud PierreRambaud approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Sudrien @PierreRambaud