Error when attempting to use fork

[joshbuker]

Gemfile:

# frozen_string_literal: true

source 'https://rubygems.org'

git_source(:github) { |repo| "https://github.com/#{repo}.git" }

gem 'argon2', github: 'athix/ruby-argon2', branch: 'feature/additional-helpers'

When running bundle / bundle install I get the following error:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/[REDACTED]/.rvm/gems/ruby-3.0.0@sorcery/bundler/gems/ruby-argon2-acbbf925b8f0/ext/argon2_wrap
/home/[REDACTED]/.rvm/rubies/ruby-3.0.0/bin/ruby -I /home/[REDACTED]/.rvm/rubies/ruby-3.0.0/lib/ruby/3.0.0 -r
./siteconf20210326-54663-cq3tmp.rb extconf.rb

current directory: /home/[REDACTED]/.rvm/gems/ruby-3.0.0@sorcery/bundler/gems/ruby-argon2-acbbf925b8f0/ext/argon2_wrap
make "DESTDIR=" clean
rm -f tests libargon2_wrap.so

current directory: /home/[REDACTED]/.rvm/gems/ruby-3.0.0@sorcery/bundler/gems/ruby-argon2-acbbf925b8f0/ext/argon2_wrap
make "DESTDIR="
make: *** No rule to make target '../phc-winner-argon2/src/argon2.c', needed by 'libs'.  Stop.

make failed, exit code 2

Gem files will remain installed in /home/[REDACTED]/.rvm/gems/ruby-3.0.0@sorcery/bundler/gems/ruby-argon2-acbbf925b8f0 for
inspection.
Results logged to
/home/[REDACTED]/.rvm/gems/ruby-3.0.0@sorcery/bundler/gems/extensions/x86_64-linux/3.0.0/ruby-argon2-acbbf925b8f0/gem_make.out

An error occurred while installing argon2 (2.0.3), and Bundler cannot continue.

In Gemfile:
  argon2

@technion thoughts?

[joshbuker]

Looks like using my local copy might be a viable workaround for this: bundle config local.argon2 <local_repo_path>

Edit: It seems to have installed correctly and be running, but anytime it loads the code it spits out this error:

find: ‘ext’: No such file or directory

[technion]

I have a feeling something about using a fork like this means you haven't initialized and built the submodule.

https://github.com/technion/ruby-argon2#building-locallytests

Your error output looks a lot like when I would work locally and forget to do that.

[joshbuker]

Hmm. When I initialized the git submodule by following your documentation, it didn't cause any changes that I could push to my fork upstream as far as I could tell. Did you have to do anything from the Github side of things?

[technion]

I only recently re-did the CI when Travis stopped working and I can't think of anything special that isn't documented. But perhaps there's a build requirement I'm not aware of. Looking here:

make: *** No rule to make target '../phc-winner-argon2/src/argon2.c', needed by 'libs'. Stop.

Does that file exist on your system? If you jump in ruby-argon2/ext/argon2_wrap, can you run "make test" ? There's a whole C test harness here.

[joshbuker]

Does that file exist on your system?

find / -name "argon2.c"

/home/[REDACTED]/.rvm/gems/ruby-3.0.0/gems/argon2-2.0.3/ext/phc-winner-argon2/src/argon2.c

Looks like it does exist, but maybe it can't find it because it's in the root of rvm instead of the gemset that it's building for?

If you jump in ruby-argon2/ext/argon2_wrap, can you run "make test" ? There's a whole C test harness here.

I had to install clang for the test suite to work, but it all passed the first time I ran it.

I tried to run it again to grab the test suite output to put on this comment, and interestingly enough I did run into this error:

make test

make: *** No rule to make target '../phc-winner-argon2/src/argon2.c', needed by 'test'.  Stop.

But running the checkout again process seems to have resolved that:

make test

clang -pthread -O3 -fsanitize=address -fsanitize=undefined -Wall -g ../phc-winner-argon2/src/argon2.c ../phc-winner-argon2/src/core.c ../phc-winner-argon2/src/blake2/blake2b.c ../phc-winner-argon2/src/thread.c ../phc-winner-argon2/src/encoding.c argon_wrap.c ../phc-winner-argon2/src/ref.c test.c -o tests -pthread -O3 -Wall -g -I../phc-winner-argon2/include -I../phc-winner-argon2/src
./tests
Ref test: 1c7eeef9e0e969b3024722fc864a1ca9f6ca20da73f9bf3f1731881beae2039e: PASS
Ref test: 5c6dfd2712110cf88f1426059b01d87f8210d5368da0e7ee68586e9d4af4954b: PASS
Ref test: dfebf9d4eadd6859f4cc6a9bb20043fd9da7e1e36bdacdbb05ca569f463269f8: PASS
Ref test: fabd1ddbd86a101d326ac2abe79660202b10192925d2fd2483085df94df0c91a: PASS
Ref test: b3b4cb3d6e2c1cb1e7bffdb966ab3ceafae701d6b7789c3f1e6c6b22d82d99d5: PASS
Ref test: b2db9d7c0d1288951aec4b6e1cd3835ea29a7da2ac13e6f48554a26b127146f9: PASS
Ref test: bb6686865f2c1093f70f543c9535f807d5b42d5dc6d71f14a4a7a291913e05e0: PASS
Ref test: $argon2id$v=19$m=65536,t=2,p=1$c29tZXNhbHQ$CTFhFdXPJO1aFaMaO6Mm5c8y7cJHAph8ArZWb2GRPPc: PASS
Ref test: $argon2id$v=19$m=256,t=2,p=1$c29tZXNhbHQ$nf65EOgLrQMR/uIPnA4rEsF5h7TKyQwu9U1bMCHGi/4: PASS
Ref test: $argon2id$v=19$m=65536,t=2,p=1$c29tZXNhbHQ$C4TWUs9rDEvq7w3+J4umqA32aWKB1+DSiRuBfYxFj94: PASS
Verify OK test i variant: PASS
Verify OK test id variant: PASS
Verify FAIL test: PASS

Edit: It also added a new copy of argon2.c to the ext folder in my ruby-argon2 project:

find / -name "argon2.c"

/home/[REDACTED]/Repos/personal/ruby-argon2/ext/phc-winner-argon2/src/argon2.c
/home/[REDACTED]/.rvm/gems/ruby-3.0.0/gems/argon2-2.0.3/ext/phc-winner-argon2/src/argon2.c

Still fails when I try to gem update with my github branch when not using the local copy though, and I don't have any options to push something upstream still (even after rerunning the git init stuff).

Guessing you have a hunch as to what the issue is? I haven't really dealt with git submodules/ruby c bindings before, so I'm unfamiliar with their gotchas.

[technion]

I've walked through a new setup on a new machine and put the output here:

https://gist.github.com/technion/b4f76cd9e3cb20981962b964d7e6c480

I'm having a sinking feeling that the fact I see version 3.0 in your output is the issue. None of my builds have Ruby 3.0 installed. I'll start some testing now with this.

[joshbuker]

@technion link to the failing build if that helps any: https://github.com/Sorcery/sorcery-rework/pull/5/checks?check_run_id=2211559532#step:4:150

I'll also try updating the rework to include the version matrix introduced to the main Sorcery repo, and see if the older versions of ruby pass.

[joshbuker]

@technion fails for ruby versions 2.5.8, 2.6.6, 2.7.2, and 3.0.0, doesn't seem to be due to Ruby 3.0 changes: https://github.com/Sorcery/sorcery-rework/runs/2214111105

Edit: Whatever the issue it, it's definitely specific to the fork. Using master also breaks the build: https://github.com/Sorcery/sorcery-rework/pull/6/checks?check_run_id=2214348350#step:3:117

[technion]

Can confirm I've deployed 3.0 and run my own tests without a problem. I'm continuing to investigate.

[technion]

I repeated the entire process from my above gist, starting with

% git clone https://github.com/athix/ruby-argon2.git

And the whole process completed the same.

I'm fairly sure, looking at your Github worker, the issue here is that pulling a gem in from a git source the way you have done above doesn't actually initialize the sub modules and build them. Any test that actually installs the gem - with the prebuilt .so file - should be fine, but I appreciate that isn't helpful to your case here.

I think if you're going to use a git branch this way, you'll need to have your workflow build it. Consider the build process in .github/workflows/ruby.yml and note what it builds out of the box for you. Your inclusion will require that but isn't getting it.

[joshbuker]

Thanks for being so thorough, I appreciate the help debugging this!

I'm updating #44 with the Ruby 3.0 signature stuff, but I'll look into your suggestion about making the workflow include the build steps once I'm done. It sounds promising. 🙂

[joshbuker]

@technion Eureka!

Turns out you just need to pass bundler submodules: true to let it auto expand the submodules. 🤦‍♂️