-
Notifications
You must be signed in to change notification settings - Fork 392
[PROF-2585] Warn on incompatible rollbar version #1300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Rollbar gem versions <= 3.1.1 have an incompatibility with ddtrace's CPU profiling. This was ack'd and fixed upstream in <rollbar/rollbar-gem#1018>. This commit adds code to detect when customers are still using a legacy version of rollbar, printing a warning to ask them to upgrade. Note: Because the fixed version of rollbar has not yet been released, the tests that validate that the warning is not shown on unpatched versions are disabled and tagged with a FIXME. See the "FIXME NEW ROLLBAR NEEDED" note in the setup_spec.rb for more details on what's disabled and what steps we need to take once the new version gets released. First commit @ Datadog 🎉
d7ea76a
to
662a44e
Compare
Codecov Report
@@ Coverage Diff @@
## feature/profiling #1300 +/- ##
==================================================
Coverage 97.91% 97.92%
==================================================
Files 848 848
Lines 39745 39779 +34
==================================================
+ Hits 38918 38952 +34
Misses 827 827
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not super familiar with the profiling branch yet but generally lgtm, deferring to @delner here though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the actual warning itself looks good! However, I think we could leverage a test helper to greatly simplify the tests for this, and eliminate much of the appraisal related changes.
We did this in another file, which tested for behavior based on the presence or non-presence of a gem (and its version):
RSpec.describe Datadog::Contrib::Faraday::Integration do
extend ConfigurationHelpers
describe '.version' do
subject(:version) { described_class.version }
context 'when the "faraday" gem is loaded' do
include_context 'loaded gems', faraday: described_class::MINIMUM_VERSION
it { is_expected.to be_a_kind_of(Gem::Version) }
end
context 'when "faraday" gem is not loaded' do
include_context 'loaded gems', faraday: nil
it { is_expected.to be nil }
end
end
I think you could use this, and remove all the appraisal changes. What do you think @ivoanjo ?
Ah, interesting. It took me a few minutes to wrap my head around what the helper was doing ;) If I understand correctly, the I think the general idea would work for this PR. The tests in my first attempt are more realistic, but they do introduce a number of extra steps (and make testing slower), as well as making it hard for testing all variants in one run (you always need to restart your ruby with different environments). I can't quite use the helper directly since I'm using $ pry
[1] pry(main)> Gem.loaded_specs.keys.include?("rollbar")
=> false
[2] pry(main)> Gem::Specification.find_all_by_name("rollbar").map { |it| [it.name, it.version] }
=> [["rollbar", Gem::Version.new("3.1.1")], ["rollbar", Gem::Version.new("2.27.1")]] ...but the approach is the same -- I shall mock the other method instead. Now I shall quote Arnie: |
The approach taken in my first attempt to test this change was rather heavy handed -- needing a lot of changes to the appraisals/rakefile and multiple runs of the test to validate. As suggested during review, let's instead mock out the Gem apis, which are documented and pretty stable, and avoid the impact on the test suite running time/complexity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the use of mocks here is appropriate (Gem::Specification
is a stable API) and serves us well in simplifying the test code; it's now much easier to reason about its mechanisms.
Nice work @ivoanjo ! Thank you :)
[PROF-2585] Warn on incompatible rollbar version
[PROF-2585] Warn on incompatible rollbar version
When I initially worked on #1300 ("Warn on incompatible rollbar version") my understanding of the issue on rollbar/rollbar-gem#1018 was incomplete and I did not realize that old rollbar versions, beyond breaking datadog profiler's CPU time instrumentation, could actually break the customer's application. One of our private beta customers mentioned he had seen both the warning but also had seen a stack trace: ``` [1] ! Unable to load application: SystemStackError: stack level too deep /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:5:in `initialize_with_rollbar': stack level too deep (SystemStackError) from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' from /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:6:in `initialize_with_rollbar' from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' ``` This led me to try to reproduce this issue. I discovered that with ```ruby ENV['DD_PROFILING_ENABLED'] = 'true' require 'ddtrace/profiling/preload' require 'rollbar' Rollbar.configure do |c| c.access_token = "nope" end Thread.new { }.join ``` I could trigger the issue, and that the current behavior was that the warning from #1300 would be printed, but then the application would proceed to fail. Having a bit more experience with the codebase now, I realize that the correct place to put this check is in the CPU extension #unsupported_reason method, which will ensure the extension is not loaded at all if an incompatible version of rollbar is around. The resulting behavior is that even with an old rollbar version, profiler will happily load and work; it will just omit the CPU time profiling, similarly to how it behaves on other platforms where for different reasons we don't support CPU time profiling.
[PROF-2585] Warn on incompatible rollbar version
When I initially worked on #1300 ("Warn on incompatible rollbar version") my understanding of the issue on rollbar/rollbar-gem#1018 was incomplete and I did not realize that old rollbar versions, beyond breaking datadog profiler's CPU time instrumentation, could actually break the customer's application. One of our private beta customers mentioned he had seen both the warning but also had seen a stack trace: ``` [1] ! Unable to load application: SystemStackError: stack level too deep /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:5:in `initialize_with_rollbar': stack level too deep (SystemStackError) from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' from /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:6:in `initialize_with_rollbar' from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' ``` This led me to try to reproduce this issue. I discovered that with ```ruby ENV['DD_PROFILING_ENABLED'] = 'true' require 'ddtrace/profiling/preload' require 'rollbar' Rollbar.configure do |c| c.access_token = "nope" end Thread.new { }.join ``` I could trigger the issue, and that the current behavior was that the warning from #1300 would be printed, but then the application would proceed to fail. Having a bit more experience with the codebase now, I realize that the correct place to put this check is in the CPU extension #unsupported_reason method, which will ensure the extension is not loaded at all if an incompatible version of rollbar is around. The resulting behavior is that even with an old rollbar version, profiler will happily load and work; it will just omit the CPU time profiling, similarly to how it behaves on other platforms where for different reasons we don't support CPU time profiling.
[PROF-2585] Warn on incompatible rollbar version
When I initially worked on #1300 ("Warn on incompatible rollbar version") my understanding of the issue on rollbar/rollbar-gem#1018 was incomplete and I did not realize that old rollbar versions, beyond breaking datadog profiler's CPU time instrumentation, could actually break the customer's application. One of our private beta customers mentioned he had seen both the warning but also had seen a stack trace: ``` [1] ! Unable to load application: SystemStackError: stack level too deep /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:5:in `initialize_with_rollbar': stack level too deep (SystemStackError) from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' from /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:6:in `initialize_with_rollbar' from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' ``` This led me to try to reproduce this issue. I discovered that with ```ruby ENV['DD_PROFILING_ENABLED'] = 'true' require 'ddtrace/profiling/preload' require 'rollbar' Rollbar.configure do |c| c.access_token = "nope" end Thread.new { }.join ``` I could trigger the issue, and that the current behavior was that the warning from #1300 would be printed, but then the application would proceed to fail. Having a bit more experience with the codebase now, I realize that the correct place to put this check is in the CPU extension #unsupported_reason method, which will ensure the extension is not loaded at all if an incompatible version of rollbar is around. The resulting behavior is that even with an old rollbar version, profiler will happily load and work; it will just omit the CPU time profiling, similarly to how it behaves on other platforms where for different reasons we don't support CPU time profiling.
[PROF-2585] Warn on incompatible rollbar version
When I initially worked on #1300 ("Warn on incompatible rollbar version") my understanding of the issue on rollbar/rollbar-gem#1018 was incomplete and I did not realize that old rollbar versions, beyond breaking datadog profiler's CPU time instrumentation, could actually break the customer's application. One of our private beta customers mentioned he had seen both the warning but also had seen a stack trace: ``` [1] ! Unable to load application: SystemStackError: stack level too deep /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:5:in `initialize_with_rollbar': stack level too deep (SystemStackError) from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' from /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:6:in `initialize_with_rollbar' from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' ``` This led me to try to reproduce this issue. I discovered that with ```ruby ENV['DD_PROFILING_ENABLED'] = 'true' require 'ddtrace/profiling/preload' require 'rollbar' Rollbar.configure do |c| c.access_token = "nope" end Thread.new { }.join ``` I could trigger the issue, and that the current behavior was that the warning from #1300 would be printed, but then the application would proceed to fail. Having a bit more experience with the codebase now, I realize that the correct place to put this check is in the CPU extension #unsupported_reason method, which will ensure the extension is not loaded at all if an incompatible version of rollbar is around. The resulting behavior is that even with an old rollbar version, profiler will happily load and work; it will just omit the CPU time profiling, similarly to how it behaves on other platforms where for different reasons we don't support CPU time profiling.
[PROF-2585] Warn on incompatible rollbar version
When I initially worked on #1300 ("Warn on incompatible rollbar version") my understanding of the issue on rollbar/rollbar-gem#1018 was incomplete and I did not realize that old rollbar versions, beyond breaking datadog profiler's CPU time instrumentation, could actually break the customer's application. One of our private beta customers mentioned he had seen both the warning but also had seen a stack trace: ``` [1] ! Unable to load application: SystemStackError: stack level too deep /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:5:in `initialize_with_rollbar': stack level too deep (SystemStackError) from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' from /usr/local/bundle/gems/rollbar-3.1.1/lib/rollbar/plugins/thread.rb:6:in `initialize_with_rollbar' from /usr/local/bundle/gems/ddtrace-0.45.0.feature.profiling.109457/lib/ddtrace/profiling/ext/cthread.rb:47:in `initialize' ``` This led me to try to reproduce this issue. I discovered that with ```ruby ENV['DD_PROFILING_ENABLED'] = 'true' require 'ddtrace/profiling/preload' require 'rollbar' Rollbar.configure do |c| c.access_token = "nope" end Thread.new { }.join ``` I could trigger the issue, and that the current behavior was that the warning from #1300 would be printed, but then the application would proceed to fail. Having a bit more experience with the codebase now, I realize that the correct place to put this check is in the CPU extension #unsupported_reason method, which will ensure the extension is not loaded at all if an incompatible version of rollbar is around. The resulting behavior is that even with an old rollbar version, profiler will happily load and work; it will just omit the CPU time profiling, similarly to how it behaves on other platforms where for different reasons we don't support CPU time profiling.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## feature/profiling #1300 +/- ##
==================================================
Coverage 97.91% 97.92%
==================================================
Files 848 848
Lines 39745 39779 +34
==================================================
+ Hits 38918 38952 +34
Misses 827 827 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Rollbar gem versions <= 3.1.1 have an incompatibility with ddtrace's
CPU profiling. This was ack'd and fixed upstream in
rollbar/rollbar-gem#1018.
This commit adds code to detect when customers are still using a legacy
version of rollbar, printing a warning to ask them to upgrade.
Note: Because the fixed version of rollbar has not yet been released,
the tests that validate that the warning is not shown on unpatched
versions are disabled and tagged with a FIXME.
See the "FIXME NEW ROLLBAR NEEDED" note in the setup_spec.rb for more
details on what's disabled and what steps we need to take once the
new version gets released.
First commit @ Datadog 🎉