Rdoc TokenStream issues on Ruby 2.5.0 (Rdoc 6.0.1)

[jules2689]

Rdoc seems to have issues on a file in Rdoc 6.0.0. I checked and it seemed to have issues handling an END. At the point of exception, t was {:line_no=>43, :char_no=>67, :kind=>:on_ignored_nl, :text=>nil, :state=>#<Ripper::Lexer::State: EXPR_END|EXPR_LABEL>}... you can see that text is nil.

Specifically, rdoc seems to choke on the lines:

 x = features                  # 'a=b:c=d'
    .split(':')                # ['a=b', 'c=d']
    .map { |f| f.split('=') }  # [['a', 'b'], ['c', 'd']]

on this file: https://gist.github.com/jules2689/0bf462a4c045ddad69df04c9655d16ff

It also fails on the lines:

@running_servers = out                     # " 1 /sbin/launchd\n..."
            .lines                                   # [" 1 /sbin/launchd\n", ...]
            .map { |line| line.chomp.split(' ', 2) } # [[" 1", "/sbin/launchd",], ...]
            .select { |_, cmd| cmd =~ server_pat }   # [["1234", "spring server | shopify | ..."]]
            .map { |pid, _cmd| pid.to_i } # [1234]

This suggests to me that rdoc can no longer support this valid ruby syntax.

Backtrace:

error generating Dev/ShellHook/AutoEnv/EnvUtil.html: no implicit conversion of nil into String (TypeError)
/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/token_stream.rb:64:in `escapeHTML'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/token_stream.rb:64:in `block in to_html'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/token_stream.rb:19:in `map'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/token_stream.rb:19:in `to_html'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/markup.rb:114:in `markup_code'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:168:in `block (5 levels) in generate_class'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:125:in `each'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:125:in `block (4 levels) in generate_class'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:118:in `each'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:118:in `block (3 levels) in generate_class'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:116:in `each'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:116:in `block (2 levels) in generate_class'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/context.rb:765:in `block in each_section'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/context.rb:764:in `each'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/context.rb:764:in `each_section'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/template/darkfish/class.rhtml:54:in `block in generate_class'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/erb.rb:885:in `eval'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/erb.rb:885:in `result'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:753:in `template_result'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:732:in `block in render_template'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:727:in `open'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:727:in `open'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:727:in `render_template'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:356:in `generate_class'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:382:in `block in generate_class_files'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:379:in `each'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:379:in `generate_class_files'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/generator/darkfish.rb:246:in `generate'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/rdoc.rb:533:in `block in generate'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/rdoc.rb:528:in `chdir'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/rdoc.rb:528:in `generate'
	/opt/rubies/2.5.0/lib/ruby/2.5.0/rdoc/rdoc.rb:507:in `document'
	/opt/rubies/2.5.0/bin/rdoc:20:in `<main>'

This does not occur on Rdoc 5.0.0 in Ruby 2.4.3.

[aycabta]

Tentatively, I made minimal reproducable code.

def meth
  variable # comment
    .chain
end