Fixed lexing of character syntax like ?z

Character syntax `?z` was lexed as an integer token `"zz"`.  This has
been fixed to lex as a single character string `"z"`.  While this won't
round-trip, it is acceptable for syntax highlighting.

This was discovered by Xavier Noria (@fxn) in rails/rails#7172.

Conflicts:
	History.rdoc

Author: drbrain

History.rdoc

@@ -3,6 +3,8 @@
 * Bug fixes
   * Fixed bug in syntax-highlighting that would corrupt regular expressions.
     Ruby Bug #6488 by Benny Lyne Amorsen.
+  * Fixed lexing of character syntax (<code>?x</code>).  Reported by Xavier
+    Noria.
 
 === 3.12.1 / 2013-02-05
 

lib/rdoc/ruby_lex.rb

@@ -536,13 +536,8 @@ def lex_init()
           @lex_state = EXPR_BEG;
           Token(TkQUESTION)
         else
-          str = ch
-          if ch == '\\'
-            str << read_escape
-          end
           @lex_state = EXPR_END
-          str << (ch.respond_to?(:ord) ? ch.ord : ch[0])
-          Token(TkINTEGER, str)
+          Token(TkSTRING, ch)
         end
       end
     end

test/test_rdoc_ruby_lex.rb

@@ -30,6 +30,17 @@ def test_class_tokenize
     assert_equal expected, tokens
   end
 
+  def test_class_tokenize_character_literal
+    tokens = RDoc::RubyLex.tokenize "?\\", nil
+
+    expected = [
+      @TK::TkSTRING.new( 0, 1,  0, "\\"),
+      @TK::TkNL    .new( 2, 1,  2, "\n"),
+    ]
+
+    assert_equal expected, tokens
+  end
+
   def test_class_tokenize_def_heredoc
     tokens = RDoc::RubyLex.tokenize <<-'RUBY', nil
 def x