Build failure on 2.x and solution (related to readline)

[ahacking]

It seems recent versions of ruby (2.x) have a hard time with readline and fail to build on many systems. The error typically being:

Installing ruby-2.1.0...

BUILD FAILED

Inspect or clean up the working tree at /tmp/ruby-build.20140317220101.16966
Results logged to /tmp/ruby-build.20140317220101.16966.log

Last 10 log lines:
checking ../.././parse.y and ../.././ext/ripper/eventids2.c
installing default ripper libraries
installing default openssl libraries
linking shared-object openssl.so
make[2]: Leaving directory '/tmp/ruby-build.20140317220101.16966/ruby-2.1.0/ext/openssl'
linking shared-object ripper.so
make[2]: Leaving directory '/tmp/ruby-build.20140317220101.16966/ruby-2.1.0/ext/ripper'
make[1]: Leaving directory '/tmp/ruby-build.20140317220101.16966/ruby-2.1.0'
uncommon.mk:180: recipe for target 'build-ext' failed
make: *** [build-ext] Error 2

While it looks like openssl error upon viewing the build log the actual culprit is readline:

make[2]: Entering directory '/tmp/ruby-build.20140317220101.16966/ruby-2.1.0/ext/ripper'
compiling ripper.c
installing default cparse libraries
linking shared-object racc/cparse.so
readline.c: In function ‘Init_readline’:
readline.c:1977:26: error: ‘Function’ undeclared (first use in this function)
     rl_pre_input_hook = (Function *)readline_pre_input_hook;
                          ^
readline.c:1977:26: note: each undeclared identifier is reported only once for each function it appears in
readline.c:1977:36: error: expected expression before ‘)’ token
     rl_pre_input_hook = (Function *)readline_pre_input_hook;
                                    ^
readline.c: At top level:
readline.c:634:1: warning: ‘readline_pre_input_hook’ defined but not used [-Wunused-function]
 readline_pre_input_hook(void)
 ^
Makefile:228: recipe for target 'readline.o' failed
make[2]: *** [readline.o] Error 1
make[2]: Leaving directory '/tmp/ruby-build.20140317220101.16966/ruby-2.1.0/ext/readline'
exts.mk:198: recipe for target 'ext/readline/all' failed
make[1]: *** [ext/readline/all] Error 2

I found the following allows ruby 2.0 variants including 2.1.0 and 2.1.1 to build successfully:

RUBY_CONFIGURE_OPTS=--with-readline-dir="/usr/lib/libreadline.so" rbenv install 2.1.1

The above worked on Arch Linux and likely works on many systems with readline installed in /usr/lib/libreadline.so and more generally /some/prefix/libreadline.so

Perhaps you can update the README to cover this error and the solution because there was precious little information on overcoming this problem that I could find.

[hsbt]

I already requested to support readline-6.3 to ruby-core. this issue isn't ruby-build issue.

ref. https://bugs.ruby-lang.org/issues/9630

[ahacking]

Are you serious closing the issue like that? Yes it is a ruby-build issue!

I would have expected you would help users to actually succeed with ruby-build as it is NOT possible to build on many systems RIGHT NOW. I burnt a lot of time figuring out how to get it to work and no doubt others just give up and you never hear about it.

A simple heads up in your readme as you do for Mac would save people a lot of time for a current problem using your tool to achieve the goal of building ruby. It sounds like you would rather have users waste their time scratching their heads. Why then would I use ruby-build? Where is the value add?

Yes I can see there is a patch on trunk which may help with a particular readline issue but that doesn't help me or others use ruby-build to get a functioning ruby 2.0.x or 2.1.x today.

[sferik]

@ahacking This issue cannot be fixed in ruby-build, it needs to be fixed upstream, in the ruby project. From now on, you can follow this issue here, where it can be addressed by the appropriate team.

[sferik]

Why then would I use ruby-build? Where is the value add?

@ahacking I’d like to offer you a 100% refund for your troubles.

ruby-build is a community project. I’m sorry you had some trouble getting it working. If you read the LICENSE it says—nay, it screams:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

To answer your question directly: the value that ruby-build provides is that you didn’t have to build your own Ruby build tool from scratch. You’re welcome.

[sferik]

A simple heads up in your readme as you do for Mac would save people a lot of time for a current problem using your tool to achieve the goal of building ruby.

@ahacking This patch would be a welcome addition. Thank you for volunteering to contribute it.

[ahacking]

I've done all I'm going to do, if you can't be bothered to paste my
solution into your readme then the last thing I want to do is to tweak your
wording and go back and forth and have a mini bikeshed over it until you
get the exact wording you could have written.

I won't be using ruby-build moving forward as there is no value in it
because the maintainers clearly don't care about their users. If you don't
care then why the fsck should I ?

I use like to use tools that actually give me something.
On Mar 18, 2014 12:21 AM, "Erik Michaels-Ober" [email protected]
wrote:

A simple heads up in your readme as you do for Mac would save people a lot
of time for a current problem using your tool to achieve the goal of
building ruby.

@ahacking https://github.com/ahacking This patch would be a welcome
addition. Thank you for volunteering to contribute it.

Reply to this email directly or view it on GitHubhttps://github.com//issues/526#issuecomment-37820404
.

[mislav]

@hsbt I want to explore if it's possible that ruby-build works around this issue by specifying --with-readline-dir or something. Do you think that's plausible? Can you help me to understand the issue better? I lack context.

@ahacking Sorry you lost time over this. I've lost time trying to get Rubies to build on systems I'm not familiar myself, and I know how it sucks and I would rather that somebody solves it for me. The truth is, we don't have excellent support for Fedora and Arch right now since Ruby build scripts have subtle bugs on them, and ruby-build is essentially a thin wrapper around ./configure && make install.

We are not able to solve all compile problems, but if there's a possibility to work around annoyances, I would like to explore that like we did with openssl issues on Mac.

[sferik]

@mislav Sorry, I didn’t mean to clobber your re-opening. You’re welcome to investigate this further, if you like. In the mean time, I’ve added instructions to the README, as was suggested @ahacking.

[mislav]

Nice of you to add the caveat. It still seems that it might be possible to detect the case and add the flag ourselves, but I'm waiting for @hsbt for advice.

[hsbt]

@ahacking At first, I'm sorry to close before your response.

@mislav @sferik RUBY_CONFIGURE_OPTS workaround looks wrong to me. This issue caused by header file. Therefore, Addition of "--with-readline-dir=/usr/lib/libreadline.so" cann't solve this issue. I think this case skipped build to readline extension.

And, I think this issue should be solved by upstream. It's most efficient solution by alternative tools like rvm, ruby-install.

[ahacking]

Thanks for looking into it.

I can't verify if readline actually built or not as I don't have the system image anymore. Its highly probable like you say that it just caused building of readline to be skipped. I will investigate more closely a bit later today when I bootstrap a new system image. I will also see what the Arch package script for ruby 2.1.1 does. I just prefer to sandbox everything I do and not rely on system rubies or gems ever.

I did try setting --with-readline-dir=/usr/include/readline as one attempt to get things building but that didn't work either (I'm not entirely sure without further digging what that option is expecting).

No doubt ruby is busted as per the patch on their trunk and there is probably not much getting around it other than using the patched source, or manually applying the patch to release 2.1.1 or waiting for upstream patch release. Since I am not close to this, does anyone have any idea on when we would expect the ruby team to release a 2.1.1pX release incorporating the readline build fix?

[mislav]

Because of ongoing issues like these, I'm beginning to think that we should ship some basic patches with ruby-build to make current releases of Ruby actually compile on different systems, a la RVM.

A future release of Ruby might tackle this issue, but anyone who tries to install Ruby 2.1.1 on Arch in the future will be greeted with the same failure. Same for Fedora: #443

[ahacking]

I didn't want to suggest that but yes I think tools like ruby-build should
support applying patches prior to the build to make building ruby fuss free
for as many people as possible.

I am surprised the readline issue slipped through the ruby team. Do they
not run test builds on a wide range of VMs?
On Mar 18, 2014 1:12 PM, "Mislav Marohnić" [email protected] wrote:

Because of ongoing issues like these, I'm beginning to think that we
should ship some basic patches with ruby-build to make current releases of
Ruby actually compile on different systems, a la RVM.

A future release of Ruby might tackle this issue, but anyone who tries to
install Ruby 2.1.1 on Arch in the future will be greeted with the same
failure. Same for Fedora: #443#443

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/526#issuecomment-37895829
.

[hsbt]

@ahacking We fixed this issue immediately in trunk when released readline-6.3 on Gentoo and Arch Linux. If you can't wait to release of next stable package and build 2.1.1 with original patch, We have no idea to fix your issue.

[ahacking]

@hsbt I don't have issue with patching and building ruby and I am not complaining about the responsiveness of the readline fix. But there is always room to improve the breadth of the testing given the success of ruby and the number of platforms it gets deployed on.

I do agree with @mislav that ruby-build could smooth over any rough edges if it shipped with patches for these kinds of problems. Ideally ruby-build should "just work" and it would be beneficial to have the ability within ruby-build to deal with any short term issues that arise like readline by applying patches.

[EdVanDance]

I'm also on archlinux.
The issue is not fixed with the README instructions, beause require 'readline' in irb fails, so it is
not build.

Using the patches below I managed to build 2.0.0-p451 and 2.1.1.

I used this patch for 2.0.0-p451:

diff -ur orig/ext/readline/readline.c fixed/ext/readline/readline.c
--- orig/ext/readline/readline.c    2014-03-18 13:53:31.866359527 +0100
+++ fixed/ext/readline/readline.c   2014-03-18 13:56:26.390247250 +0100
@@ -1883,7 +1883,7 @@

     rl_attempted_completion_function = readline_attempted_completion_function;
 #if defined(HAVE_RL_PRE_INPUT_HOOK)
-    rl_pre_input_hook = (Function *)readline_pre_input_hook;
+    rl_pre_input_hook = (rl_hook_func_t *)readline_pre_input_hook;
 #endif
 #ifdef HAVE_RL_CATCH_SIGNALS
     rl_catch_signals = 0;

And this one for 2.1.1 (from archlinux package repository):

From 4c4da3fc650a3595ecc06f49072f1ffae07db706 Mon Sep 17 00:00:00 2001
From: Thomas Dziedzic <[email protected]>
Date: Sat, 1 Mar 2014 21:41:28 -0800
Subject: [PATCH] Fix undeclared identifier error by using the actual type of
 rl_pre_input_hook

---
 ext/readline/readline.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ext/readline/readline.c b/ext/readline/readline.c
index 659adb9..7bc0eed 100644
--- a/ext/readline/readline.c
+++ b/ext/readline/readline.c
@@ -1974,7 +1974,7 @@ Init_readline()

     rl_attempted_completion_function = readline_attempted_completion_function;
 #if defined(HAVE_RL_PRE_INPUT_HOOK)
-    rl_pre_input_hook = (Function *)readline_pre_input_hook;
+    rl_pre_input_hook = (rl_hook_func_t *)readline_pre_input_hook;
 #endif
 #ifdef HAVE_RL_CATCH_SIGNALS
     rl_catch_signals = 0;
-- 
1.9.0

Build with rbenv install --patch 2.x.x < the.patch.
When prompted for file to be patched, I entered: ext/readline/readline.c.

[mislav]

Here is the 2.1.1 patch I ported from Ruby trunk: https://gist.github.com/mislav/a18b9d7f0dc5b9efc162

Usage:

curl -fsSL https://gist.github.com/mislav/a18b9d7f0dc5b9efc162.txt | rbenv install --patch 2.1.1

Patches by @EdVanDance might also work; I haven't tried.

[ahacking]

Confirmed this works for me on Arch 3.13.

Thankyou!

I guess the readme now needs to be updated.
On Mar 18, 2014 11:42 PM, "Mislav Marohnić" [email protected]
wrote:

Here is the 2.1.1 patch I ported from Ruby trunk:
https://gist.github.com/mislav/a18b9d7f0dc5b9efc162

Usage:

curl -fsSL https://gist.github.com/mislav/a18b9d7f0dc5b9efc162.txt | rbenv install --patch 2.1.1

Patches by @EdVanDance https://github.com/EdVanDance might also work; I
haven't tried.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/526#issuecomment-37933242
.

[sferik]

@ahacking Patches welcome!

[mislav]

Troubleshooting instructions for some Ruby version + OS version combinations and tips for workarounds should go to the wiki, not the Readme, so they can be updated and adjusted outside of git.

[dajoen]

@mislav Thanks for your patch. It worked for me on Xubuntu 14.04.

[istana]

Thanks, it works in Debian unstable too. Do i understand correctly, that the patch for this is only in the 2.2.0-dev for now and will be merged to 2.1.1 later? (https://github.com/ruby/ruby/tree/ruby_2_1 ends with backport 9299 and this is 9630)

[LeonB]

I had the same issue on Ubuntu 14.04. curl -fsSL https://gist.github.com/mislav/a18b9d7f0dc5b9efc162.txt | rbenv install --patch 2.1.1 solved it. Thanks!

[adermard]

Downgrade to readline 6.2.4 worked for me on MacOS Mavericks. This solution seems like the cleanest fix for now until the code is updated.

[aviris]

Instead of downgrading to 6.2.4 I did the following:

1. delete everything in the /usr/local/Cellar/readline/6.3.3 folder (manually - not with 'brew uninstall')
2. Download the "tar file of the current version with all official patches applied" from http://tiswww.case.edu/php/chet/readline/rltop.html#Availability
3. configure, make & install to /usr/local/Cellar/readline/6.3.3

./configure --prefix=/usr/local/Cellar/readline/6.3.3

Install Ruby 2.1.1

[groverz]

Patch worked on Ubuntu 14.04 LTS when compiling Ruby 2.1.1 via rbenv install:

curl -fsSL https://gist.github.com/mislav/a18b9d7f0dc5b9efc162.txt | \
rbenv install --patch --keep 2.1.1

thanks @mislav

[prusswan]

great stuff, fixed in 2.0.0-p481
https://www.ruby-lang.org/en/news/2014/05/09/ruby-2-0-0-p481-is-released/

[hsbt]

2.0.0-p481 and 2.1.2 is released.

• Added definition of 2.0.0-p481 #567
• Ruby 2.1.2 is released! #566

Please update latest version of ruby-build, and build above version of Ruby.

[jalcine]

👯 about time! Thank Ruby core devs and @mislav and everyone :)

[AlexVPopov]

This worked for me on Ubuntu 14.04.. I had to locate libreadline.so first with:

sudo find / -name 'libreadline.so'

It was /usr/lib/x86_64-linux-gnu/libreadline.so. I then ran:

RUBY_CONFIGURE_OPTS=--with-readline-dir="/usr/lib/x86_64-linux-gnu/libreadline.so" rbenv install 2.1.1

as suggested by @ahacking and it worked. However, this only helps in installing ruby, but the installed ruby does not have readline support. Thus I got this error, when trying to start the console: ``require': cannot load such file -- readline (LoadError).

Finally, I did what @mislav suggested:

curl -fsSL https://gist.github.com/mislav/a18b9d7f0dc5b9efc162.txt | rbenv install --patch 2.1.1

and everything worked.

[x-yuri]

Unfortunately, I'm forced to switch to rvm. Because I need solution that just works. As a parting gift, I wanted to describe how I installed ruby-2.0.0-p247 on ubuntu-14.04.

---

So, I believe the best solution in the thread is of @mislav, if I'm not missing anything. It works for ruby-2.1.1, but I've got ruby-2.0.0-p247 here.

One thing that might come to mind is why not just use rvm's patches. There are two dirs for that in rvm's repository. I don't know the details, but from this file one might conjecture, that we need these patches. Looking at ruby's issue confirms this idea, sort of. There are also r45488 and r45518 revisions mentioned.

It might make sense to describe briefly how to match those revisions with commits. In order to search for a corresponding commit I did this, for example:

git log --all --grep=@45518

The output was like:

commit d2a8e28597df946842f44d6e0d2cbfe98863b877
Author: nobu <nobu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e>
Date:   Fri Apr 4 15:31:23 2014 +0000

    readline/extconf.rb: rl_hook_func_t

    * ext/readline/extconf.rb (rl_hook_func_t): check pointer type.
      [ruby-dev:48089] [Bug #9702]

    git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@45518 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

As you can see, we have the revision in git-svn-id line here. And also we have a branch (trunk). So if we were to look for r45772, we would get:

commit ccbe6f31df6434823d9d48db7a0c4d3bcc0e3262
Author: nagachika <nagachika@b2dd03c8-39d4-4d8f-98ff-823fe69b080e>
Date:   Thu May 1 11:55:03 2014 +0000

    merge revision(s) r45225,r45240,r45488: [Backport #9578] [Backport #9630]

        * ext/readline/readline.c (Init_readline): Use rl_hook_func_t instead
          of Function to support readline-6.3.  (rl_hook_func_t is available
          since readline-4.2.)
          Reported by Dmitry Medvinsky.  [ruby-core:61141] [Bug #9578]

        * ext/readline/extconf.rb (rl_hook_func_t): define as Function for
          very old readline versions.  [ruby-core:61209] [Bug #9578]

        * ext/readline/extconf.rb: fix typo, `$defs` not `$DEFS`.
          [ruby-core:61756] [Bug #9578]


    git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_1@45772 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Which is on the ruby_2_1 branch. Supplying --name-status option is also advantageous, to list the files being changed.

Now then, we're interested in revisions r45225 and r45240. So I thought this would do:

cat <(curl -fsSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/patches/ruby/changeset_r45225.diff) <(curl -fsSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/patches/ruby/changeset_r45240.diff) | rbenv install -p 2.0.0-p247

But I had yet several more things to do down the road:

can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/ext/readline/readline.c  (revision 45224)
|+++ b/ext/readline/readline.c  (revision 45225)
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
can't find file to patch at input line 14
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/ext/readline/extconf.rb  (revision 45239)
|+++ b/ext/readline/extconf.rb  (revision 45240)
--------------------------
No file to patch.  Skipping patch.
2 out of 2 hunks ignored

So it didn't get the patch level.

On a side note, the patch is applied here. So, you can add some debugging statements to be able to experiment with applying a patch, like these ones:

printf -- '-%.s' {1..80} ; echo
pwd 
echo patch -p$striplevel --force -i "$patchfile"
cat "$patchfile"
printf -- '-%.s' {1..80} ; echo

Moving on, my next attempt was as follows:

cat <(curl -fsSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/patches/ruby/changeset_r45225.diff) <(curl -fsSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/patches/ruby/changeset_r45240.diff) | sed -E 's/^((---|\+\+\+) )(a|b)\/([^ ]+\s+\(revision [0-9]+\))$/\1\4/' | rbenv install -p 2.0.0-p247

Now, the second patch didn't apply:

patching file ext/readline/readline.c
Hunk #1 succeeded at 1883 (offset -91 lines).
patching file ext/readline/extconf.rb
Hunk #2 FAILED at 98.
1 out of 2 hunks FAILED -- saving rejects to file ext/readline/extconf.rb.rej

Apparently, there were some changes introduced between 2.0.0-p247 and r45240. You can explore them with the following command (should I use three dots there...):

git log -p v2_0_0_247..2bb881148493651c577d95d55a4571c36de589b6 ext/readline/extconf.rb

So, my next and last attempt was as follows:

cat <(curl -fsSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/patches/ruby/changeset_r45225.diff) <(curl -fsSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/patches/ruby/changeset_r45240.diff) | sed -E 's/^((---|\+\+\+) )(a|b)\/([^ ]+\s+\(revision [0-9]+\))$/\1\4/' | sed '/rl_delete_text/d' | rbenv install -p 2.0.0-p247

Pretty "ssh public key"-like solution, huh? :) Not sure if that's all to be done, but at least it installs.

---

Couple of suggestions for developers. It makes sense to describe how to search for and apply patches for people to be able to do it themselves, unless rbenv isn't going to do it automatically. In README or in the wiki, I suppose. Additionally, being able to accept patches like this is also advantageous, either by detecting proper patch level, or by accepting patch level as an argument.

[mislav]

Thanks for the tips @x-yuri. Sorry you had so much trouble applying a couple of patches. The thing is, ruby-build will detect and automatically instruct patch to strip a/ and b/ prefixes when the patchfile includes the diff --git line: 63f6fa2

Because these patches have the prefixes but seem to be saved without the diff --git directive, you could pipe them through filterdiff --strip=1 before passing to ruby-build instead of making a complicated sed command.

[weezilla]

Digital Ocean Ubuntu 140.04. 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The patch method suggested for 2.1.1 and other versions worked for me.

curl -fsSL https://gist.github.com/mislav/a18b9d7f0dc5b9efc162.txt | rbenv install --patch 2.1.0

[gabriella4c]

If you just want to get Ruby 2.0.0-p353 working without upgrading the patch version have a look here: #587 (comment).