-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
16.7.0: Rake aborts launch with "failed to get urandom" #2872
Comments
I also encountered the same problem cat /proc/version docker version Client: Server: |
I have same issue running 16.7.0. The logs for the gitlab service container:
|
I did some more digging and I'm fairly certain this is due to old linux kernels. Since I'm running GitLab on a Synology, there's nothing I can to about running a rather prehistoric 3.10 kernel. Short of moving GitLab to a different system I guess the only option would be to run it on Docker inside a VM on the same Synology. I don't much fancy the performance overhead of that approach. Anybody got practical experience with that? |
Have the same issue with gitlab on Synology. It seems the only hope is when Synology update one day the gitlab package which is based on this repo / docker image. |
@Algorab I was under the impression that Synology made clear they wouldn't be releasing any more GitLab updates. I had a quick look, but couldn't find any source on that, so I could be wrong. But even if they release another GitLab update, I'm pretty sure that's not the issue here. 16.7 requires Ruby 3.1. And Ruby 3.1 doesn't work with older Linux kernels, like the 3.10 Linux kernel found on any Synology that's at least 5 years old or so. Inexplicably, Synology make it a policy not to update the Linux kernel of their storage products, except for the occasional back-port of critical security updates. Unless I'm missing something, that means that if a Synology can't run GitLab 16.7 today, then it never will, except perhaps in a VM. |
Just found myself in the same situation. Still I ask myself, where this urandom method request was introduced. On the GitLab code or in the underlying OS Image. |
@CDann3r I think you should be able to just start it with 16.6.5 — 16.7 never comes up and never even gets started with any migrations. So your database should still be in the state it was in when 16.3.6 was running. But chances are you won't be able to upgrade GitLab, except for the remaining updates to 16.6 that will presumably be released over the next moth or so. |
@mihelm That is what I did the last hour and thanks god it works. Thank you that you reported it. It helped me to find a quick solution. |
I too have this problem running GitLab with docker-compose on a Synology RS1219+. Keeping Gitlab on 16.6.5 for now. But the admin page is getting angrier every time a new update gets released. |
No, I have the same issue and Docker containers do need the kernel to run. The issue seems to be that it's the old kernels used in Synology systems. |
Well, it's not so much a GitLab issue as a Ruby issue — GitLab 16.7 requires a newer version of Ruby, which introduced the problem. It's conceivable a newer version of Ruby could fix it. But I have no idea if GitLab 16.7 would work with a newer version of Ruby than the one specified. It might though — the release notes of 16.7 state:
|
I don't think it's a Ruby issue. It's a GitLab issue. Before 16.7.0 I'm guessing GitLab didn't use the Random.urandom function from Ruby. I tried running these Ruby versions in docker containers on my Synology: 3.0.6, 3.1.4, 3.2.0, 3.2.3 and 3.3.0. Every one of them threw an error when I tried to run Random.urandom(1) in irb. On my Windows machine this didn't happen. The documentation for Random.urandom also states that it's a platform feature.
I mapped this script to the container and defined IRBRC to point to it. This way I could call Random.urandom on the Synology. If we could somehow make sure that this override would be present in the environment GitLab is running in, this might solve it. I was thinking of making a ruby alias (alias ruby=ruby -r /path/to/filemapping) but where to put it? I want to try /etc/profile.d/ but I'm not sure GitLab runs in a way that reads the /etc/profile... any ideas? I don't think this issue is solvable in any other way than with a newer Linux version... so a fix like this one is probably the only way. And then I don't have the first clue about the consequences of overriding a function that provides "cryptographically secure pseudo-random number in binary form" ¯\(ツ)/¯ |
@Peter-Oestergaard I don't think so. The error occurs while Ruby/Rails is being initialised — I think when https://github.com/rails/rails/blob/main/activesupport/lib/active_support.rb#L26 This isn't GitLab making a strange call. This is one of the |
@mihelm You are of course entitled to your own opinion. I did however just succeed in upgrading from 16.6.5 to 16.7.3 to 16.7.4 to 16.8.1 by mapping a modified securerandom.rb file where i changed Random.urandom calls to Random.bytes calls. I wouldn't exactly call this a bulletproof solution as I will now have to keep Ruby versions in mind whenever GitLab updates as I will then need to reflect any changes to securerandom.rb in my own version... you can't win them all. |
Could you please elaborate in detail, how your solution can be implemented? That would be really great - I have running GitLab in Docker on my Synology and have the same problem... |
@Peter-Oestergaard Wouldn't you say that by injecting a patch for Incidentally, Your workaround, together with what was posted in #2886, makes me think that building 16.7 with Ruby 3.2 would be worth a try. |
@mihelm There are multiple reasons why I still think this is a change in GitLab (is this part of GitLab open source? If it is it should be trivial to confirm):
|
@mbuchber Sure thing. It really doesn't take much:
I did encounter an error about securerandom not being found. The error was not "file not found", but something like unable to load. This was because the file inside the container had the wrong settings. I did a |
@mihelm Ok. I am ready to concede. I took a look at the GitLab source. Nowhere are calls made to urandom directly. I found a lot of calls to the other methods in the SecureRandom module. But nothing that could back my guess that GitLab made any changes. So I went back in to the different Ruby containers on the Synology. And I see what you mean with regards to the dependency check as the error is reported in 3.1.4 when calling TLDR: Random.urandom will always fail on the Synology, but that function has never been used. Functions in SecureRandom will fail on versions 3.1.4, 3.2.0 and 3.2.3 and that is a problem since those are functions that GitLab uses. But maybe when GitLab bumps Ruby to version 3.3.0 I can disable my fix again... |
@Peter-Oestergaard Unfortunately, GitLab still crashes with Ruby 3.2 on Linux kernel 3.10, just much later during startup. @kkimurak created PR #2893 to upgrade Ruby. I built a 16.7.4 image from his 16.7.x branch and tried it on my Synology. The container no longer crashed right at the start — but it did crash, just later while trying to initialise It was worth a try, alas it didn't work. I doubt adapting current versions of Ruby (and its libraries) to work with such ancient kernels is high up the list of priorities for the Ruby project. Consequently, that leaves your workaround as the only option for getting GitLab to work on these old kernels. |
There is one thing that everybody using @Peter-Oestergaard 's instructions should keep in mind: Since the path on which the modified That's not a problem per se, just something to remember on future updates. |
According to the backtrace reported by @mihelm (#2893 (comment)), the strange thing is
This behavior have been fixed on upstream (ruby/ruby@64e503e) but applied only for 3.3.x release. |
@kkimurak I tried building the image with Ruby 3.3.0, but the build failed — the API has drifted too far, and at least one needed command wasn't available anymore. I'm wondering if patching bugs in Ruby during build isn't starting to stray beyond the scope of this project... Are there are instances where this sort of thing is done? |
If you think so I will just stop working on this issue. I'm ok with it because I don't use Synology.
This project applies some patches to gitlab on build. GitLab itself applies some patches for dependencies like rails/ujs, vue-jest and so on. I'm just a contributor so I just write a patch. If the maintainer think that such a patch is not needed for this project, it will be rejected. That's all.
Maybe I got same result - failed on installing rails 7.0.8 ( Here is a build log. I executed build log around the failue
|
Edit 2024-02-13 : as 16.8.2 and 16.7.5 released, I have removed these branches. If you really need to restore them, refer kkimurak/docker-gitlab@0398b16 (based on 16.8.1) and kkimurak/docker-gitlab@025fff4 (based on 16.7.4). By the way, I have pushed some branches that update ruby to 3.2.3 and backport ruby/ruby@64e503e . I have confirmed that it built successfully and at least it launches, sends notification mail (like admin password reset) and new blank project can be created from Web UI. Anyone want to try to run it on Synology ?
|
@kkimurak I'm happy to test that. Thanks for the work! |
@kkimurak It worked! Took me a while to wangle the image over to the Synology, but once it was there and properly tagged, it came up without the slightest issue! I tested the 16.7 branch: I was worried the bug you fixed with the backport might just be the first to pop up. I'm glad I was wrong on that! |
Awesome! Happy to see such a good progress. |
how did you do the upgrade using this branch please? Am on Synology also and gitlab 16.6.6 at the moment |
@drguiom I built the image on a different system, then transferred it to the Synology, tagged it and edited my kkimurak has submitted PRs to merge the fix into this repository. I think your best hope is that the maintainers approve those PRs. Short of that, you could follow the instructions Peter-Oestergaard posted earlier. That workaround achieves the same result, but needs to be tweaked every time Ruby is updated in the image. |
ok many thanks for the prompt response, am upgrading the NAS soon so hopefully the kernel will have been updated. am fine on 16.6.6 and was upgrading "because I can" as opposed to needing to. Its internal use only anyway ;-) |
I can confirm that the newly built image for 16.7.5 came up without any problems on my Synology. It took 5-6 minutes, which is a bit longer than it took for 16.6, but I guess that's just the CPU showing its age — GitLab launch times have been trending up on my Synology for a while now. @kkimurak Thank you very much for figuring out the backport and for submitting those PRs! I'm sure that's helped a lot of people! |
@mihelm Glad it works. |
same here, great work! |
now up to 16.8.2 on Synology NAS 415+ and running well |
Hi
16.7.0 refuses to start for me. Rake aborts with "failed to get urandom".
Went back to 16.6.2, which came up without a hitch.
Any ideas what might be causing that?
I use docker-compose. Docker isn't a terribly recent version, and unfortunately I have no control over which version I can run.
Cheers, michael.
The text was updated successfully, but these errors were encountered: