[Bug]: rocblas link fails with relocation R_X86_64_PC32 out of range #1448
Comments
|
You are running into the issue that ld can only link objects whos sections are at most a 32bit signed away from eatch other. |
|
If you want to remove an architecture i would recommend gfx803 as this architecture is currently broken anyhow, unless you disable the asm kernels provided by tensile. |
|
Thanks for the quick feedback. Yes, if I'd build for fewer targets it would succeed, but I already removed gfx1103 as I've been building for a older codebase where gfx1103 could not be enabled. So removing gfx803 will hide the problem and it would kick the can down the road, but it doesn't appear a satisfactory long term solution. If we don't work on a solution for this now the end result is that every rocm accelerated app binary has to be built multiple times against independent and incompatible rocm builds just as if they were separate GPU compute stacks with nothing in common. This multiplies also the build time and the disk space requirements of every app, maybe not xN, but close. It would provide a sub par experience also to the end user that has then to figure the right binary to install invoke, instead of rocm solving that gpu detail in a way that is transparent to the end user. |
|
jup, this is the major reason why rocm supports so few gpus, and if they dont address this soon it has the potential to sink rocm since it forces them to drop support for old gpus exreamly fast (ever accelerating in pace as rocm get larger even) which ultimately utterly destroys customer confidence. |
|
@cgmb I think you had some other suggestions by using generic targets, but I can't remember how much progress has happened there. |
|
Thank you for bringing this issue to our attention. We appreciate your feedback and suggestions. We recommend building with the suggested targets in relation to the ROCm stack. The default target list for 6.0 includes:
The team is aware of the issue and is exploring possible solutions. Thank you for your understanding and cooperation. |
sure https://llvm.org/docs/AMDGPUUsage.html#amdgpu-generic-processor-table could be used at the cost of some performance for the non gfx10-3-generic targets. Ultimately this just kicks this can further down the road, but for now yes this would be sufficient. right now there is also no support for ELFABIVERSION_AMDGPU_HSA_V6 so those targets dont work yet, but soon i presume. |
|
Would it be possible to split the librocblas.so.4.0 in librocblas-gfx900.so.4.0 librocblas-gfx90a.so.4.0 librocblas-gfxXYZ.so.4.0... so each individual gfx target lands in a different shared library, and then have the main librocblas.so.4.0 dynamically load only the gfx targets available in hardware either during initialization of the main library or even better lazily on demand? |
|
@aagit that separate gfx .so design has been evalutated as one possible solution but we are also looking at other strategies. For now until the full list of gfx that lands in a specific release requires a new build and packaging pattern we suggest you build and package the version specific set of gfx listed in the top level CMakeLists.txt. This corresponds to our build scripts default option. |
|
I appreciate your suggestion above. I agree that's the least bad solution for the time being and I already gave it. If there's other ways to fix it, would you share them so they can be discussed here? Overall I would recommend to pick the simplest way to fix it and to ship it ASAP, because while working on a rocm accellerated app, I noticed that rocm has already been packaged in the open by building it N times and installing it in incompatible paths. The technical justification is to work around this issue (so it's like if there's a /opt/rocm1 /opt/rocm2 /opt/rocm3 /opt/rocmN installed, each one supporting a small subset of gfxes so that the link does not fail and gfx8 and gfx1103 can be enabled too). If the duplication was just on the rocm side it would be (perhaps) a lesser concern, but this causes all apps to be rebuilt N times and the build time is multiplied xN times. Last but not the least the end user would then have to pick the right binary (among N available) for its GPU or it won't work, and possibly just because of minor path differences. For example: I built an app linked against rocm that way and the total size of the N builds against N rocms, was 96GB. Then I run |
|
another temopray option if you dont want to drop any gpus in your builds might be to build "gfx90a" or just "gfx90a:xnack-" the xnack+ configuration is very rare and omitting it do sent leave any user totally in the cold (just with possibly reduced performance depending on workload) and "gfx90a" should emit code that works in both xnack+ and xnack- modes. |
|
all gfx9 gpus support xnack+, the fact that only gfx90a is built both ways is a clear hint here as to how common this is |
|
We have changed to only build our source kernels with xnack "any" for gfx90a after commit 6a267fd. We expect to adjust our gfx list before release and as always we ensure there are no linking issues on all supported OS and with any final target list. Other subdivisions of the library along functionality are also possible but none are trivial changes. Clang compiler and linker mcmodel flag changes are also possible with the current library design along with the target varations mentioned in earlier comments. This bug should likely be considered fixed and the issue closed as when you built rocblas with our supported gfx list you didn't get the error. A new issue could be created as it is unclear to me your N different ROCm use case and why the app is rebuilt and linked against all of them and not built against the latest. If your application is open source please refer to it in your new issue and detail why it is built separately for each gfx. Or if this is really just a request to support more gfx then word it as such along with your use case and gfx list. If you rebuilt rocm or rocblas with one gfx in each version please also clarify that in your new feature request issue. It could be your new issue should be in ROCm if not particular to rocBLAS. |
|
I think wanting to build a version of rocblas for all targets that work as opposed to just the default targets is a reasonable desire esp from a distro maintainers perspective where amd's support status of a specific architecture is not important, it only needs to work. Thus at the very least this i request this be left open as at least a feature request. |
|
https://src.fedoraproject.org/rpms/python-torch/blob/rawhide/f/python-torch.spec#_998 rocsolver has also the loop, but because it depends on rocblas it's not possible to tell if rocsolver is like yet another app depending on rocblas (including pytorch) having to be rebuilt N times as a dependency on rocblas being rebuilt N times. For example, see there's no rocm build loop in packages that don't depend on rocblas like the opencl runtime: https://src.fedoraproject.org/rpms/rocm-runtime/blob/rawhide/f/rocm-runtime.spec#_50 For the time being rocblas appears the origin of the rocm build loop caused by the link failure above, but if other rocm parts also share this same issue, once rocblas is fixed, the fix can trickle there too I assume. The core does not seem to have this issue and in fact it is being built only once as one would expect. I can open a new issue but we'd lose part of the context so for now I post it here. We can always open a new issue later. The goal is to fix the build so it doesn't fail linkage, once fixed it wouldn't add any noticeable feature to the software other than succeeding a build that previously failed as far as I can tell. Thank you! |
|
About the possible solutions mentioned, I agree that your -mcmodel= (I suppose "medium") suggestion for the short term appears the most attractive solution. In fact, wishful thinking, I wonder if there's a chance it could already be switched on through some environment variable. |
|
The compiler team is looking into adjusting the layout of the shared libraries. I'm not sure of the details of their proposal, but the gist of it is to move all the offload bundles to either the beginning or the end of the library, which will ensure that all host code is close together (and therefore not need more than 32-bit offsets). I'm not sure the status of that proposal, but it seemed promising.
The last time I checked, the mcmodel flag did not actually work when compiling HIP code. Although, perhaps that has changed.
The rocBLAS ABI does not change depending on the gfx architecture it was built for. Fedora has built rocBLAS for multiple architectures as separate packages, but you can build your application against any of those rocblas packages and it will work with all of them. I don't think they're rebuilding the rocSOLVER or pytorch libraries because of rocBLAS. It is likely because those libraries/packages also benefit from the same architecture-splitting that they did with rocBLAS. @Mystro256 or @trixirt might be able to shed more light on Fedora's choices here. |
|
We faced this problem last year getting started when I was first building rocBLAS for all the targets. There was discussion within Fedora about how to work around this and what we have in place is the solution. The builds are split along major family lines. This was done to keep the explanation of where your gpu's was in the split simple. So we have atm gfx8, gfx9, gfx10 and gfx11 in F40. The prefix for this is /usr/lib64/rocm/gfxXX . There is also a special set 'default' which is the union of gfx10 and gfx11, these install the normal prefix /usr. The main tradeoff that was made was to include as many targets as possible . This splitting does make it more challenging for packaging but that is why we have spec files and do work to make it happen in Fedora, with pytorch being an example of making it happen. Time to build is not a major concern. If/when something changes in the upstream, I and other folks in Fedora ROCm packaging sig will readdress how rocBLAS and similar are built to maximize inclusiveness of gpu targets. IMO this is a case of perfection being the enemy of good. |
|
One thing to note on this split package option is of course that any system with multiple gpus of different architectures will not be supported by your scheme. For this reason i would strongly recommend building as few versions of rocblas as possible, at most 2, to raise the chance of any given heterogeneous system being supported |
|
@IMbackK agreed: running different gfxN in the same app has become impossible as result of the 5 rocm builds with the finegrined split. That is on top of having to pick the right rpm and/or binary of the AI APP or it won't work even with one GPU. I asked to do a most 2 builds some time ago, so thanks for suggesting it too. The "Good" to me is that there's just 1 binary of every AI app (be it llama-cpp, vllm, mojo, triton, pytorch, etc..) and there's no multiplication x5 of both rpm and binaries of every AI/GPU app under the sun, so when the AI dev that normally runs on the popular, but proprietary, GPU compute stack tries his favorite AI app on Open Source ROCm, it just works without extra complications compared to the previous experience. Rejecting all suggestions above, some of which also allows to achieve full inclusiveness with a single build of rocblas (or at most 2 builds) to achieve minor extra optimizations, to me defines as the "Perfection". |
|
Hello,
It's hard to see how my app can work against all rocblas packages because it won't know where to find the file it needs at runtime, unless such knowledge is injected with "module load" just before the build, by rebuilding it N times with N different "module load" commands. In other words the reason of the app being rebuilt N times in the best case could be just path differences, as workaround that rocblas wasn't meant to be built N times and installed in N different places. rocblas rebuilt N times is a workaround around this rocblas build time link failure, and if you wish the app rebuilt N times are further orthogonal workarounds for the path differences caused by the first workaround, and the buildsystem takes one more hit at every step of the way, workaround on top of workaround. I would have been content to get away with just 1 app binary for testing, using ROCBLAS_TENSILE_LIBPATH (or some other hidden env variable I found randomly to try to force it find the right files for the needed rocblas). That didn't appear to work and it segfaulted. Possibly I did some mistake, maybe that could have worked if I insisted in that direction, but I don't think an user should be required to set ROCBLAS_TENSILE_LIBPATH to some directory by hand specific to the GPU in use, for the app not to spawns some error about missing files in /usr/ with no hint on how to resolve it. Among other suggestions to avoid the rebuild of all apps N times, before filing this issue, I tried to ask if the N rocblas builds could be installed in the same path location hoping then one app binary would just work. Problem is there's some file collisions, each build generates files in the same location that can't be automatically disambiguated. I had a quick look at the collisions of the non ELF-x86 parts and they didn't seem hard to disambiguate, but now I wonder if even /usr/lib64/librocblas.so (the otherwise >2G shared lib) would be one of them. I could imagine this path would be workable if you then accept not to be able to run on all gfx using the same container image (not just at runtime from the same binary among the N built, which as discussed earlier is already not possible). Thanks! |
This cant work, rocblas contains gpu code inside librocblas.so not just its modules and that gpu code must be available for all gpus in the system supported by the runtime. You can, however just build several librocblas.so with different prefixes, and then use LD_LIBRARY_PATH or LD_PRELOAD to select witch one, no application rebuild required. |
|
Hello, is this issue still being worked on? |
|
Yes we have have ongoing work on this topic, when anything relevant lands in develop branch I will comment here. Larger changes may not occur until a major release. gfx940 and gfx941 could be removed from the target list you build IMO. |
|
Just to keep you all in the loop, one possible solution has landed in develop commit bb81a83 in which we enable llvm clang's hip offload compiler option --offload-compress. See https://clang.llvm.org/docs/ClangCommandLineReference.html This may provide a significant (around 90%) size reduction with little impact on build time, so along with removing the gfx940 and gfx941 targets as suggested earlier, it will greatly reduce the linked .so size for develop branch. The compiler flag requires the ROCm 6.2 clang or later. We are still evaluating it, but if it passes all our requirements it should be used by default in a future release. |
Thanks for suggestion and it is simple to implement! |
|
The high compression ratio is not a big surprise given that there a really only 2 isas. i would expect all of gcn+cdna and all of rdna to generate each essentially the same code aside from edge cases. |
Describe the bug
Build fails during final shared lib linking.
To Reproduce
Steps to reproduce the behavor:
-DBUILD_FILE_REORG_BACKWARD_COMPATIBILITY=OFF
-DROCM_SYMLINK_LIBS=OFF
-DHIP_PLATFORM=amd
-DAMDGPU_TARGETS=${ROCM_GPUS}
-DCMAKE_INSTALL_LIBDIR=$ROCM_LIB
-DCMAKE_INSTALL_BINDIR=$ROCM_BIN
-DBUILD_WITH_TENSILE=ON
-DBUILD_WITH_PIP=OFF
Expected behavior
Build should not fail.
Log-files
Environment
Should not matter, it is not a runtime issue.
Additional context
Despite I don't see this reported among the github issues, this should be a very well known issues. So I wonder if this is planned not to be ever fixed?
If the above assumption is correct, I would like to know if upstream is willing to take in a fix for it, assuming a fix is possible.
The text was updated successfully, but these errors were encountered: