Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConanFile for C++ deployments #15363

Closed
luckielordie opened this issue May 14, 2018 · 31 comments
Closed

ConanFile for C++ deployments #15363

luckielordie opened this issue May 14, 2018 · 31 comments
Assignees
Labels
area/packaging & distribution disposition/help wanted Maintainers do not have enough resources to allocate to this at the moment. Help is appreciated! kind/enhancement lang/c++ priority/P3

Comments

@luckielordie
Copy link

What version of gRPC and what language are you using?

C++

What operating system (Linux, Windows, …) and version?

All

What runtime / compiler are you using (e.g. python version or version of gcc)

All

Hi Guys,
What is the support for creating a conanfile.py for the grpc project? Would it be something that would be considered for inclusion into the repo? Conan is a C++ package manager and I would like to test the waters for an official grpc conanfile.

Thanks,
Matt

@jtattermusch
Copy link
Contributor

We've been thinking of adding support for Conan in the past (as our C++ installation story is not ideal) but never actually go to it - help would be really welcome here.
A few questions:

  • would this support all of linux, windows and mac out of the box? (I think that's a requirement)
  • do conan packages exist for all our dependencies? (if not, what would we do about this?)
  • is there already some "unofficial" conan package for grpc we could test?

@Ozaq
Copy link

Ozaq commented May 15, 2018

@jtattermusch I am currently trying to create a conanfile for grpc (so far only for internal use) , I found most dependencies are available:

zlib/1.2.11@conan/stable
OpenSSL/1.0.2o@conan/stable
protobuf/3.5.1@bincrafters/stable
c-ares/1.14.0@conan/stable
gflags/2.2.1@bincrafters/stable

For a full build with all dependencies coming in as provider=package only google benchmark is missing. So I managed to get this working on windows and only the linux build is giving me trouble when creating a debug build. In this case libprotobuf.a is not found because it is created as libprotobufd.a

EDIT: configuring grpc build to find protobuf in CONFIG mode does the trick for debug builds.

In case you are interested here are my conan files for windows/linux (untested on osx)

For benchmark:

from conans import ConanFile, CMake, tools
import os


class GoogleBenchmarkConan(ConanFile):
    name = "benchmark"
    version = "1.4.0"
    folder = "benchmark-%s" % version
    description = "Google micro benchmarking tools."
    license = "Apache-2.0"
    settings = "os", "compiler", "build_type", "arch"
    generators = "cmake"
    source_subfolder = "src"
    bulild_policy = "missing"

    def source(self):
        tools.get("https://github.com/google/benchmark/archive/v{}.zip".format(self.version))
        os.rename(self.folder, self.source_subfolder)

    def build_cmake_prefix_path(self, cmake, *paths):
        prefix = cmake.definitions["CMAKE_PREFIX_PATH"] = ";".join(paths)

    def build(self):
        cmake = CMake(self)
        cmake.definitions["BENCHMARK_ENABLE_TESTING"] = "OFF"
        cmake.definitions["CMAKE_INSTALL_PREFIX"] = "install"
        cmake.configure(source_folder=self.source_subfolder)
        cmake.build()
        cmake.install()

    def package(self):
        self.copy("*", dst="include", src="install/include")
        self.copy("*", dst="lib", src="install/lib")
        self.copy("*", dst="bin", src="install/bin")

    def package_info(self):
        self.cpp_info.libs = ["benchmark"]

For grpc:

from conans import ConanFile, CMake, tools
import os


class gRPCConan(ConanFile):
    name = "gRPC"
    version = "1.11.1"
    folder = "grpc-%s" % version
    description = "Google's RPC library and framework."
    license = "Apache-2.0"
    requires = ("zlib/1.2.11@conan/stable", "OpenSSL/1.0.2o@conan/stable",
               "protobuf/3.5.1@bincrafters/stable", "c-ares/1.14.0@conan/stable")
    build_requires = "gflags/2.2.1@bincrafters/stable", "benchmark/1.4.0@MY_LOCAL_REGISTRY/stable"
    settings = "os", "compiler", "build_type", "arch"
    generators = "cmake"
    short_paths = True  # Otherwise some folders go out of the 260 chars path length scope rapidly (on windows)
    source_subfolder = "src"
    default_options = "protobuf:static_rt=False"
    bulild_policy = "missing"

    def source(self):
        tools.get("https://github.com/grpc/grpc/archive/v{}.zip".format(self.version))
        os.rename("grpc-{}".format(self.version), self.source_subfolder)

    def build_cmake_prefix_path(self, cmake, *paths):
        prefix = cmake.definitions["CMAKE_PREFIX_PATH"] = ";".join(paths)

    def build(self):
        cmake = CMake(self)
        cmake.definitions["gRPC_INSTALL"] = "ON"
        cmake.definitions["gRPC_BUILD_TESTS"] = "OFF"
        cmake.definitions["gRPC_PROTOBUF_PROVIDER"] = "package"
        cmake.definitions["gRPC_ZLIB_PROVIDER"] = "package"
        cmake.definitions["gRPC_CARES_PROVIDER"] = "package"
        cmake.definitions["gRPC_SSL_PROVIDER"] = "package"
        cmake.definitions["gRPC_GFLAGS_PROVIDER"] = "package"
        cmake.definitions["gRPC_BENCHMARK_PROVIDER"] = "package"
        cmake.definitions["gRPC_PROTOBUF_PACKAGE_TYPE"] = "CONFIG"
        cmake.definitions["CMAKE_INSTALL_PREFIX"] = "install"
        cmake.definitions["ZLIB_ROOT"] = self.deps_cpp_info["zlib"].rootpath
        cmake.definitions["OPENSSL_ROOT"] = self.deps_cpp_info["OpenSSL"].rootpath
        self.build_cmake_prefix_path(cmake,
            self.deps_cpp_info["c-ares"].rootpath,
            self.deps_cpp_info["protobuf"].rootpath,
            self.deps_cpp_info["gflags"].rootpath,
            self.deps_cpp_info["benchmark"].rootpath,
        )
        cmake.configure(source_folder=self.source_subfolder)
        cmake.build()
        cmake.install()

    def package(self):
        self.copy("*", dst="include", src="install/include")
        self.copy("*", dst="lib", src="install/lib")
        self.copy("*", dst="bin", src="install/bin")

    def package_info(self):
        self.cpp_info.libs = [
            "address_sorting",
            "gpr",
            "grpc",
            "grpc_cronet",
            "grpc_csharp_ext",
            "grpc_plugin_support",
            "grpc_unsecure",
            "grpc++",
            "grpc++_cronet",
            "grpc++_error_details",
            "grpc++_reflection",
            "grpc++_unsecure"]

Note that I am publishing the benchmark package to a local artifactory so you need to substitute your own for 'MY_LOCAL_REGISTRY'

@luckielordie
Copy link
Author

This is exactly what I was hoping for! Thankyou @Ozaq !

@Croydon
Copy link

Croydon commented Jul 12, 2018

We have our own gRPC Conan package for a while now, but upstream support would be obviously the best https://github.com/inexorgame/conan-grpc/

@Croydon
Copy link

Croydon commented Jul 12, 2018

For the purpose of creating a good gRPC Conan recipe I have also opened #15949.

@jtattermusch
Copy link
Contributor

@Croydon thanks for sharing the conan recipe.

@Croydon
Copy link

Croydon commented Jul 26, 2018

Todo

Hard blockers:

Nice to have:

I will keep this list up-to-date till this is done.

@michaelmaguire
Copy link

michaelmaguire commented Oct 23, 2018

Is there a reason we don't use the Conan tools collect_libs function:

    self.cpp_info.libs = tools.collect_libs(self)

instead of manually hard-coding the list of libraries?

When I use that against v1.15.1 it finds:

address_sorting gpr grpc++ grpc++_cronet grpc++_error_details grpc++_reflection grpc++_unsecure grpc grpcpp_channelz grpc_cronet grpc_plugin_support grpc_unsecure

which looks equivalent and seems far less brittle

@Croydon
Copy link

Croydon commented Oct 23, 2018

tools.collect_libs(self) is only useful when the order of the linking doesn't matter. When someone could help me with that then I will happily complete this https://github.com/inexorgame/conan-grpc/blob/88908f9d7c99e074233f3c5b6f74e12dc3c450da/conanfile.py#L127

@EkremH
Copy link

EkremH commented Dec 28, 2018

I'm also very interested in a Conan package for GRPC, need this for my current project.

@jtattermusch
Copy link
Contributor

jtattermusch commented Jan 2, 2019

@EkremH looks like you should be able to use the community-provided conan package here:
https://github.com/inexorgame/conan-grpc/blob/testing/1.17.2/conanfile.py

I haven't tried the package myself but it seems like others have been using it.
@Croydon is there any way to see the usage statistics for the conan package you created? If it is used successfully by users and it works for them, we'd be happy to provide some visibility and invite the community to contribute. Would you be interested in writing a blogpost (using gRPC with Conan) for the grpc.io website?

@Croydon
Copy link

Croydon commented Jan 6, 2019

is there any way to see the usage statistics for the conan package you created?

Bintray logs downloads, but this is in no way representative for real interest in a grpc Conan recipe, since

  • it is right now only within the repository of Inexor and not conan-center or @bincrafters, so it has very little exposure and discoverability
    • since you have shown interest in Conan support I don't like to move it to bincrafters (even though I'm also part of this community, which maintains ~400 Conan recipes) and I'm definitely not trying to bring it into conan-center from Inexor, otherwise we would need to get through a deprecation process and migrate users as soon as grpc has official Conan support
    • to summarize: my wish is that we can get the grpc recipe mature within the boundaries of Inexor and then getting it into grpc itself
  • it is measuring downloads from Bintray, meaning if somebody uses it from GitHub it won't be noticed. Similar to when someone is copying it to a own Bintray Conan repository, which many people are doing - especially companies who want to control exactly what is getting into their build systems
  • when our grpc recipe is consumed from Bintray by CI jobs then we can have hundreds of downloads within no-time, but that still doesn't represents users and even less user interest

With that being said, statistics are directly visible on the package: https://bintray.com/inexorgame/inexor-conan/grpc%3Ainexorgame#statistics


Would you be interested in writing a blogpost (using gRPC with Conan) for the grpc.io website?

I'm definitely interested in doing this. However, only after Protobuf/protoc_installer landed in conan-center (see todo list), before that we would fight at to many places at the same time I guess.

With that being said I still hope for some early tester and contributors.
There are currently three issues which are non-Conan specific and could be solved/answered by some people who have more insight into grpc than me. Namely:

I have updated the todo list above accordingly.

Thanks! 😄

@Siron777
Copy link

Siron777 commented Jan 8, 2019

At the request of Croydon (from the issue #17438), the recipe we are using in my company is the following one:

from conans import ConanFile, CMake, tools
import os,sys;

class gRPCConan(ConanFile):
    name = "gRPC"
    version = "1.16.0"
    license = "MIT"
    url = "https://grpc.io/"
    description = "gRPC"
    settings = "os", "compiler", "build_type", "arch"
    options =   {   
                    "withOpenSSL" : ["ON", "OFF"]
                }
    default_options = "withOpenSSL=ON"
    generators = "cmake"

    scm =   {
                "type": "git",
                "subfolder": "gRPC_v%s" % version,
                "url": "https://github.com/grpc/grpc",
                "revision": "v%s" % version,
                "submodule" : "recursive"
            }

    def build_requirements(self):
        if (self.options.withOpenSSL == "ON"):            
            self.build_requires("OpenSSL/1.1.1@AI/extern")
        self.build_requires("protobuf/3.6.1@AI/extern")

    def requirements(self):
        self.requires("zlib/1.2.11@AI/extern")
        self.requires("c-ares/1.14.0@AI/extern")
        
    def build_cmake_prefix_path(self, cmake, *paths):
        prefix = cmake.definitions["CMAKE_PREFIX_PATH"] = ";".join(paths)
        
    def build(self):
                        
        cmake = CMake(self, set_cmake_flags=True)    
        
        if (self.options.withOpenSSL):
            cmake.definitions["gRPC_SSL_PROVIDER"]="package"
            cmake.definitions["OPENSSL_ROOT_DIR"] = self.deps_cpp_info["OpenSSL"].rootpath 
            
        cmake.definitions["gRPC_INSTALL"] = "ON"
        cmake.definitions["gRPC_BUILD_TESTS"] = "OFF"
        cmake.definitions["gRPC_PROTOBUF_PROVIDER"] = "package"
        cmake.definitions["gRPC_ZLIB_PROVIDER"] = "package"
        cmake.definitions["gRPC_CARES_PROVIDER"] = "package"
        cmake.definitions["gRPC_PROTOBUF_PACKAGE_TYPE"] = "CONFIG"
        cmake.definitions["gRPC_GFLAGS_PROVIDER"] = "module"
        cmake.definitions["ZLIB_ROOT"] = self.deps_cpp_info["zlib"].rootpath
        self.build_cmake_prefix_path(cmake,
            self.deps_cpp_info["c-ares"].rootpath,
            self.deps_cpp_info["protobuf"].rootpath
        )
        
        argList = list()      
       
        cmake.configure(source_folder="%s" % self.scm["subfolder"], args=argList)
        cmake.build()

    def package(self):    
        cmake = CMake(self, set_cmake_flags=True)
        cmake.install()

    def package_info(self):
        self.cpp_info.libdirs = ['lib']
        self.cpp_info.includedirs = ['include']
        self.cpp_info.bindirs = ["bin"] 
        self.env_info.path.append(os.path.join(self.package_folder, 'bin'))
        self.env_info.path.append(os.path.join(self.package_folder, 'lib'))        
        self.cpp_info.builddirs = ["cmake", "bin", "lib/cmake/grpc"]     
        self.cpp_info.libs = [
            "address_sorting",
            "gpr",
            "grpc",
            "grpc_cronet",
            "grpc_csharp_ext",
            "grpc_plugin_support",
            "grpc_unsecure",
            "grpc++",
            "grpc++_cronet",
            "grpc++_error_details",
            "grpc++_reflection",
            "grpc++_unsecure"]

I think it was inspired from the one of bincrafter.

@Croydon
Copy link

Croydon commented Jan 8, 2019

@Siron777 Thanks for sharing!

I think it was inspired from the one of bincrafter.

@bincrafters don't have a grpc recipe. I'm part of Bincrafters, but we will go with the strategy that we want to bring it to a mature state at the Inexor repository and then we would like to see official support upstream 😄

Please consider contributing to the recipe

@stale
Copy link

stale bot commented Oct 14, 2019

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 180 days. It will be closed automatically if no further update occurs in 1 day. Thank you for your contributions!

@Croydon
Copy link

Croydon commented Oct 15, 2019

Still work in progress.

I'm still maintaining the (unstable) grpc recipe and will eventually push it in the Conan Center Index

@Firefly35
Copy link

Hi everyone, is there any date foreseen when a stable grpc package could be available in conan-center ?

@jtattermusch
Copy link
Contributor

CC @zackgalbreath @KyleFromKitware

@Croydon
Copy link

Croydon commented Dec 11, 2019

The progress of getting packages into Conan Center has changed in the meantime

There is now the Conan Center Index.
Even packages who are in Conan Center via the old linking should be re-added and there are new criteria packages need to fulfill.

Future work on grpc is stalled until Protobuf and Protoc are added to the Conan Center Index.

Until grpc is ready for the Conan Center index, the recipe is getting maintained at https://github.com/inexorgame/conan-grpc/

@Firefly35 but no there is no date yet

@Firefly35
Copy link

Firefly35 commented Dec 11, 2019 via email

@KyleFromKitware
Copy link
Contributor

@Croydon I just tried out your gRPC Conan recipe with mixed success. It successfully built but I got the following error when the test ran:

grpc/1.25.0@inexorgame/stable (test package): Running test()
14: failed to connect to all addresses
Greeter received: RPC failed

@Croydon
Copy link

Croydon commented Dec 18, 2019

@KyleFromKitware While I agree that this result is super confusing and should be improved, this is actually the expected result for now. Conan test_packages don't need to test the functionality of the library, only if the library was build successfully. The current test_package covers this by running grpc

@jtattermusch
Copy link
Contributor

@KyleFromKitware While I agree that this result is super confusing and should be improved, this is actually the expected result for now. Conan test_packages don't need to test the functionality of the library, only if the library was build successfully. The current test_package covers this by running grpc

May I ask how can a test failure be expected result? Is the current "test" basically a placeholder for some actual tests? If so, can we just use a placeholder test that passes (it should be easy to come up with a simple program that uses grpc but actually passes).

@KyleFromKitware
Copy link
Contributor

May I ask how can a test failure be expected result?

Upon further digging, it looks like their test is running a client without running the corresponding server. So the test builds successfully, and tries to connect to localhost, but obviously can't because localhost isn't listening.

The fix would be to either run the server locally, or silence the error, either in the client program or at the Conan level by simply not running the executable.

@KyleFromKitware
Copy link
Contributor

@jtattermusch @Croydon I've opened inexorgame-obsolete/conan-grpc#24 to run the server along with the client. The package test now successfully completes on my machine.

@stale
Copy link

stale bot commented May 6, 2020

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 30 days. It will be closed automatically if no further update occurs in 7 day. Thank you for your contributions!

@jtattermusch jtattermusch added the disposition/help wanted Maintainers do not have enough resources to allocate to this at the moment. Help is appreciated! label May 6, 2020
@stale stale bot removed the disposition/stale label May 6, 2020
@jtattermusch
Copy link
Contributor

We acknowledge this to be a valid feature request but since no one has come forward to implement this, we are going to close it for now while maintaining the label/Help wanted to indicate that this can be picked up as future work

@Croydon
Copy link

Croydon commented May 6, 2020

I'm still on it as previously mentioned (I did not joke about it 😉)

It is just really, really hard and still a work-in-progress, but we close to land protoc in the Conan Center Index and Conan has landed new features that we need to continue working on an usable abseil and grpc recipe.

Closing this issue might still be okay as a Conan recipe will go now to the Conan Center Index.

For people who might want to get notificated on availability can track conan-io/wishlist#55

In the meantime, there is always the current state available at https://github.com/inexorgame/conan-grpc. The last more-or-less stable version is 1.25.0 currently. Somethings are broken, but it might still be useful to some people.

@michaelmaguire
Copy link

michaelmaguire commented Feb 15, 2021

I still don't see this recipe in https://github.com/conan-io/conan-center-index -- in the end did it ever make it?

I see conan-io/wishlist#55 (comment) is still open.

@chausner
Copy link

I still don't see this recipe in https://github.com/conan-io/conan-center-index -- in the end did it ever make it?

Apparently not. Would also love to see this in ConanCenter, ideally kept up to date by the maintainers. Until then https://github.com/inexorgame/conan-grpc looks to be a great alternative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/packaging & distribution disposition/help wanted Maintainers do not have enough resources to allocate to this at the moment. Help is appreciated! kind/enhancement lang/c++ priority/P3
Projects
None yet
Development

No branches or pull requests