-
-
Notifications
You must be signed in to change notification settings - Fork 563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dockerfile (alpine based) + some quick ideas/suggestions #636
Comments
@roscopecoltran this sounds like an awesome plan! Having a Dockerfile would be great indeed. Some other notes OTH:
|
Hey, Thanks for the reply ! It would be awesome to have a flow based processing for starting concurrent tasks on the code. defined in some human readable yaml files ^^. Also, for any meta based additional info, about a project, I would recommend to use searx as it can flexible on the scope of metasearch to gather for public/web related info, if u want to build a context around the code audit. Have a good week-end :-) Cheers, |
Bonjour Philippe, :-) Hope you are all well ! I am bundling scancode into a docker alpine container but some errors occured. I think that some tweaks are required while running the configure script. Mainly, it would be great to use fallbacks to find the library outside the scancode project, like operating a search in standard locations, and keeping the musl order, eg. paths = ['/lib', '/usr/local/lib', '/usr/lib'], or maybe to trigger a build event if the pre-compiled lib failed to be loaded. In our case, I add such error message while trying to link the libmagic2.so shared library, the configure scripts fails with "__snprintf_chk: symbol not found"; probably due to musl-dev or glibc). From my understanding, these functions are used to find and map the pre-compiled libmagic shared lib.
The easier solution would be to use an ubuntu based container, but the size of Ubuntu container is such a none-sense for me: ❯ docker images | awk '{print $1"\t"$2"\t"$7" "$8}'
REPOSITORY TAG SIZE
sample_alpine latest 167.6 MB
sample_ubuntu latest 447.8 MB
<none> <none> 187.9 MB
ubuntu latest 187.9 MB
redis latest 109.3 MB
mongo latest 261.6 MB
golang latest 709.5 MB
alpine latest 5.249 MB Refs used:
Waiting for your input/point of view on that question. :-) Cheers, |
@roscopecoltran this is effectively a todo: the prebuilt binaries need to be unbundled ... this is tracked in #469 |
Here's a simple
Source can also be found here: And the container here: Here's how to build it with a version tag:
|
Hmm, while a simple
…a real scan doesn't work yet. I'm running into (apparently) the same issue as @roscopecoltran :
I will do some more research. |
I've created a 2nd Stuff is over here: |
@znerd Thanks you for figuring all this out and sorry for some of the troubles: keep me posted as you progress! At the moment, an Alpine container would require a bit of manual work as we there are some bundled, pre-built binaries that would need to be rebuilt by hand for this to work with an Alpine-style static build context. Things should work fine with Debian/Ubuntu/CentOS/Fedora/Suse and similar Here are two other examples: https://github.com/clearlydefined/tool-images/blob/0392771a408dbfb2ab5fcd88f702c43f207aa4ce/scancode/Dockerfile I am not sure which version of Scancode you used but I would suggest using the latest 2.9.x pre v3 releases. |
I'm currently experimenting with a minimal
But building the image gives
@pombredanne, any idea what the issue is? |
@sschuberth sorry for the late reply... Could you try to use a release archive with this Alpine setup instead to eliminate some moving parts? Just our if curiosity is using Alpine really proving such big benefits for the pain it brings? |
What are the criteria by which glibc is matched?
Will try.
Well, actually ScanCode is the first project I run into which resists to easily run on Alpine. But yes, image-size wise I believe it's worth the effort. |
I have no idea :D that said, it (making things work on Alpine) could be something that could may be dealt with in the GSoC project of @aj4ayushjain ? |
@pombredanne, so when installing from source while building the Alpine-Docker-image I get
Not really much more telling about what exactly is the issue why |
@sschuberth |
Ok, this
But now it still wants to build
|
@aj4ayushjain, that basically what I had been asking for in #1262, but it turns out to be quite laborious, and as mentioned here I by now believe the better approach is to use an Alpine Docker image that has https://github.com/sgerrand/alpine-pkg-glibc installed, which is what I'm trying right now. |
I tried installing alpine-pkg-glibc and it didn't work for me were you able to get this to work? .. still getting this error Traceback (most recent call last): |
@earlyster sorry for that! @sschuberth if you have an Alpine Dockerfile taht works, we can make this part of the repo here alright. @earlyster also the work that @aj4ayushjain is doing on packaging in general (and debian in particular) will likely help here too as we would possibly have a clearer way to get things from Alpine packages. |
Thanks @pombredanne for the update! |
IIRC the last version of my Alpine-based Dockerfile has the same |
@sschuberth ok so we are in the same boat .. not able to use scantool with alpine based docker image. |
Trying to run on alpine but it's not working. Any updates on this? |
I finally managed to create an Alpine-based Docker image with Python 3.6 that is able to run ScanCode. Feel free to give it a try: https://github.com/sschuberth/docker-files/blob/4ebd681dfef5a8142b92c1157edd4f5495f0706b/scancode/Dockerfile |
@sschuberth you rock!
... |
That depends on your definition of "trusted" 😉 I was also a bit reluctant at first as it's "just some random single user" (and not e.g. a company / foundation) maintaining these images. But @frolvlad seems to be very active in the Docker / Alpine community and quite a few people seem to be using his images. So I decided for myself to trust these base images. |
It feels a tad engaged to me if I dig a little:
Just curious, how big a startup speed gain and size gain do you get with this? |
I haven't bothered to measure this so far TBH... |
FWIW, there is a new support for Alpine musl in the Python wheel manylinux including support on PyPI. This could pave the way to support Alpine images. |
Oh, wow, I am quite amused to find these types of topics and articles throughout the internet using my Alpine-baked images here and there ( |
Hi guys,
Hope you are all well !
I found your great repo while I was working on a personal project aiming to do, also, some code analysis with a distributed bot, working as a virtual agent to build a semi-structured database of meta informations about some open source projects (mainly SQL and graph based).
Context:
The goal is to enrich automatically, with a virtual assistant/bot, some of my starred repos by 'left joining', or apply some graphql queries, on some data like, known frameworks detection, github stats or github trends, in order to extract more meta data from some of my starred repositories.
In a nutshell, it is all about to have a more convenient local search engines on my starred repos, create a domain specific topic modeling api. By detecting some patterns, I want to manage a dynamic tree of events that could be dedicated to some sub-tasks like to generate Docker/Docker-Compose files from a database a snippets if some dependencies are detected or matched.
examples of pattern detection to build dynamic dockerfiles:
1. Deploy local instances with Docker/Docker Compose
For some tasks, it sounds clear that some content lexer, grep, parser would provide faster responses and results; dependencies scanning, licences; so that s how I found your projects, and that really cool :-)
So first of all, as a dev ops ^^, I was wondering if it would not be easier to bundle scancode into some docker/docker-compose files, of course alpine based in order to keep the size of containers reasonable.
2. Some suggestions/features
So far, I started to give a try to your projects, and it sounds that I will create a fork with some personal ideas/features, close to want the context mentioned above, and wanted to share it with you, so any feedback or experience sharing is tremendously welcomed :-)
- Administration panel
- Topic modelling
- Code parsing/search
-Starred Github managers:
3. Questions related to your roadmap:
Last question, do you plan to migrate to Python3 your stack of scripts ? Is it something that you have on the roadmap and would like to have a community efforts ?
Ps. sorry for the long post, but I was really inspired by your work ^^, so thanks for reading it all :-)
Please have a great day !
Cheers,
Richard
The text was updated successfully, but these errors were encountered: