Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Documentation to use docker #35

Open
5 of 7 tasks
remicres opened this issue Nov 24, 2020 · 37 comments · Fixed by #54 or #55
Open
5 of 7 tasks

Documentation to use docker #35

remicres opened this issue Nov 24, 2020 · 37 comments · Fixed by #54 or #55
Labels
docker Questions related to the use of otbtf docker images Documentation Request for more documentation
Milestone
Enhance documenta...

Comments

@remicres
Copy link
Owner

remicres commented Nov 24, 2020 •
edited
Loading

More documentation related to the use of docker should ease greatly user experience!
Here is a list of stuff that could be very useful to put in the documentation:

  • how to use OTBTF docker images on the following systems:
    • linux
    • windows,
    • mac
  • how to mount volumes in docker
  • how to use GPUs with nvidia-docker
    • linux
    • windows
  • how to update the OTB version inside the docker image, how to recompile it, add external remote modules

There are some resources online like here but it could be nice to have it all in the repository!
@remicres remicres added enhancement New feature or request docker Questions related to the use of otbtf docker images labels Nov 24, 2020
@viniciuspg
Copy link

Hi Remi

I wrote a message on Researchgate because I couldn't find a way to contact you here. Well, I would like to know if it is already available a way to use OTBTF on win10, and if there is any tutorial that I can follow because unfortunately I can't install dual boot on my notebook (acer predator), because I run the risk of losing windows. As I use the computer for regular work, I can't take this risk. In this sense, it would be very good if I could install via docker, since I already use ODM this way. Would this be possible?

@daspk04
Copy link
Contributor

daspk04 commented Apr 17, 2021 •
edited
Loading

Hello @viniciuspg,

The OTBTF can be installed via docker on win10 via Docker Desktop. You can install the CPU version on windows 10 via Docker. Just docker pull any OTBTF (CPU) image from (https://gitlab.com/latelescop/docker/otbtf/container_registry/) and it should work. For example docker pull mdl4eo/otbtf2.4:cpu.

At present, I am using OTBTF with GPU in windows 10 using WSL2. In case you want to use WSL2 with CUDA (GPU enabled) you need to have NVIDIA GPU(which you already have I guess based on your notebook) and have to install the latest Windows Insider version from the Dev Preview ring. https://blogs.windows.com/windows-insider/2020/06/15/introducing-windows-insider-channels/

How to install WSL2 with Cuda on windows 10
https://docs.nvidia.com/cuda/wsl-user-guide/index.html
https://docs.docker.com/docker-for-windows/wsl/#gpu-support

@viniciuspg
Copy link

viniciuspg commented Apr 18, 2021 via email

@remicres
Copy link
Owner Author

remicres commented Apr 18, 2021 •
edited
Loading

Hi @viniciuspg ,

From what I understand, you should have installed the CUDA drivers for WSL (downloaded from here), in Windows.
You should also have installed successfully WSL2 in windows, and Ubuntu 20.04.

So now, you run Ubuntu. The Ubuntu command prompt appears.

  • In the Ubuntu command prompt, you have to install the CUDA ToolKit (not the driver!*) as explained here
  • Still in the Ubuntu command prompt, you can then install the nvidia-docker, as explained here
  • Finally, still in the Ubuntu command prompt, you will be able to use the OTBTF docker with your GPU : docker run -it --gpus all mdl4eo/otbtf2.4:gpu otbcli_PatchesExtraction (you should see the help of the PatchesExtraction OTB Application). You can also enter the docker image with docker run -it --gpus all mdl4eo/otbtf2.4:gpu bash then interactively run your command lines. To mount some file systems, you should be able to use the -v option (after docker run)

Is that working for you?

@viniciuspg
Copy link

viniciuspg commented Apr 18, 2021 via email

@viniciuspg
Copy link

viniciuspg commented Apr 19, 2021 via email

@remicres
Copy link
Owner Author

Hi @viniciuspg ,

I think that to use the CPU only is more straightforward, and as @Pratyush1991 mentioned you can do that using the Docker Desktop.

I am sorry if my instructions weren't right. We need some "bulletproof" instructions to use OTBTF docker image + WSL2 + NVIDIA GPU, and I am sure we will have that soon in the documentation section.
The docker/wsl2/nvidia thing looks evolving greatly in Windows, but unfortunately it looks like some issues are still remaining ahead...

You can give a try with CPU (it should be ok to begin with the first tutorials), and in the meantime keep an eye open here, where we will add some documentation for windows soon!

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021

@remicres I just sent you an email, there is something wrong with your r2.4 build on dockerhub, no way to import tf (core dump)...

@remicres
Copy link
Owner Author

😱

I think that I will push your docker images on dockerhub...

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021

I guess you should I you don't want to rebuild TF once again...
Why did you build with branch r2.4 instead of default v2.4.1 ? It's probably an unstable branch they're using to dev...

Also, I could send you the content of my bazel remote cache dir via sftp or something (<10Go when compressed), so you can rebuild 2.4.1 in a matter of 10mn !

@remicres
Copy link
Owner Author

I think that I pushed the wrong images...

My local image all work fine, with tf 2.4.1. I will try to re-push the image on dockerhub

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021 •
edited
Loading

Are you sure about 2.4.1 ? Because on Saturday you sent me this command by email :
docker build --network='host' -t otbtf:cpu --build-arg BASE_IMG=ubuntu:20.04 --build-arg TF=r2.4 --build-arg NUMPY_SPEC===1.17.4 .

@remicres
Copy link
Owner Author 

cresson@cin-mo-gpu:~/decloud$ docker run -ti mdl4eo/otbtf2.4:cpu bash -c "python -c 'import tensorflow; print(tensorflow.__version__)'"
2.4.1

I don't understand why the image I push on dockerhub is not the right one!

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021

Indeed, for me this command prints nothing !
May be you messed up your tags ?

@remicres
Copy link
Owner Author

Weird. It looks like I pushed the good images. But I am unable to use them on my (intel) laptop, even though they are running fine on the gpu servers.

I will push your images on dockerhub for now...

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021

Interesting, could you show me your lscpu output ?
It could be related to bazel --config=opt, may be you should avoid this to make sure your build is portable.

@remicres
Copy link
Owner Author

Server 1 (working):

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz
Stepping:            4
CPU MHz:             2731.387
BogoMIPS:            7200.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            16896K
NUMA node0 CPU(s):   0,2,4,6
NUMA node1 CPU(s):   1,3,5,7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d

Server 2 (working):

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz
Stepping:              4
CPU MHz:               1480.289
CPU max MHz:           4500,0000
CPU min MHz:           1200,0000
BogoMIPS:              7391.83
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              11264K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d

Laptop (not working):

Architecture :                          x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Boutisme :                              Little Endian
Address sizes:                          39 bits physical, 48 bits virtual
Processeur(s) :                         8
Liste de processeur(s) en ligne :       0-7
Thread(s) par cœur :                    2
Cœur(s) par socket :                    4
Socket(s) :                             1
Nœud(s) NUMA :                          1
Identifiant constructeur :              GenuineIntel
Famille de processeur :                 6
Modèle :                                158
Nom de modèle :                         Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Révision :                              9
Vitesse du processeur en MHz :          1540.825
Vitesse maximale du processeur en MHz : 3900,0000
Vitesse minimale du processeur en MHz : 800,0000
BogoMIPS :                              5799.77
Virtualisation :                        VT-x
Cache L1d :                             128 KiB
Cache L1i :                             128 KiB
Cache L2 :                              1 MiB
Cache L3 :                              8 MiB
Nœud NUMA 0 de processeur(s) :          0-7
Vulnerability Itlb multihit:            KVM: Mitigation: Split huge pages
Vulnerability L1tf:                     Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:                 Mitigation; PTI
Vulnerability Spec store bypass:        Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:               Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:               Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:                    Mitigation; Microcode
Vulnerability Tsx async abort:          Mitigation; Clear CPU buffers; SMT vulnerable
Drapaux :                               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_ti
                                        mer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021

Strange, there's SSE4, AVX2 support everywhere so it shouldn't be an issue.
I tried with your latest push and it seems ok (our OVH server is also Intel Xeon), which one is it ?

I did not build the MKL version !

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021 •
edited
Loading

Oh I see, may be it is related to AVX512 instruction, which my Intel Xeon does not have, same as your laptop.

@remicres
Copy link
Owner Author

Thanks for this analysis... so you were right about bazel optimization flags.

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021

There's a dedicated package for AVX512, so this would also match with my explantation, it seems this instruction isn't compatible with regular CPUs...
https://pypi.org/project/intel-tensorflow-avx512/

@remicres
Copy link
Owner Author

So for building docker images that work on most computers, better sticking to basic optimization flags...

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021 •
edited
Loading

Yes, you need to remove either --config=opt from BZL_CONFIGS (should be enough) or -march=native from CC_OPT_FLAGS in build-env-tf.txt

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021 •
edited
Loading

OR you could remove -march=native and add regular opt like --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 which should work with any recent hardware... I do believe it's only a problem for AVX512, but SSE4.2 and AVX2 could also be a problem with old CPUs

@remicres
Copy link
Owner Author

Thanks.

This was a bit off topic, but clearly deserved some explanation.
And one less issue for windows users...

@remicres remicres linked a pull request Apr 19, 2021 that will close this issue
Docker : small changes (NUMPY_SPEC) + comments about CPU opt instructions #54
Merged
@viniciuspg
Copy link

viniciuspg commented Apr 19, 2021 via email

@remicres remicres reopened this Apr 19, 2021
@remicres
Copy link
Owner Author

remicres commented Apr 19, 2021 •
edited
Loading

Hi @viniciuspg ,

I can't see your screen capture 😮

I never tried docker desktop myself, but it looks like you can use the Windows PowerShell to run docker images (source).
Open a PowerShell terminal and type docker run -it mdl4eo/otbtf2.4:cpu bash to enter the docker in interactive mode.

@vidlb
Copy link
Contributor

vidlb commented Apr 19, 2021 •
edited
Loading

Hi @viniciuspg ,
You can use either PowerShell or cmd.exe to execute docker pull, create or run commands.
Then I believe the easiest way would be to create a persistent container, which you can access later from the Docker Desktop GUI (it's nicer than CMD...)
Commands would be as follow :
docker create --name otbtf-cpu --interactive --tty mdl4eo/otbtf2.4:cpu

Then you could start an interactive shell with
docker start -i otbtf-cpu

Or just use the GUI in Docker Desktop to start a shell, the container you just created should now appear in the GUI main page !

@viniciuspg
Copy link

Sorry Remi,

I replied directly in the email and not here on the forum, so I believe the image did not go up. I uploaded it below, just so you and others can see what it is about. I believe it is now visible.

I will try the power shell approach to see if I can move forward.

Att,
Vinicius

docker_desktop_command_line

@viniciuspg
Copy link

viniciuspg commented Apr 19, 2021 •
edited
Loading

Thank you very much friends!

With this docker create --name otbtf-cpu --interactive --tty mdl4eo/otbtf2.4:cpu command in Powershell I believe the implementation was accomplished. I believe this will help many other colleagues, because I just copied and pasted this line into Powershell and the magic happened. The OTBTF container appeared in the docker GUI. After that, I just clicked on the play button, and it turned green, meaning that it is running, and then clicking on the container and on the "CLI" icon (see print), a window opened, which apparently is where I should type the commands for using OTBTF, based on what was described by Pratyush1991 at #10, correct? I will try with these instructions and as described in Remi's book "Deep Learning for Remote Sensing Images with Open Source Software".

Att,
Vinicius
1powershell
2docker_desktop_OTBTF_CREATED
3docker_desktop_OTBTF_RUNNING
4docker_desktop_OTBTF_COMMAND_LINE_WINDOW

@remicres
Copy link
Owner Author

That is great!

I will push some documentation in a few minutes.

It could be great to add a section to help windows user.
Can you tell me if that is correct:

  1. install docker desktop
  2. open a PowerShell terminal, and type docker create --name otbtf-cpu --interactive --tty mdl4eo/otbtf2.4:cpu
  3. in docker desktop, check that the docker is running in the Container/Apps menu
  4. in docker desktop, click on the icon that you highlighted in your screenshot, and use the bash terminal that pops up

Is that it?

@viniciuspg
Copy link

Yes Remi,

it was exactly these steps, according to the screenshots I posted. However, it may be worth explaining the following. My Docker was already installed and enabled with WSL2. I am not sure if this will make any difference, but when I installed Docker, WSL2 was not enabled, and it was necessary to follow the steps described at https://docs.microsoft.com/en-us/windows/wsl/install-win10#manual-installation-steps, especially step 5 to enable WSL2 in Docker.

Att,
Vinicius

@remicres
Copy link
Owner Author

Great, I will add your screenshots in the documentation, if you allow me.

With WSL2 enabled I feel like you are very close to be able to use your GPU... but lets wait a bit for that 😉
Documentation incoming...

@viniciuspg
Copy link

Please use whatever you need, if necessary I can provide more details. Just ask me and I'll be available!

Thank you very much!

@remicres
Copy link
Owner Author

@viniciuspg if you success in using OTBTF with GPU enabled on Windows 10, it could be nice to detail precisely the steps.
I would add it in the documentation!

@viniciuspg
Copy link

Dear Remi,

you can be sure that if I succeed, I will report back. I even took the liberty of sending NVIDIA a request for help. I believe that if they can point us to a developer who can help us, we have a better chance of success.

Att,
Vinicius

@remicres remicres linked a pull request Apr 19, 2021 that will close this issue
Docker use #55
Merged
@viniciuspg
Copy link

Hi Remi,

last weekend we were trying at all costs to prepare the environment to enable GPU processing with otbtf on win10, without success, remember? One of the great difficulties was precisely the step described by Nvidia, which seemed to me not to be a complete instruction, although they bring several details. Well, I made a contact with the company through Networking Academy, because as I understood they have a dedicated session for academic projects, as is the case of OTB.

I believe that if OTBTF can become a reality for use, integrated into QGIS, as is OTB itself, we will have a great benefit to the community of users, especially in developing countries as is the case of Brazil, because it can be used not only by experts in programming, but also by professionals in other areas of knowledge, who depend on remote sensing to develop their research.

Well, I was returned with this contact from Shani Berko, who I believe can help in this development.

In this sense, I would like to have your e-mail contact so that I can put you in touch with Shani Berko, so that together you can assess whether this approach is promising. If so, I would be happy to help you test the application as well as help you describe the step-by-step approach. Sorry to open this issue again, but this is the only contact I could get with you. I don't know how to send a private message here, but my e-mail address is [email protected]. Feel free to contact me there.

Att,
Vinicius

@remicres remicres added Documentation Request for more documentation and removed enhancement New feature or request labels May 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
docker Questions related to the use of otbtf docker images Documentation Request for more documentation
Projects
None yet
Milestone
Enhance documentation
4 participants
@remicres @daspk04 @viniciuspg @vidlb