r.init() is very slow - network issue to github.com, pack() or download manually #376

zhaoyukoon · 2022-04-28T07:25:21Z

when i try to run import rpa as r; r.init(), the second line code never stop with following message:

[RPA][INFO] - setting up TagUI for use in your Python environment
[RPA][INFO] - downloading TagUI (~200MB) and unzipping to below folder...
[RPA][INFO] - /Users/ygzhao_1

I try to downlaod tagUI and unzip to /Users/ygzhao_1, the problem still exists.

I guess it is because of the slow speed in China.

What can I do to fix this?
Thanks

The text was updated successfully, but these errors were encountered:

kensoh · 2022-04-28T13:01:56Z

Hi @zhaoyukoon yes I suspect it is network issue. I have other users reporting similar issue. In China, it seems that the network is not very reliable to download files from GitHub, which this package does during the initial run.

Can you try on your personal laptop or use VPN software to see if it works? If it does, you can use pack() function to create a zip file for you to move to your company intranet computer. More info on using pack() - #36 (comment)

In the future, @kangyiwen may be looking at hosting some of these resource files on a website which can be reliably accessed in China. When that happens, I can upgrade the package to support downloading from alternative hosts.

kensoh · 2022-04-28T13:03:56Z

Let me know how it goes. There is another way which is to download manually, but there are a few steps and more complicated to do. Need to download manually the zip file, and also the delta update files, and then create a dummy marker file to tell the package that it is already updated.

zhaoyukoon · 2022-04-29T09:08:57Z

Thanks for reply. It is internet issue. It is very unstable to download file from github. Usually i download files manually via Chrome with proxy. My suggestion is that the setup script can be updated to check whether the required file exist or not.

Currently i am trying to change the tagui.py for checking.

kensoh · 2022-04-29T09:38:14Z

I see.. There are a few steps, first is download the TagUI zip file for your OS, then update them with the delta files from this folder (including the file in tagui.sikuli) - https://github.com/tebelorg/Tump/tree/master/TagUI-Python

Finally, need to create a dummy file rpa_python_1.47.0 in your tagui folder, as a marker so that the package does not try to download or update again on init(). Can you let me know if doing all the above steps is easier for you than using pack()?

If yes, I will update the package to support users who prefer to manually download the files instead of using pack(). Thanks in advance for your feedback! It will help improve the usability of rpa package.

zhaoyukoon · 2022-04-29T09:55:01Z

What is the purpose of _tagui_delta, which try to download files from https://raw.githubusercontent.com/tebelorg/Tump/master/TagUI-Python/? The program always fails here as many files are downloaded.

I suggest the program can either download a zip file or support manual download here.

kensoh · 2022-04-29T10:04:30Z

The TagUI zip file is big, 100-200MB. The delta file design is to only update the small changes (<1MB), instead of downloading big latest zip file every time. Thanks for your feedback, I will think over how to make it easier.

The real work is the manual downloading. Maybe a good way is write clearly the steps how to download manually for Windows, Mac, Linux. I think that might be the best and easier path for users. I will try to do that when I get some time.

zhaoyukoon · 2022-04-29T10:16:35Z

Cool! Wait for the update.

kensoh · 2022-04-30T00:21:29Z

These are the steps to manually download and set up the dependencies. If you are using init() with network access or use pack(), the steps below will be done automatically by rpa package. Do below only if you cannot use those options.

1. Download TagUI engine from https://github.com/tebelorg/Tump/releases

for Windows, download TagUI_Windows.zip
for Mac, download TagUI_macOS.zip
for Linux, download TagUI_Linux.zip

2. Unzip the file to the respective folders for your operating system

for Windows, look for %APPDATA% (cd %APPDATA%), and put your tagui folder there
for Mac or Linux, look for ~ folder (cd ~), put your tagui folder there, rename it to .tagui

3. Download update files from https://github.com/tebelorg/Tump/tree/master/TagUI-Python

tagui, tagui.cmd, end_processes, end_processes.cmd, tagui_header.js, tagui_parse.php
put these files into the tagui\src (for Windows) or .tagui/src folder (for Mac or Linux)
download the tagui.sikuli\tagui.py and overwrite the one in your src\tagui.sikuli folder
choose overwrite when prompted, to overwrite your existing files with the above files

4. Create marker file so that rpa package will not try to download anything on init()

under the tagui folder (Windows) or the .tagui folder (Mac or Linux),
create an empty file rpa_python_1.47.0 (or the version number you pip install)

5. For Windows, check if PHP is working correctly and no broken dependency

from command prompt, type cd %APPDATA%\tagui\src\php and type php -v
if there is error message, usually due to missing MSVCR110.dll, download and install
https://raw.githubusercontent.com/tebelorg/Tump/master/vcredist_x86.exe

6. For Mac and Linux, do the following to ensure execute permissions are there

run the command from terminal cd ~ and then chmod -R 755 .tagui

7. For Mac only, do the following steps to fix the OpenSSL issue with Mac

go to this url https://github.com/tebelorg/Tump/releases
download and unzip phantomjs-2.1.1-macosx.zip
delete the phantomjs folder under .tagui/src folder and
replace with the above unzipped folder (rename to phantomjs)

kensoh · 2022-04-30T00:28:51Z

@zhaoyukoon above are the steps to download and set up manually, let me know if you run into any problems!

kangyiwen · 2022-04-30T01:04:26Z

@zhaoyukoon Github下载的确非常慢，我放到国内了

各类TagUI安装/插件/资源国内下载，请访问：
链接：https://pan.baidu.com/s/1GrkOxmu9fTBpwzO1JzaNUw?pwd=tagu

如果百度网盘下载还慢，加我微信： kyw2004 我单独发你。

kensoh · 2022-04-30T06:17:51Z

Thanks @kangyiwen! For rpa package, it uses a forked version of TagUI that I maintain, in order to adapt TagUI engine to work well with Python process.

Thus rpa package will not work with the standard TagUI (human language version) zip files which have iterated in another direction. The files at steps 1357 above would be separately required to be downloaded.

kensoh · 2022-04-30T10:28:41Z

Adding on to above discussions, @zhaoyukoon and @kangyiwen, if you can,

Can you try below Python script file to see how is automatic downloading from my own server instead of GitHub?

That will give me more information if it makes sense to host a separate download location as an option to GitHub.

test_download.py

import rpa as r
r.download('https://tebel.org/TagUI_Windows.zip')

kensoh · 2022-05-04T15:02:48Z

I'm suspecting downloading from my tebel.org will not be much better than GitHub. My server is hosted in Singapore.

Anyway, see how your testing goes. And if it makes sense to set up a secondary download server as part of package.

zhaoyukoon · 2022-05-05T02:16:36Z

It is still very slow for https://tebel.org/TagUI_Windows.zip

kensoh · 2022-05-09T23:50:01Z

I see.. Looks like for now, need to try these steps here - #376 (comment)

Because even if iterate codebase for rpa package to support files that are pre-downloaded by users, most of steps above are still needed to download manually from GitHub, using VPN or other ways. For users without access to github.com.

kensoh · 2022-05-09T23:52:33Z

Alternative is try pack() on a PC with internet access. Also, in theory, Python can support setting proxy.

But I have not validated that approach to work, no feedback from users so far on that method.

zhaoyukoon · 2022-05-10T07:07:45Z

These are the steps to manually download and set up the dependencies. If you are using init() with network access or use pack(), the steps below will be done automatically by rpa package. Do below only if you cannot use those options.

1. Download TagUI engine from https://github.com/tebelorg/Tump/releases

for Windows, download TagUI_Windows.zip

for Mac, download TagUI_macOS.zip

for Linux, download TagUI_Linux.zip

2. Unzip the file to the respective folders for your operating system

for Windows, look for %APPDATA% (cd %APPDATA%), and put your tagui folder there

for Mac or Linux, look for ~ folder (cd ~), put your tagui folder there, rename it to .tagui

3. Download update files from https://github.com/tebelorg/Tump/tree/master/TagUI-Python

tagui, tagui.cmd, end_processes, end_processes.cmd, tagui_header.js, tagui_parse.php

put these files into the tagui\src (for Windows) or .tagui/src folder (for Mac or Linux)

download the tagui.sikuli\tagui.py and overwrite the one in your src\tagui.sikuli folder

choose overwrite when prompted, to overwrite your existing files with the above files

4. Create marker file so that rpa package will not try to download anything on init()

under the tagui folder (Windows) or the .tagui folder (Mac or Linux),

create an empty file rpa_python_1.47.0 (or the version number you pip install)

5. For Windows, check if PHP is working correctly and no broken dependency

from command prompt, type cd %APPDATA%\tagui\src\php and type php -v

if there is error message, usually due to missing MSVCR110.dll, download and install

https://raw.githubusercontent.com/tebelorg/Tump/master/vcredist_x86.exe

6. For Mac and Linux, do the following to ensure execute permissions are there

run the command from terminal cd ~ and then chmod -R 755 .tagui

7. For Mac only, do the following steps to fix the OpenSSL issue with Mac

go to this url https://github.com/tebelorg/Tump/releases

download and unzip phantomjs-2.1.1-macosx.zip

delete the phantomjs folder under .tagui/src folder and

replace with the above unzipped folder (rename to phantomjs)

I try to install related file based on your instruction, it works.

>>> import rpa as r
>>> r.init()
True

Thanks!

kensoh · 2022-05-11T15:52:30Z

Thanks @zhaoyukoon for sharing back your findings! Ok I'll close the issue now. In the future, if there's a better way around this, I will improve the rpa package with the new way.

DoubleK6 · 2023-10-11T08:27:06Z

Using proxies to download is faster and easier, open a cmd window, set the proxy to the cmd window, and execute python rpa.py in the cmd window. Only the pack() function is called in rpa.py. You get the zip file. Download speed depends on the agent speed. Even with the manual download method above, there is no proxy is still a problem, since there is a proxy, why not use the pack() function?

kensoh · 2023-10-25T19:44:17Z

Also adding on, it is also possible to set proxy within the Python code -
https://stackoverflow.com/questions/31639742/how-to-pass-all-pythons-traffics-through-a-http-proxy

kensoh changed the title ~~r.init() is very slow~~ r.init() is very slow - network access issue to reach github.com, some ideas Apr 28, 2022

kensoh added the query label Apr 28, 2022

kensoh mentioned this issue Apr 28, 2022

How to use the package if internet access is blocked - use pack() and update() #362

Closed

kensoh mentioned this issue Apr 30, 2022

Use in air-gap environments with network restriction and no internet [done] #36

Closed

kensoh changed the title ~~r.init() is very slow - network access issue to reach github.com, some ideas~~ r.init() is very slow - network access issue to github.com, download manually Apr 30, 2022

kensoh changed the title ~~r.init() is very slow - network access issue to github.com, download manually~~ r.init() is very slow - network access issue to github.com, pending feedback on 2nd server May 4, 2022

kensoh changed the title ~~r.init() is very slow - network access issue to github.com, pending feedback on 2nd server~~ r.init() is very slow - network issue to github.com, pending feedback on 2nd server May 4, 2022

kensoh changed the title ~~r.init() is very slow - network issue to github.com, pending feedback on 2nd server~~ r.init() is very slow - network issue to github.com, pending tests on 2nd server May 4, 2022

kensoh changed the title ~~r.init() is very slow - network issue to github.com, pending tests on 2nd server~~ r.init() is very slow - network issue to github.com, pack() or download manually May 9, 2022

kensoh closed this as completed May 11, 2022

kensoh mentioned this issue Jul 2, 2022

Run python script in Linux with error - internet restriction, try out these options #403

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

r.init() is very slow - network issue to github.com, pack() or download manually #376

r.init() is very slow - network issue to github.com, pack() or download manually #376

zhaoyukoon commented Apr 28, 2022

kensoh commented Apr 28, 2022

kensoh commented Apr 28, 2022

zhaoyukoon commented Apr 29, 2022

kensoh commented Apr 29, 2022

zhaoyukoon commented Apr 29, 2022

kensoh commented Apr 29, 2022

zhaoyukoon commented Apr 29, 2022

kensoh commented Apr 30, 2022 •

edited

Loading

kensoh commented Apr 30, 2022

kangyiwen commented Apr 30, 2022 •

edited

Loading

kensoh commented Apr 30, 2022 •

edited

Loading

kensoh commented Apr 30, 2022 •

edited

Loading

kensoh commented May 4, 2022

zhaoyukoon commented May 5, 2022

kensoh commented May 9, 2022 •

edited

Loading

kensoh commented May 9, 2022

zhaoyukoon commented May 10, 2022 •

edited

Loading

kensoh commented May 11, 2022

DoubleK6 commented Oct 11, 2023

kensoh commented Oct 25, 2023

r.init() is very slow - network issue to github.com, pack() or download manually #376

r.init() is very slow - network issue to github.com, pack() or download manually #376

Comments

zhaoyukoon commented Apr 28, 2022

kensoh commented Apr 28, 2022

kensoh commented Apr 28, 2022

zhaoyukoon commented Apr 29, 2022

kensoh commented Apr 29, 2022

zhaoyukoon commented Apr 29, 2022

kensoh commented Apr 29, 2022

zhaoyukoon commented Apr 29, 2022

kensoh commented Apr 30, 2022 • edited Loading

kensoh commented Apr 30, 2022

kangyiwen commented Apr 30, 2022 • edited Loading

kensoh commented Apr 30, 2022 • edited Loading

kensoh commented Apr 30, 2022 • edited Loading

kensoh commented May 4, 2022

zhaoyukoon commented May 5, 2022

kensoh commented May 9, 2022 • edited Loading

kensoh commented May 9, 2022

zhaoyukoon commented May 10, 2022 • edited Loading

kensoh commented May 11, 2022

DoubleK6 commented Oct 11, 2023

kensoh commented Oct 25, 2023

kensoh commented Apr 30, 2022 •

edited

Loading

kangyiwen commented Apr 30, 2022 •

edited

Loading

kensoh commented Apr 30, 2022 •

edited

Loading

kensoh commented Apr 30, 2022 •

edited

Loading

kensoh commented May 9, 2022 •

edited

Loading

zhaoyukoon commented May 10, 2022 •

edited

Loading