Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error during merging the change layers #6

Open
carolinestudent opened this issue Feb 20, 2020 · 33 comments
Open

Error during merging the change layers #6

carolinestudent opened this issue Feb 20, 2020 · 33 comments
Assignees
Labels
bug Something isn't working

Comments

@carolinestudent
Copy link

Hello !

I follow this procedure : https://smbyc.github.io/PCA4CD/#2a-generate-the-change-layer-by-component in order to detect change between two images.
Everything works like a charm until the merging process. The window is opened, i can choose the method to merge the change layers and choose the path. However when i click on "ok", i have a error message "error during merging the change layers". I don't know how to solve it…

Thank you for your answer !

@XavierCLL XavierCLL self-assigned this Feb 20, 2020
@XavierCLL
Copy link
Member

Hi Caroline,

Thanks for the report. To fix and understanding what is the error, could you provide more information? OS? version Qgis?, is it possible that you can share a file to replicate the error?

Regards

@XavierCLL XavierCLL added the bug Something isn't working label Feb 20, 2020
@carolinestudent
Copy link
Author

carolinestudent commented Feb 21, 2020

Hi !

First, thanks for your quick answer :)
I am using QGIS 3.10.2, on "Windows 10 Entreprise", 64bits.
You can find here two images on which i performed the test (https://drive.google.com/drive/folders/1JSm3dnuA0TdPrhbMV3ObLOI-qr10c_t2?usp=sharing)(images were too big to be share with GitHub plateform). For your information I tried it on other images, tiff and jpeg, RGB and graylevel, and I have the same error. I also tried to save them on the server and on a local disk.
If you have any other question, feel free to ask.

Regards

@XavierCLL
Copy link
Member

Screenshot_20200221_114445

I could not reproduce the error, for me everything works fine. Could you please try to save the result in local device, in different path and filename?

@XavierCLL
Copy link
Member

XavierCLL commented Feb 21, 2020

pca4cd.zip

Thanks for the data for test it, I did some changes to show the error, could you please download the zip file and install the plugin manually in Qgis with that changes and repeat the operation? with that changes, the plugin shows the complete error, please send me that error messages, thanks

@carolinestudent
Copy link
Author

carolinestudent commented Feb 24, 2020

PCA4CD_error messages.zip

Here are all the error messages I could find ("Impossible d'ouvrir" means "Impossible to open"), I hope that help.

@carolinestudent
Copy link
Author

carolinestudent commented Feb 24, 2020

Capture_path

If that help : I looked at the path the error message said "impossible to open...".
The images from PC1 and PC2 are here. As long as the PCA4CD window is open, there is also the file called "pc_2_detection", that I can load in a QGIS project.

@XavierCLL
Copy link
Member

pca4cd.zip

Thanks for the data for test it, I did some changes to show the error, could you please download the zip file and install the plugin manually in Qgis with that changes and repeat the operation? with that changes, the plugin shows the complete error, please send me that error messages, thanks

Thank Caroline for the report, but have you tested with the plugin that I send you in the above message?

@carolinestudent
Copy link
Author

Yes, the error messages I send you are the one I have with the plugin you send here

@carolinestudent
Copy link
Author

Capture_error1

After talking with my tutor, I realised I misunderstood the "manually install" part of your message. Here are the error message you were probably looking for :

Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\utils\system_utils.py", line 36, in wrapper
return func(*args, **kwargs)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\utils\system_utils.py", line 86, in wrapper
obj_returned = func(*args, **kwargs)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 328, in do_merge_change_layers
call_code, call_out = getstatusoutput(" ".join(cmd))
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 567, in getstatusoutput
data = check_output(cmd, shell=True, text=True, stderr=STDOUT)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 376, in check_output
**kwargs).stdout
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 455, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 907, in communicate
stdout = self.stdout.read()
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 88: invalid start byte

@XavierCLL
Copy link
Member

thank you very much, that's what I was looking for, I will check it

@XavierCLL
Copy link
Member

pca4cd.zip

Could you please test this version? (install it manually)

@carolinestudent
Copy link
Author

I tested it, it shows the same error message as initially (error during merging the change layers check the QGIS log), and it can't display any error handler window like the one I send you in my previous message.

@XavierCLL
Copy link
Member

I've tried to reproduce the problem but it was impossible, then the only way to get new information of the error is through you, sorry for bothering you again, please install this version and run again the test, and copy the message in the new dialog appear when you run the union or intersection process:

pca4cd.zip

@carolinestudent
Copy link
Author

PCA_errormessage

You are not bothering me at all, I really appreciate the fact that you are trying your best to understand what could be wrong, and I'm sorry I can't help much...
I ran the test and got the message on the picture that I paste here :

UnboundLocalError: local variable 'call_code' referenced before assignment
Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 328, in do_merge_change_layers
call_code, call_out = getstatusoutput(" ".join(cmd))
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 567, in getstatusoutput
data = check_output(cmd, shell=True, text=True, stderr=STDOUT)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 376, in check_output
**kwargs).stdout
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 455, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 907, in communicate
stdout = self.stdout.read()
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 88: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 312, in open_merge_change_layers
self.do_merge_change_layers(merge_dialog)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 331, in do_merge_change_layers
msgBox.setInformativeText("{}\n{}".format(call_code, call_out))
UnboundLocalError: local variable 'call_code' referenced before assignment

Version de Python : 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
Version de QGIS : 3.10.2-A Coruña A Coruña, d4cd3cfe5a

Chemin Python :
C:/PROGRA1/QGIS31.10/apps/qgis/./python
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/PROGRA1/QGIS31.10/apps/qgis/./python/plugins
C:\PROGRA1\QGIS31.10\apps\Python37
C:\PROGRA1\QGIS31.10\apps\Python37\Scripts
C:\Program Files\QGIS 3.10\bin\python37.zip
C:\PROGRA1\QGIS31.10\apps\Python37\DLLs
C:\PROGRA1\QGIS31.10\apps\Python37\lib
C:\Program Files\QGIS 3.10\bin
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\win32
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\win32\lib
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\Pythonwin
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:\Users\SIEXT5646\AppData\Local\pca4cd\python37
C:\Users\SIEXT5646\AppData\Local\pca4cd\python37\lib\site-packages

@carolinestudent
Copy link
Author

While working on another software (Whitebox), I realised that my computer doesn't have Java. It is likely that a lot of "commun" stuff like that are also missing. Is there any requirement that would explain the bug ?

@XavierCLL
Copy link
Member

Thanks so much for your time and testing the plugin, the plugin doesn't require java, it has everything to run that is'not the problem.

Here another change that I made in the plugin for testing:

pca4cd.zip

@carolinestudent
Copy link
Author

Error message

I got this error message :

FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable
Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 308, in open_merge_change_layers
self.do_merge_change_layers(merge_dialog)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 324, in do_merge_change_layers
subprocess.run(cmd)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 756, in init
restore_signals, start_new_session)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable

Version de Python : 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
Version de QGIS : 3.10.2-A Coruña A Coruña, d4cd3cfe5a

Chemin Python :
C:/PROGRA1/QGIS31.10/apps/qgis/./python
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/PROGRA1/QGIS31.10/apps/qgis/./python/plugins
C:\PROGRA1\QGIS31.10\apps\Python37
C:\PROGRA1\QGIS31.10\apps\Python37\Scripts
C:\Program Files\QGIS 3.10\bin\python37.zip
C:\PROGRA1\QGIS31.10\apps\Python37\DLLs
C:\PROGRA1\QGIS31.10\apps\Python37\lib
C:\Program Files\QGIS 3.10\bin
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\win32
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\win32\lib
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\Pythonwin
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:\Users\SIEXT5646\AppData\Local\pca4cd\python37
C:\Users\SIEXT5646\AppData\Local\pca4cd\python37\lib\site-packages

@XavierCLL
Copy link
Member

That error was my fault, please try with this (again):

pca4cd.zip

@carolinestudent
Copy link
Author

Error message

Here is the error message

['gdal_merge', '-of', 'GTiff', '-o', 'C:\Users\SIEXT5646\Desktop\2016_gris_pca4cd.tif', '-n', '0', '-a_nodata', '0', '-ot', 'Byte', 'C:\Users\SIEXT5~1\AppData\Local\Temp\tmp6l4apue9\pc_2_detection.tif']

[True]

@XavierCLL
Copy link
Member

Finally... I think that I found the problem, please test with this:

pca4cd.zip

@carolinestudent
Copy link
Author

carolinestudent commented Mar 6, 2020

EDIT : Even if I have an error message, a .tif file is created, and I am able to import it in QGIS afterward. However, this is working only if the change map is created from one Principal Component.

I'm sorry to tell you this, but...

Error message

Une erreur est survenue lors de l'éxécution du code Python:

FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable
Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 309, in open_merge_change_layers
self.do_merge_change_layers(merge_dialog)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 353, in do_merge_change_layers
subprocess.run(cmd)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 756, in init
restore_signals, start_new_session)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable

Version de Python : 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
Version de QGIS : 3.10.2-A Coruña A Coruña, d4cd3cfe5a

Chemin Python :
C:/PROGRA1/QGIS31.10/apps/qgis/./python
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/PROGRA1/QGIS31.10/apps/qgis/./python/plugins
C:\PROGRA1\QGIS31.10\apps\Python37
C:\PROGRA1\QGIS31.10\apps\Python37\Scripts
C:\Program Files\QGIS 3.10\bin\python37.zip
C:\PROGRA1\QGIS31.10\apps\Python37\DLLs
C:\PROGRA1\QGIS31.10\apps\Python37\lib
C:\Program Files\QGIS 3.10\bin
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\win32
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\win32\lib
C:\PROGRA1\QGIS31.10\apps\Python37\lib\site-packages\Pythonwin
C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:\Users\SIEXT5646\AppData\Local\pca4cd\python37
C:\Users\SIEXT5646\AppData\Local\pca4cd\python37\lib\site-packages

@XavierCLL
Copy link
Member

Issue related #7

Please confirm if for you is fixed too, updating the plugin to v20.4.20 and testing it, thanks

@carolinestudent
Copy link
Author

Hey Xavier ! Great to hear from you again.

I update the plugin to v20.4.20. When performing the export, it is working when only one PC is considered. When two PC are considered, the options "union" and "intersect" doesn't work. The plugin doesn't show an error, but it does not create the tif file, only the "QGIS Layer Settings" file.

Capture

Take care

@XavierCLL
Copy link
Member

Hey Caroline, mmm interesting, I will check and as always thanks for your great feedback!

@carolinestudent
Copy link
Author

Hello Xavier, any chance you manage to solve the problem ?

@XavierCLL
Copy link
Member

I think I got it, maybe... please test this zip

pca4cd.zip

@carolinestudent
Copy link
Author

I have the same problem than in my previous message

@XavierCLL
Copy link
Member

That is really weird, I've tested in a virtual machine with Windows and everything works, I think that this problem is very specific for your settings, could you please run this in the python console inside Qgis and copy the return:

import locale
locale.getdefaultlocale()

@XavierCLL
Copy link
Member

Could you please test again with this zip? sorry for putting you as a tester but is the only way at the moment:

pca4cd.zip

@carolinestudent
Copy link
Author

I agree, it is very weird !
The code return :
('fr_FR', 'cp1252')

It doesn't bother me to test it.
When performing 'union' export, the following error appears :

` FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\SIEXT51\AppData\Local\Temp\tmp1k8m0vdb\tmp_merge.tif'
Traceback (most recent call last):
File "C:\PROGRA
1\QGIS31.10\apps\Python37\lib\shutil.py", line 557, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable: 'C:\Users\SIEXT5
1\AppData\Local\Temp\tmp1k8m0vdb\tmp_merge.tif' -> 'C:\Users\SIEXT5646\Desktop\PCA_union.tif'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 309, in open_merge_change_layers
self.do_merge_change_layers(merge_dialog)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 330, in do_merge_change_layers
shutil.move(tmp_merge, merged_change_layer)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\shutil.py", line 571, in move
copy_function(src, real_dst)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\shutil.py", line 257, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\SIEXT5~1\AppData\Local\Temp\tmp1k8m0vdb\tmp_merge.tif'`

When performing the 'intersection' export, a similar error appears :

`FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\SIEXT51\AppData\Local\Temp\tmp1k8m0vdb\tmp_merge.tif'
Traceback (most recent call last):
File "C:\PROGRA
1\QGIS31.10\apps\Python37\lib\shutil.py", line 557, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 2] Le fichier spécifié est introuvable: 'C:\Users\SIEXT5
1\AppData\Local\Temp\tmp1k8m0vdb\tmp_merge.tif' -> 'C:\Users\SIEXT5646\Desktop\PCA_inter.tif'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 309, in open_merge_change_layers
self.do_merge_change_layers(merge_dialog)
File "C:/Users/SIEXT5646/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\pca4cd\gui\main_analysis_dialog.py", line 347, in do_merge_change_layers
shutil.move(tmp_merge, merged_change_layer)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\shutil.py", line 571, in move
copy_function(src, real_dst)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\shutil.py", line 257, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "C:\PROGRA1\QGIS31.10\apps\Python37\lib\shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\SIEXT5~1\AppData\Local\Temp\tmp1k8m0vdb\tmp_merge.tif'`

The export of one PC by one PC still works without problems

@XavierCLL
Copy link
Member

Thanks Caroline, I think that the problem is the locale strings in the paths, please test with this zip:

pca4cd.zip

@carolinestudent
Copy link
Author

The problem is not fixed... It is the same as my comment from the 21 April.

@XavierCLL
Copy link
Member

Hello Caroline, thanks for your reports and helps, seeing that after some tests the bug is still present, the only way is replicant the bug using the same environment that you have, that is windows and same locale setting. While, if you need to use it, I recommend you either have a virtual machine with lang set in English or change your setting to English and UTF-8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants