Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LiCSBAS12_loop_closure.py #399

Open
oguzhannysr opened this issue Sep 5, 2024 · 66 comments
Open

LiCSBAS12_loop_closure.py #399

oguzhannysr opened this issue Sep 5, 2024 · 66 comments

Comments

@oguzhannysr
Copy link

Hello @yumorishita, the computer gets stuck at this step. What is the reason for this? I do it without cropping the area.
image
202409050828batch_LiCSBAS_01_16.log

@yumorishita
Copy link
Owner

Try n_para=1.

n_para="" # Number of paralell processing in step 02-05,12,13,16. default: number of usable CPU

@oguzhannysr
Copy link
Author

@yumorishita , I try by setting n12_para to 1

@oguzhannysr
Copy link
Author

oguzhannysr commented Sep 5, 2024

@yumorishita , Now I'm stuck here, which n_para should I change?

  Running  64000/ 65117th point...
  Running  63000/ 65117th point...
  Running  61000/ 65117th point...
  Running  62000/ 65117th point...
  Running  65000/ 65117th point...
  Running  60000/ 65117th poi
```nt...
  Elapsed time for 11th patch: 00h 05m 58s

Process 624/2689th line (12/52th patch)...
  Reading 724 ifg's unw data...
  84278/183768 points removed due to not enough ifg data...

  Identifing gaps, and counting n_gap and n_ifg_noloop,
  with 16 parallel processing...
    Running  1/16th patch...
    Running  2/16th patch...
    Running  3/16th patch...
    Running  4/16th patch...
    Running  5/16th patch...
    Running  6/16th patch...
    Running  9/16th patch...
    Running 10/16th patch...
    Running 11/16th patch...
    Running 12/16th patch...
    Running 14/16th patch...
    Running 15/16th patch...
    Running 16/16th patch...
    Running  8/16th patch...
    Running  7/16th patch...
    Running 13/16th patch...

@yumorishita
Copy link
Owner

I already mentioned above.

@oguzhannysr
Copy link
Author

oguzhannysr commented Sep 7, 2024

@yumorishita,

202409052217batch_LiCSBAS_01_16.log

Process 2673/2689th line (243/251th patch)...
  Reading 724 ifg's unw data...
  38874/38874 points removed due to not enough ifg data...
  Elapsed time for 243th patch: 00h 05m 04s

Process 2684/2689th line (244/251th patch)...
  Reading 724 ifg's unw data...
  38874/38874 points removed due to not enough ifg data...
  Elapsed time for 244th patch: 00h 00m 57s

Process 269
5/2689th line (245/251th patch)...
  Reading 724 ifg's unw data...
Traceback (most recent call last):
  File "/home/yasar/LICSBAS/bin/LiCSBAS13_sb_inv.py", line 996, in <module>
    sys.exit(main())
  File "/home/yasar/LICSBAS/bin/LiCSBAS13_sb_inv.py", line 541, in main
    unw = np.fromfile(f, dtype=np.float32, count=countl).reshape((lengththis, width))*coef_r2m
ValueError: cannot reshape array of size 17670 into shape (11,3534)

@yumorishita
Copy link
Owner

Maybe some of the GEOCml1/*/*unw files are corrupted. Please check all the files have the same size.

@oguzhannysr
Copy link
Author

@yumorishita , There are more than 1000 folders, how can I do this more easily? Also, if there is a corrupt file, what should I do? Should I go back to the beginning?

@yumorishita
Copy link
Owner

Remove corrupted files and restart from step11.

@oguzhannysr
Copy link
Author

@yumorishita , I checked for corrupt files, but all files were the same size and as they should be. I don't know what to do now?

@yumorishita
Copy link
Owner

I have no idea, maybe some other files are corrupted. What I can suggest is removing all the files and rerunning from the start.

@oguzhannysr
Copy link
Author

thank you...

@oguzhannysr
Copy link
Author

oguzhannysr commented Sep 11, 2024

@yumorishita , Dear professor, I have a different technical question for you apart from this topic. Using LICSBAS, I produced cumulative deformation los maps in ascending and descending orbits (2016-2024). How technically correct is it to separate each image pair as east-west and up-down using the cumulative los data produced for the ascending and descending orbits as of 2016? Because after a certain year (failure of one of the Sentinel-1 satellite constellations), the ascending and descending image dates do not match. I need your opinion on this matter..
up-down folder
ud
east-west folder
ew
up-down timeseries example
up-down timeseries example
east-west timeseries example
image

@as1234554321
Copy link

haii @oguzhannysr
hello, how do I get this much decompose?, I only got 1 UD file
can you help me to tells how to process decomposition appropriately and slowly. Thank You

@yumorishita
Copy link
Owner

How technically correct is it to separate each image pair as east-west and up-down using the cumulative los data produced for the ascending and descending orbits as of 2016?

What do you mean by "as of 2016"?

@oguzhannysr
Copy link
Author

2016 yılı itibariyle yükselen ve alçalan yörüngeler için üretilen kümülatif los verilerini kullanarak her görüntü çiftini doğu-batı ve yukarı-aşağı olarak ayırmak teknik olarak ne kadar doğrudur?

"2016 itibariyle" derken neyi kastediyorsunuz?

@yumorishita, I meant to say starting from 2016 and until 2024, so I did my work within this date range.

@yumorishita
Copy link
Owner

How did you treat the date difference between ascending and descending in the decomposition?

@oguzhannysr
Copy link
Author

@yumorishita , as it can be seen in the image, after the date 20211218, the dates of the couples start not to match with each other. Therefore, I made ud-ew separation based on two pairs in 6-day periods. For example, I matched the date 20211224 in the ascending orbit with the date 20211230 in the descending orbit and made the separation as ud-ew. How technically correct is this? Does it provide me with the correct trend of the deformation?

image

@yumorishita
Copy link
Owner

If the deformation is slow and we can ignore the 6 day difference, I think your approach makes sense.

@oguzhannysr
Copy link
Author

@yumorishita , Thank you sir, but is the reason why the data dates do not match after a certain time (after 2021) due to the failure of the Sentinel-1 satellite or due to incomplete processing of data in the COMET system?

@yumorishita
Copy link
Owner

I don't know. You should check it yourself by searching for data, e.g., on ASF.

@oguzhannysr
Copy link
Author

oguzhannysr commented Sep 16, 2024

dear @yumorishita , Is there an anomaly in n_loop_err in my results here? Do you think I should set p15_n_loop_err_thre=0 and run it again?
202409120903batch_LiCSBAS_01_16.log

Noise index    : Threshold  (rate to be masked)
- coh_avg      : 0.05       ( 0.0%)
- n_unw        :  531       ( 0.0%)
- vstd         :  100 mm/yr ( 0.0%)


- maxTlen      :    1 yr    ( 0.0%)
- n_gap        :   10       ( 0.0%)
- stc          :    5 mm    ( 0.0%)
- n_ifg_noloop :   50       ( 0.5%)
- n_loop_err   :    5       (20.2%)
- resid_rms    :    2 mm    ( 2.5%)

Masked pixels  : 224/1109 (20.2%)
Kept pixels    : 885/1109 (79.8%)

@yumorishita
Copy link
Owner

The result looks normal.

@oguzhannysr
Copy link
Author

@yumorishita , I got an error like the following. I couldn't understand why. My other files are below. What do you think is the reason?

202409190909batch_LiCSBAS_01_16.log
batch_LiCSBAS.txt

Create unw and cc
  1 parallel processing...
    0/1258th IFG...
/home/yasar/LICSBAS/bin/LiCSBAS02_ml_prep.py:441: RuntimeWarning: invalid value encountered in cast
  cc = cc.astype(np.uint8) ##nan->0, max255, auto-floored
/home/yasar/LICSBAS/bin/LiCSBAS02_ml_prep.py:441: RuntimeWarning: invalid value encountered in cast
  cc = cc.astype(np.uint8) ##nan->0, max255, auto-floored
/home/yasar/LICSBAS/bin/LiCSBAS02_ml_prep.py:441: RuntimeWarning: invalid value encountered in cast
  cc = cc.astype(np.uint8) ##nan->0, max255, auto-floored

@as1234554321

This comment was marked as resolved.

@yumorishita
Copy link
Owner

@oguzhannysr I am not sure but cc files may be corrupted. Please check the contents.

@yumorishita
Copy link
Owner

@as1234554321

  1. It can be normal
  2. automatically deleted
  3. I have no idea without the full log

@yumorishita
Copy link
Owner

These messages are just warnings, not errors, and can be ignored. The processing was successfully finished with no error.

@oguzhannysr
Copy link
Author

@yumorishita , Now I'm stuck here, which n_para should I change?

  Running  64000/ 65117th point...
  Running  63000/ 65117th point...
  Running  61000/ 65117th point...
  Running  62000/ 65117th point...
  Running  65000/ 65117th point...
  Running  60000/ 65117th poi
```nt...
  Elapsed time for 11th patch: 00h 05m 58s

Process 624/2689th line (12/52th patch)...
  Reading 724 ifg's unw data...
  84278/183768 points removed due to not enough ifg data...

  Identifing gaps, and counting n_gap and n_ifg_noloop,
  with 16 parallel processing...
    Running  1/16th patch...
    Running  2/16th patch...
    Running  3/16th patch...
    Running  4/16th patch...
    Running  5/16th patch...
    Running  6/16th patch...
    Running  9/16th patch...
    Running 10/16th patch...
    Running 11/16th patch...
    Running 12/16th patch...
    Running 14/16th patch...
    Running 15/16th patch...
    Running 16/16th patch...
    Running  8/16th patch...
    Running  7/16th patch...
    Running 13/16th patch...

@yumorishita ,Hello sir, the problem still persists. I want to process the entire tile at once, but I stay in the same step. I can't understand whether my computer's RAM is not enough.

@yumorishita
Copy link
Owner

@oguzhannysr What is your RAM size?

@yumorishita
Copy link
Owner

@as1234554321 I have no idea from this limited information.

@oguzhannysr
Copy link
Author

@oguzhannysr 16GB should be generally enough. Is your environment WSL? Which OS do you use?

@yumorishita, Yes, I run LICSBAS in WSL in Windows.

@yumorishita
Copy link
Owner

@as1234554321 Now I see your point. You should set the reference point in the south area.

@yumorishita
Copy link
Owner

@oguzhannysr I have never used WSL but it seems that the stuck tends to happen in WSL according to the reported issues. I suggest using the VirtualBox as I do.

@as1234554321
Copy link

hi sir @yumorishita
i was set reference point with -- licsbas cum2vel --ref.geo but the result same. the velocity of the south area remains positive/uplift.
pic 1 ( process licsbas cum2vel done)
image
pic 2 the plot LiCSBAS_plot_ts.py of the result licsbascum2 vel. left is ascending and right is desending. both of them in south area, velocity value still positif, but in other journal/paper this area was landsubsidence. what causes this difference?
image

@yumorishita
Copy link
Owner

step05 does not set any reference point but just clips the area.
The auto-determined reference is selected using the median under the assumption that more than half of the pixels are stable. In your case, this assumption does not hold and the stable area falsely has uplift.
The option --ref_geo 110.22/110/59/-7.19/-6.90 specifies the entire area, not the southern area. You must specify the small area in the stable southern area, e.g., -r 200:210/150:160.

@as1234554321
Copy link

as1234554321 commented Sep 25, 2024

Ohh I see, thankyou for your answer and suggestion
1.ref point stable means an area where there is no uplift/land subsidence, is that right/wrong?
2.how to determine a stable ref point?
3. i was ran your suggestion to change ref area -r 200:210/150:160 with licsbascum2vel.py in ascending it works, but when displaying the results with licsbasplot ref the area at the top left of the velocity window does not change. here's the picture. did I take the wrong steps?
image

  1. I was ran point no 3 in decending but the result nan value. I was confused about this. Another journal explains that the southern area has land subsidence but the result is licsbas uplift.
    sorry for many quetions in this part, I hope you don't mind answering it. Thankyou

@yumorishita
Copy link
Owner

  1. Right.
  2. It needs external information or can be arbitrary.
  3. You should also set -r option for LiCSBAS_plot_ts.py
  4. I guess there was no valid point in the selected reference area. You should change the reference area.

@as1234554321
Copy link

as1234554321 commented Sep 26, 2024

  1. Benar.
  2. Memerlukan informasi eksternal atau dapat bersifat acak
    -Can the GPS CORS point be used as a ref point? If it is determined randomly, the result is likely to be a nan value like what I did before.
    -can you suggestion to me for ref point are correct?
  3. Anda juga harus mengatur -ropsi untuk LiCSBAS_plot_ts.py
    Can you give a correct example of how to write it so that it is successful?
  4. Saya kira tidak ada titik yang valid di area referensi yang dipilih. Anda harus mengubah area referensi.
    is there an example where changing the ref point will change the velocity value from (+) to (-)?

@yumorishita
Copy link
Owner

The pixel corresponding to the GPS CORS can be a ref point if it exists.
Please read the usage of LiCSBAS_plot_ts.py.
Your case is the good example if you change the ref point.

@as1234554321
Copy link

as1234554321 commented Sep 27, 2024

thankyou sir, I will try it
permission to make sure sir, so
1.I changed the reference point using licsbascum2vel.py then run on licsbasplot.py + licsbascum2vel result + -r GPS CORS point. is that right?
2. Hi sir, I want to change the ref point using gps cors. but the available data is in lat/lot. How to change it to pixel because -r (X1/Y1....). here are the coordinates. can you help me to give an example
Lat/lot GPS CORS : -6. 987259/110.37693

Thank you

@yumorishita
Copy link
Owner

licsbascum2vel.py does not affect licsbasplot.py at all because the output of licsbascum2vel.py is not used in licsbasplot.py.
You can use --ref_geo option instead of -r. Please read the usage.

@as1234554321
Copy link

licsbascum2vel.py sama sekali tidak memengaruhi licsbasplot.py karena output licsbascum2vel.py tidak digunakan dalam licsbasplot.py. Anda dapat menggunakan opsi --ref_geo sebagai ganti -r. Harap baca penggunaannya.

so what tools do i use to change the ref point and how do i know if the velocity has changed?
sorry I don't know how to do it, thankyou

@yumorishita
Copy link
Owner

You should specify the reference for each command.

@as1234554321
Copy link

Anda harus menentukan referensi untuk setiap perintah.

yes I know I have to change the ref on every command. but I'm confused about is what commands should be used? Can you provide step by step tools that should be used. I've read the issue related to the ref area but I still miss understanding.
thankyou
#102 #131

@yumorishita
Copy link
Owner

It depends on what output you want. Please carefully read the usage for each command. All the required information is there.

@as1234554321

This comment was marked as resolved.

@yumorishita
Copy link
Owner

How about -r 190:200/100:110?

@as1234554321

This comment was marked as resolved.

@yumorishita
Copy link
Owner

For example, --ref_geo 110.376/110.377/-6.987/-6.988.

@as1234554321
Copy link

Hi sir, I want to ask you again about this,
gambar kiri (hasil licsbas) gambar kanan (STAMPS)
image
I have changed the CORS ref point so the brown color (Licsbas) showing the LOS value is uplift while the blue color (STAMPS) shows no deformation (stable). How could this be different?
what causes this?
can you help solve the problem.
Is there a provision that the area can be said to be stable (even if it experiences small uplift/subsidence)?

@yumorishita
Copy link
Owner

I cannot see a significant difference between them.

@as1234554321
Copy link

licsbas result range uplift 0-30 mm but STA MPS just 0 (stable)
Is there a provision for the area to be said to be stable even if there is a small uplift/subsidence?

@yumorishita
Copy link
Owner

If you set the reference there, the displacement will become 0.

@as1234554321
Copy link

Hi sir
i want to change the reference point in the vel.filt file. However the results of cum2vel.py are the same as vel.filt.mskd (pixel are masking).
How to change the cum2vel reference point in the vel.filt file (without masking pixel)?
image

@yumorishita
Copy link
Owner

Change the value for --ref_geo option.

@as1234554321
Copy link

as1234554321 commented Oct 24, 2024

Change the value for --ref_geo option.

sorry i dont understand about this, can you give an example please?
cum2vel results like the left image, I want cum2vel results like the right image (without pixel noise mask), just change ref geo in vel.filt
image

@as1234554321
Copy link

as1234554321 commented Oct 24, 2024

Jika Anda menetapkan referensi di sana, perpindahan akan menjadi 0.

sorry I follow up about this, I process this using a batch script with default parameter >change ref area with cum2vel> decompose
Hello Sir, I have changed the reference area but the result is the same, only the values ​​are different.
There is no significant difference. The brown area remains uplift while the STAMPS, gmtsar, ps insar processing results show subsidence. Can you help me solve this issue?

@yumorishita
Copy link
Owner

I see. Use --nomask option.

@as1234554321
Copy link

Jika Anda menetapkan referensi di sana, perpindahan akan menjadi 0.

sorry I follow up about this, I process this using a batch script with default parameter >change ref area with cum2vel> decompose Hello Sir, I have changed the reference area but the result is the same, only the values ​​are different. There is no significant difference. The brown area remains uplift while the STAMPS, gmtsar, ps insar processing results show subsidence. Can you help me solve this issue?

how about this sir?

@yumorishita
Copy link
Owner

I cannot understand your situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants