CSV datasource not converted from 3.16 to 3.22 correctly - projects not usable anymore #48587

mlechner · 2022-05-17T16:20:53Z

What is the bug or the crash?

In a project created with QGIS 3.16.x the datasource of a CSV layer is give like:
<datasource>./aef/2und3k - Kopie.aef?type=csv&delimiter=\t&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=no</datasource>
When openinng the project in QGIS 3.22.5 the datasource is not found, even it exists at the correct place.

it is not possible to "Auto-Find" or "Browse" the source - nothing happens on click (well on Auto-Find a small window appears and disappears with no content
when keepning all the unavailable layers on project opening it is not possible to "repair the datasource", because the files (extension *.aef) are not showing up and the "Original source iin the Repair Data Source Windows shows up with lots of duplicates:
Original source URI: t&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=nott&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=no&t&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=nost&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=nokt&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=noit&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=nopt&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=noLt&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=noit&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=nont&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=noet&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center East&yField=Center North&crs=EPSG ...

If I add the CSV file into a fresh project on 3.22.5 the datasource is saved as:
<datasource>file:./aef/2und3k%20-%20Kopie.aef?encoding&type=csv&delimiter=%5Ct&skipLines=115&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&decimalPoint=,&xField=Center%20East&yField=Center%20North&crs=EPSG:32636&spatialIndex=no&subsetIndex=no&watchFile=no</datasource>

And works without problems.

Seems that there have been changes in datasource handling and 3.22.x does not convert the old datasource values correctly.

This results in QGIS projects that can not be used anymore and have to be created from scratch when upgrading to 3.22.x - I guess this is a critical bug

Steps to reproduce the issue

see Workflow described above

Versions

3.22.5

Supported QGIS version

I'm running a supported QGIS version according to the roadmap.

New profile

I tried with a new QGIS profile

Additional context

No response

The text was updated successfully, but these errors were encountered:

mlechner · 2022-05-17T16:22:30Z

may be this is connected with #48422 (wich is closed, but not solved AFAIK)

mlechner · 2022-05-17T16:28:59Z

may be this has to do with https://www.qgis.org/en/site/forusers/visualchangelog322/index.html#feature-create-a-generic-data-item-provider-for-all-file-based-datasources
@nyalldawson ?

elpaso · 2022-05-20T07:46:59Z

@mlechner can you please attach the failing project and data that I can use for debugging?

mlechner · 2022-05-20T08:08:24Z

@elpaso as it is not my project, but from colleagues, and the data is sensible, i can not provide you with the failing project or data. I can try to create a simple project with example data that is appropriate for repoducing the bug.

In the meantime I tried some thing with python, pointing out that a
v.setDataSource(v.dataProvider().dataSourceUri().replace('/t', '%5Ct'), v.name(), v.providerType())
seems to fix the problem. In the QGIS projectfile (*.qgs) the tab-delimiter is "\t" and results in
?type=csv&delimiter=/t& from QgsMapLayer.source()

May be this hints helps until I can create a sample project

elpaso · 2022-05-20T09:56:56Z

@elpaso as it is not my project, but from colleagues, and the data is sensible, i can not provide you with the failing project or data. I can try to create a simple project with example data that is appropriate for repoducing the bug.

Maybe you can send the project with the CSV redacted (just leave a couple of rows with fake data in it), data are probably not relevant to the bug, I think it's in the datasource URI encoding.

mlechner · 2022-05-20T12:41:41Z

see attached a minified example project (incl. *.qgs and data as *.zip) for reproducing the bug. just open in QGIS >= 3.22.5 (may be 3.22.x)

delimitedtext_example.zip

elpaso · 2022-05-20T15:38:20Z

Thanks for sharing, I have no problems to open the sample file with master or 3.24.2.

Closing because I cannot reproduce the issue, it may be already fixed.

mlechner · 2022-05-23T07:42:20Z

I can NOT reproduce the bug on 3.22.7 LTR on Linux/Ubuntu anymore (don't know about 3.22.6). I will check 3.22.7 LTR on M$ Windoof.

ukulle · 2022-05-24T12:31:01Z

It seems to me my similar problem/bug opening delimited text files was gone after moving the data files to a local drive like D:\data. The layer (resp. the original data file) loaded from a network drive was invalid every time I load the project again. The problem appeared after I updated from LTR 3.16 to 3.22.6 (Windows 10).

mlechner · 2022-05-24T13:48:11Z

@ukulle : can you verify that the only change made was moving the project from a network drive to local? Our status of exploring at the moment is like following:

QGIS 3.22.5 / Win10 - not working
QGIS 3.226 / Win10 - wokrking (but has to be checked again)
QGIS 3.22.7 / Win10 - not working
QGIS 3.22.7 / Ubuntu 22.04 - working

I will post more details tomorrow, as this is not fully clearified for me, yet.

Everything seems to break with the delimiter in the project being interpreted as "\t", "/t" or "%5Ct" while it definitely is "delimiter=\t&" in the orginal *.qgs file.

ukulle · 2022-05-24T16:36:24Z

Sorry, it was worded a bit misleadingly (corrected above now). I left the project file on a network drive and moved the data files to a local drive. I have simple data (points) but they are constantly updated with a text editor. For the workaround ("local drive"), I had to import the data files that were previously moved to D:\data for the layer once and save the project file.

I have tested with different delimiters (see file names below).

`
using mixed delimiters

id latitude longitude
5746 49.906009;6.540942
5747 49.888455 6.544912
5748 49.870383;6.544483
`

QGIS 3.22.7 (mixed delimiters: blank, tab, semicolon) source network drive -NOT working
... source="./data/layer-by-delimited-text-delimiters-mixed.txt?type=csv&delimiter= \t;& ...
QGIS 3.22.7 (mixed delimiters: blank, tab, semicolon) source local drive - working
... source="file:///D:/data/layer-by-delimited-text-delimiters-mixed.txt?type=csv&delimiter=%20%5Ct;& ...
QGIS 3.22.7 (delimiter tab only) source network drive - NOT working
... source="./data/layer-by-delimited-text-delimiters-tab-only.txt?type=csv&delimiter=\t& ...
QGIS 3.22.7 (delimiter tab only) source local drive - working
... source="file:///D:/data/layer-by-delimited-text-delimiters-tab-only.txt?type=csv&delimiter=%5Ct& ...

No idea how this is caused, especially since it always worked in QGIS version 3.16

mlechner · 2022-05-25T07:04:36Z

@ukulle, @elpaso there definitely seems to be a bug on QGIS with handling on tab-delimiters (\t) just on Windows!
I added more example projects and all work on Linux while the projects only work on Windows when the \t delimiter manually has been edited to %5Ct - this affects 3.22.5 and still 3.22.7 but Windows only!

The only working example-projects are

example_encodedDelimiter.qgs
example_withFailure-onlyEncDeli.qgs
all the others fail to open.

delimitedtext_example.zip

In addition I can say that the hovertext on the defect layer in the layer tree prits "/t" being the delimiter even "\t" is in the *.qgs file!

@elpaso - could you please reopen this issue, as it definitely is not fixed yet, but only seems to affect Windows-versions.

agiudiceandrea · 2022-05-25T14:49:12Z

It seems to me the issue does occur on Windows with any QGIS version >= 3.22.0, while it doesn't occur with QGIS versions <= 3.20.3 (it this case, the only non working project is example_changedslash2).

mlechner · 2022-06-02T07:53:59Z

Until this is fixed in QGIS core I created a small experimental Plugin to fix a projects layers.
See https://github.com/OpenBfS/bfs_delimitedtext_pathfixer
The Plugin just replaces the delimiter in datasource "/t" with "%5Ct" and sets the project dirty.

mlechner · 2022-06-02T09:49:06Z

It seems to me the issue does occur on Windows with any QGIS version >= 3.22.0, while it doesn't occur with QGIS versions <= 3.20.3 (it this case, the only non working project is example_changedslash2).

that is correct. I just tried out several alternative slash/encoding configurations in the example file, without being sure that all of these can occur in practice. But it definitely can be said, that the problem can be narrowed to wrong parsing on Windows of "delimiter=\t" in the datasource definition in the project file. Well, the source() string of the layer is "delimiter=/t" on Windows, even thereis a backslash used in the project file (opened with an editor). So I guess fixing the parsing of this specific substring should fix QGIS. But that is far from my knowledge of QGIS code.

mlechner · 2022-06-22T16:14:22Z

The QGIS Plugin repository does have an experimental Plugin trying to fix this problem. Just as a temporarily workaround until the problem itself is fixed. See

https://plugins.qgis.org/plugins/bfs_delimitedtext_pathfixer

grandebooz · 2022-06-23T14:56:26Z

i found that changing the datasource section in the file qgs
from
"./datafile.txt...."
to
"file:./CAVALLOTTI.txt...."
Causes it to be loaded correctly!!!!

Just dont care about delimiter, they work well both , 'file:' is the key !

grandebooz · 2022-06-23T14:57:53Z

or may be an other way to workaround...

pathmapper · 2022-06-28T15:59:06Z

Came across a similar issue which looks like it's related to an umlaut in a header field of the CSV.

Minimal test case in #49186.

pathmapper · 2022-06-28T16:09:16Z

I think it's in the datasource URI encoding.

I think so, too.

agiudiceandrea · 2022-06-28T17:08:43Z

a similar issue which looks like it's related to an umlaut in a header field of the CSV

It seems the data files provided by @mlechner don't have any umlaut in the header fields names.

pathmapper · 2022-06-29T10:18:09Z

Right, I meant it is similar in the way that it seems to be related to source URL encoding/decoding and Windows specific.

mlechner · 2022-07-21T06:37:56Z

@elpaso at at the QGIS Changelog for 3.26 you wrote that it "Works for me on 3.24 and master". As this Bug is OS-specific (Windows only), did it work for you on Win/3.24 Win/master?
Because the affected 3.22.x is LTR, would this call for a backport/fix 3.22.x LTR?

ed76 · 2023-02-14T08:50:36Z

It seems the problem has still not been solved. When will this error finally be fixed? at the moment its not possible to use a qgis version >3.16.

mlechner · 2023-02-14T09:27:36Z

It seems the problem has still not been solved. When will this error finally be fixed? at the moment its not possible to use a qgis version >3.16.

Hi ed76, I can not fox the problem in 3.16 core (as I am not deep in the core code), but did you try my experimental plugin #48587 (comment) as a dirty workaround?

Anyhow, I hope the bug will be fixed from one of the core developers who know how to fix it.

elpaso · 2023-02-14T11:35:17Z

@mlechner , @ed76 I am happy to have another look at the issue (on windows using current QGIS master) but I am having problems to understand how to reproduce it: I downloaded the last sample files from #48587 (comment), I created a new project and added the data from example.aef using the datasource manager and the delimited text provider and I am not having any issue re-opening the project.

The delimiter is correctly encoded ad %5Ct in the datasource in the QGS project file.

mlechner · 2023-02-14T15:59:14Z

@elpaso (@ed76) it does not happen when just working on a recent version. The problem occurs when you open a project that has been created with 3.16.x QGIS on Windows with a current version on Windows. Then the CSV-Layers in the project can not be found anymore if the delimiter is a tab. The problem has been detected when 3.22.x became LTR and a bunch of existing 3.16.x (former LTR) projects did and do not open without problems in the current LTR version. I am not effected personally, but colleagues of mine are (and other users as well, I guess - e.g. @ed76) and I think this is a problematic bug, as the normal QGIS update procedure within the LTR slot results in broken projects!

elpaso · 2023-02-14T16:09:34Z

@mlechner thank you for clarifying. So it is not related to the data being on a network folder?

ukulle · 2023-02-14T19:00:44Z

@elpaso (@ed76 @mlechner) - the reproduction of behavior seems to be complex. Maybe I have a special case here.

I created a new project (QGIS 3.22.14) and added the data from example.aef using the datasource manager and the delimited text provider.
I saved the project (project and datasource on a networl drive!).
I am not having issue re-opening the project (NOTE: always opening from main menu by STRG + O)
I open the project.qgz file by double click now no problem so far.
This is important - I save the project file.
I open the project.qgz file - and the issue occurs!

The difference found in the project files:
new created:
<datasource>file:./data/example.aef?type=csv&delimiter=%5Ct&skipLines=8& ...
after opened by double click:
<datasource>./data/example.aef?type=csv&delimiter=\t&skipLines=8& ...

In case the datasource is located on a local drive exactly this problem part does not occur.
different from above but same in the described test case:
<datasource>file:///D:/_temp/example.aef?type=csv&delimiter=%5Ct&skipLines=8& ...

I hope this can be reproduced and am looking forward to the feedback.

elpaso · 2023-02-14T20:17:34Z

I think I have found the problem.

when importing old projects, delimiter might be unencoded. Fix qgis#48587

elpaso · 2023-02-15T16:18:06Z

@ukulle I have fixed the encoding problem for local projects with #51881 , the network problem might be a different one.

If my PR does not fix the network problem please file a new issue (search if a similar issue does not yet exist first).

when importing old projects, delimiter might be unencoded. Fix #48587

mlechner added the Bug Either a bug report, or a bug fix. Let's hope for the latter! label May 17, 2022

Pedro-Murteira added Project Crash/Data Corruption Data Source Manager labels May 17, 2022

elpaso self-assigned this May 20, 2022

elpaso closed this as not planned Won't fix, can't repro, duplicate, stale May 20, 2022

elpaso reopened this May 25, 2022

elpaso added the Windows Related to Windows operating system label May 25, 2022

elpaso removed their assignment May 25, 2022

elpaso added the Feedback Waiting on the submitter for answers label Feb 14, 2023

elpaso self-assigned this Feb 14, 2023

elpaso added a commit to elpaso/QGIS that referenced this issue Feb 15, 2023

Fix delimited text delimiter encoding on windows

32df453

when importing old projects, delimiter might be unencoded. Fix qgis#48587

elpaso mentioned this issue Feb 15, 2023

Fix delimited text delimiter encoding on windows #51881

Merged

nyalldawson closed this as completed in #51881 Feb 21, 2023

nyalldawson pushed a commit that referenced this issue Feb 21, 2023

Fix delimited text delimiter encoding on windows

2ed8424

when importing old projects, delimiter might be unencoded. Fix #48587

qgis-bot pushed a commit that referenced this issue Feb 21, 2023

Fix delimited text delimiter encoding on windows

b4b2e1e

when importing old projects, delimiter might be unencoded. Fix #48587

nyalldawson pushed a commit that referenced this issue Feb 21, 2023

Fix delimited text delimiter encoding on windows

0c14b2f

when importing old projects, delimiter might be unencoded. Fix #48587

elpaso mentioned this issue Feb 24, 2023

QGIS won´t load delimited text layers #52016

Closed

2 tasks

agiudiceandrea mentioned this issue Sep 29, 2023

Occasional corruption of tab-delimited text layer when loading QGIS-file #54785

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CSV datasource not converted from 3.16 to 3.22 correctly - projects not usable anymore #48587

CSV datasource not converted from 3.16 to 3.22 correctly - projects not usable anymore #48587

mlechner commented May 17, 2022

mlechner commented May 17, 2022

mlechner commented May 17, 2022 •

edited

Loading

elpaso commented May 20, 2022

mlechner commented May 20, 2022

elpaso commented May 20, 2022

mlechner commented May 20, 2022

elpaso commented May 20, 2022

mlechner commented May 23, 2022

ukulle commented May 24, 2022 •

edited

Loading

mlechner commented May 24, 2022

ukulle commented May 24, 2022 •

edited

Loading

mlechner commented May 25, 2022 •

edited

Loading

agiudiceandrea commented May 25, 2022 •

edited

Loading

mlechner commented Jun 2, 2022 •

edited

Loading

mlechner commented Jun 2, 2022

mlechner commented Jun 22, 2022

grandebooz commented Jun 23, 2022

grandebooz commented Jun 23, 2022

pathmapper commented Jun 28, 2022

pathmapper commented Jun 28, 2022

agiudiceandrea commented Jun 28, 2022 •

edited

Loading

pathmapper commented Jun 29, 2022

mlechner commented Jul 21, 2022

ed76 commented Feb 14, 2023

mlechner commented Feb 14, 2023

elpaso commented Feb 14, 2023

mlechner commented Feb 14, 2023

elpaso commented Feb 14, 2023

ukulle commented Feb 14, 2023

elpaso commented Feb 14, 2023

elpaso commented Feb 15, 2023 •

edited

Loading

CSV datasource not converted from 3.16 to 3.22 correctly - projects not usable anymore #48587

CSV datasource not converted from 3.16 to 3.22 correctly - projects not usable anymore #48587

Comments

mlechner commented May 17, 2022

What is the bug or the crash?

Steps to reproduce the issue

Versions

Supported QGIS version

New profile

Additional context

mlechner commented May 17, 2022

mlechner commented May 17, 2022 • edited Loading

elpaso commented May 20, 2022

mlechner commented May 20, 2022

elpaso commented May 20, 2022

mlechner commented May 20, 2022

elpaso commented May 20, 2022

mlechner commented May 23, 2022

ukulle commented May 24, 2022 • edited Loading

mlechner commented May 24, 2022

ukulle commented May 24, 2022 • edited Loading

` using mixed delimiters

mlechner commented May 25, 2022 • edited Loading

agiudiceandrea commented May 25, 2022 • edited Loading

mlechner commented Jun 2, 2022 • edited Loading

mlechner commented Jun 2, 2022

mlechner commented Jun 22, 2022

grandebooz commented Jun 23, 2022

grandebooz commented Jun 23, 2022

pathmapper commented Jun 28, 2022

pathmapper commented Jun 28, 2022

agiudiceandrea commented Jun 28, 2022 • edited Loading

pathmapper commented Jun 29, 2022

mlechner commented Jul 21, 2022

ed76 commented Feb 14, 2023

mlechner commented Feb 14, 2023

elpaso commented Feb 14, 2023

mlechner commented Feb 14, 2023

elpaso commented Feb 14, 2023

ukulle commented Feb 14, 2023

elpaso commented Feb 14, 2023

elpaso commented Feb 15, 2023 • edited Loading

mlechner commented May 17, 2022 •

edited

Loading

ukulle commented May 24, 2022 •

edited

Loading

ukulle commented May 24, 2022 •

edited

Loading

`
using mixed delimiters

mlechner commented May 25, 2022 •

edited

Loading

agiudiceandrea commented May 25, 2022 •

edited

Loading

mlechner commented Jun 2, 2022 •

edited

Loading

agiudiceandrea commented Jun 28, 2022 •

edited

Loading

elpaso commented Feb 15, 2023 •

edited

Loading