ogr2ogr wfs to postgis discard some (random) field

[pieri70]

Hello
I am trying to import WFS layers to Postgis using Osgeo4W gdal (2.4.0-7) Windows 10 Pro x64.
I made a script to load a single WFS layer to a specific Postgis table, with overwrite action.
It works fine but not all WFS fields are created and then populated into Postgis table.

For example this WFS layer:

"C:\OSGeo4W64\bin\ogrinfo.exe" -so -nomd -noextent -fields=YES "WFS:http://serviziogc.regione.fvg.it/geoserver/wfs?service=WFS&request=GetFeature&version=1.1.0&typeName=IDROGRAF:CORSI_ACQUA" "IDROGRAF:CORSI_ACQUA"

gives me this field list:

gml_id: String (0.0) NOT NULL
CODICE_FVG: String (0.0)
TIPOLOGIA: Integer (0.0)
CARATTERISTICA: String (0.0)
CODICE_RD: Integer (0.0)
CODICE_PT: Integer (0.0)
CODICE_FORESTE: String (0.0)
ORDINE: Integer (0.0)
CORSO_ACQUA_PRINCIPALE: String (0.0)

These are the fields created by:

"C:\OSGeo4W64\bin\ogr2ogr.exe" -lco GEOMETRY_NAME=geom -overwrite -f "PostgreSQL" -a_srs EPSG:3045 PG:"dbname=postgisdatabase user=postgisuser host=postgishost" -nln schema.table "WFS:http://serviziogc.regione.fvg.it/geoserver/wfs?service=WFS&request=GetFeature&version=1.1.0&typeName=IDROGRAF:CORSI_ACQUA&srsName=EPSG:3045" -nlt PROMOTE_TO_MULTI

CREATE TABLE schema.table
(
ogc_fid serial NOT NULL,
gml_id character varying NOT NULL,
codice_fvg character varying,
tipologia integer,
caratteristica character varying,
codice_rd integer,
codice_pt integer,
codice_foreste character varying,
ordine integer,
corso_acqua_principale character varying,
geom geometry(MultiLineString,3045),
CONSTRAINT irdat_corsi_acqua_3045_pkey PRIMARY KEY (ogc_fid)
)
WITH (
OIDS=FALSE
);

If I load this specific WFS layer into Qgis it has many more fields and if I load the same WFS to Postgis trough Qgis/DBmanager I get all the WFS fields:

CREATE TABLE schema.table
(
id serial NOT NULL,
geom geometry(LineString,3045),
codice_fvg character varying,
natura character varying,
nome character varying,
tipologia double precision,
caratteristica character varying,
nome_ctr character varying,
nome_rd character varying,
nome_foreste character varying,
nome_consorzio character varying,
nome_1 character varying,
nome_2 character varying,
nome_3 character varying,
codice_rd double precision,
codice_pt double precision,
codice_foreste character varying,
ordine double precision,
note character varying,
corso_acqua_principale character varying,
se_anno_cad_data character varying,
CONSTRAINT corsidbmanager_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);

In BOLD missing fields

Same behaviour with other WFS layers both number and text fields

Is this normal or is there any problem with the script code or ogr?

Thanks
Pietro

[jratike80]

You can get more information about what GDAL is doing by adding --debug on into your command. Is seems that GDAL gets the schema by interpreting just the first feature from the WFS service. That feature does not have for example attribute NATURA. I do not quite understand what happens but it looks like GDAL is reading the full schema from the server with DescribeFeatureType but decides still to use the self-parsed .gfs schema instead

HTTP: Fetch(http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA)
WFS: Turn off loading of multiple layer definitions at a single time
WFS: http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA
HTTP: Fetch(http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA)
WFS: http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=GetFeature&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA&MAXFEATURES=1
HTTP: Fetch(http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=GetFeature&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA&MAXFEATURES=1)
GML: Using Expat reader
HTTP: Fetch(http://serviziogc.regione.fvg.it/geoserver/wfs?service=WFS&version=1.1.0&request=DescribeFeatureType&typeName=IDROGRAF%3ACORSI_ACQUA)
GML: Generating /vsimem/tempwfs_000001E93EBBC3E0/file.gfs file, ignoring /vsimem/tmp_gml_xsd_000001E93EC80A30.xsd
GDAL: GDALOpen(/vsimem/tempwfs_000001E93EBBC3E0/file.gml, this=000001E93EC80A30) succeeds as GML.
GDAL: GDALClose(/vsimem/tempwfs_000001E93EBBC3E0/file.gml, this=000001E93EC80A30)
WFS: http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=GetFeature&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA&RESULTTYPE=hits
HTTP: Fetch(http://serviziogc.regione.fvg.it/geoserver/wfs?SERVICE=WFS&REQUEST=GetFeature&VERSION=1.1.0&TYPENAME=IDROGRAF:CORSI_ACQUA&RESULTTYPE=hits)
GDAL: GDALClose(WFS:http://serviziogc.regione.fvg.it/geoserver/wfs?service=WFS&request=GetFeature&version=1.1.0&typeName=IDROGRAF:CORSI_ACQUA, this=000001E93E667990)

[pieri70]

Thanks jratike80
Is this behaviour normal?
Can I force Gdal to fetch the real field sequence?

[rouault]

Minimum fix pushed. The issue was the SE_ANNO_CAD_DATA attribute of type xsd:hexBinary, this type being not supported.

[pieri70]

Then it created a mess with other fields?

[rouault]

Then it created a mess with other fields?

yes, as soon as the OGR GML .xsd parser detects something it doesn't understand, it just stops. And it then fallbacks to analyzing the GML content itself without schema (which is also error prone)