Administrative boundaries not imported at all

[michael-hy]

When i try to import data from the following file:
http://download.geofabrik.de/europe/germany/bayern/oberbayern-latest.osm.bz2
(not very large 144MB, so you could try)
i don't get the (for me very import) administrative boundaries in the polygons table (perhaps there are one or two meaningless ones).

Example: There is no "Stadtbezirk 04 Schwabing-West" polygon.
But even the Boundary of the capital München is missing. All the administrative boundaries are missing. I did not check for other polygons or other database files yet, but i try to make a concrete example here, that you could trace it down.

I still had an old database, and there is everything well imported. I checked the ids of the missing polygons and there was something interesting. all these polygons had negative ids in my old database. In the xml file there were only positive ids (i guess you know this well, but this is all new for me). The ids of the xml file were the same like in the database but just positive ones.

I checked the xml file and i seem to miss this one (next to all the others):

<relation id="56389" version="25" timestamp="2012-12-16T09:21:11Z" changeset="14290051" uid="130472" user="fx99">
                <member type="way" ref="28823678" role="outer"/>
                <member type="way" ref="30033834" role="outer"/>
                <member type="way" ref="195085889" role="outer"/>
                <member type="way" ref="196473438" role="outer"/>
                <member type="way" ref="194705249" role="outer"/>
                <member type="way" ref="195085890" role="outer"/>
                <member type="way" ref="69685291" role="outer"/>
                <member type="way" ref="69685244" role="outer"/>
                <member type="way" ref="30101361" role="outer"/>
                <member type="way" ref="196473436" role="outer"/>
                <member type="way" ref="45048519" role="outer"/>
                <member type="way" ref="194705245" role="outer"/>
                <member type="way" ref="196167289" role="outer"/>
                <member type="way" ref="196473435" role="outer"/>
                <member type="way" ref="42941597" role="outer"/>
                <member type="way" ref="30033065" role="outer"/>
                <member type="way" ref="30033066" role="outer"/>
                <member type="way" ref="196328310" role="outer"/>
                <member type="way" ref="196328308" role="outer"/>
                <member type="relation" ref="66998" role=""/>
                <member type="relation" ref="66999" role=""/>
                <member type="relation" ref="67000" role=""/>
                <tag k="admin_level" v="9"/>
                <tag k="boundary" v="administrative"/>
                <tag k="is_in" v="München,Bayern,Bundesrepublik Deutschland,Europe"/>
                <tag k="name" v="Stadtbezirk 04 Schwabing-West"/>
                <tag k="note" v="München, Stadtbezirk 4, 3 Bezirksteile"/>
                <tag k="type" v="multipolygon"/>
        </relation>

The file can be found on:
http://download.geofabrik.de/europe/germany/bayern.html

I use the following command:
./osm2pgsql/osm2pgsql --cache 5000 --number-processes 7 -S osm2pgsql/default.style --latlong -H localhost -U michael -P 9201 -d osm oberbayern-latest.osm.bz2

My version of osm2pgsql:
./osm2pgsql/osm2pgsql --version
osm2pgsql SVN version 0.83.0 (64bit id space)

I tried with native ubuntu 12.04 libraries, but now i got the latest ones as you can see at the end of my post (ldd command).

I tried with postgis 1.5 and 2.0.

I use postgres 8.4.

I configured osm2pgsql with 32 bit space and 64 bit space. No difference. Before I switchted to this version of osm2pgsql i could import everything, but had other problems (with appending spain to germany database). I don't know this old version any more now.

here are the used libraries

ldd osm2pgsql
        linux-vdso.so.1 =>  (0x00007fff7b1ff000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fce528bb000)
        libpq.so.5 => /usr/lib/libpq.so.5 (0x00007fce5268f000)
        libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fce52333000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fce52123000)
        libgeos-3.4.0dev.so => /usr/local/lib/libgeos-3.4.0dev.so (0x00007fce51d89000)
        libproj.so.0 => /usr/lib/libproj.so.0 (0x00007fce51b46000)
        libprotobuf-c.so.0 => /usr/lib/libprotobuf-c.so.0 (0x00007fce51936000)
        liblua5.2.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.2.so.0 (0x00007fce51705000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fce51404000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fce51108000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fce50ef2000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fce50cd4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fce50915000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fce506b7000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fce502db000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fce5000d000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fce4fe09000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fce4fbca000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fce4f97b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fce4f777000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fce52ae3000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fce4f54e000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fce4f346000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fce4f141000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fce4ef25000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fce4ed17000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fce4eafb000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007fce4e8bd000)
        libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007fce4e601000)
        libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007fce4e382000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007fce4e17b000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007fce4def4000)
        libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007fce4dc54000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007fce4da20000)
        libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007fce4d80a000)
        libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007fce4d5f9000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fce4d3e7000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fce4d1e2000)
        libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007fce4cfb9000)
        libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007fce4cdaa000)
        libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007fce4cb5f000)
        libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fce4c8bc000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fce4c683000)

please help, i am really stuck here.
Thanks, Michael

[woodpeck]

The .bz2 file on the Geofabrik server does contain the relation you are looking for:

$ bzgrep 'id="56389"'  oberbayern-latest.osm.bz2
    <relation id="56389" version="25" timestamp="2012-12-16T09:21:11Z" changeset="14290051" uid="130472" user="fx99">

It seems your download is faulty. Don't be confused by the negative IDs, they are created by osm2pgsql automatically from positive relation IDs in the downloaded file. Also, you should use the .osm.pbf and not the .osm.bz2 - same data, smaller file, faster loading.

[michael-hy]

Thanks for your help but i do not understand completely what you mean. So you grep the file and then you get a matching line. So doesn't this mean that the relation is included? I did the greping again, look:

bzgrep -A 29 'id="56389"'  oberbayern-latest.osm.bz2                                                                                                                            
        <relation id="56389" version="25" timestamp="2012-12-16T09:21:11Z" changeset="14290051" uid="130472" user="fx99">
                <member type="way" ref="28823678" role="outer"/>
                <member type="way" ref="30033834" role="outer"/>
                <member type="way" ref="195085889" role="outer"/>
                <member type="way" ref="196473438" role="outer"/>
                <member type="way" ref="194705249" role="outer"/>
                <member type="way" ref="195085890" role="outer"/>
                <member type="way" ref="69685291" role="outer"/>
                <member type="way" ref="69685244" role="outer"/>
                <member type="way" ref="30101361" role="outer"/>
                <member type="way" ref="196473436" role="outer"/>
                <member type="way" ref="45048519" role="outer"/>
                <member type="way" ref="194705245" role="outer"/>
                <member type="way" ref="196167289" role="outer"/>
                <member type="way" ref="196473435" role="outer"/>
                <member type="way" ref="42941597" role="outer"/>
                <member type="way" ref="30033065" role="outer"/>
                <member type="way" ref="30033066" role="outer"/>
                <member type="way" ref="196328310" role="outer"/>
                <member type="way" ref="196328308" role="outer"/>
                <member type="relation" ref="66998" role=""/>
                <member type="relation" ref="66999" role=""/>
                <member type="relation" ref="67000" role=""/>
                <tag k="admin_level" v="9"/>
                <tag k="boundary" v="administrative"/>
                <tag k="is_in" v="München,Bayern,Bundesrepublik Deutschland,Europe"/>
                <tag k="name" v="Stadtbezirk 04 Schwabing-West"/>
                <tag k="note" v="München, Stadtbezirk 4, 3 Bezirksteile"/>
                <tag k="type" v="multipolygon"/>
        </relation>

Thanks for the tip with pbf format. But can you also extract it on the fly for osmosis, then filter and then compress it again. All through pipes (like: bzcat myfile | osmosis filter stuff | bzip > myfilteredfile). I dont have much hdd space, so i need to process stuff in ram.

[woodpeck]

Ah, sorry. I misunderstood - I thought that you were missing the relation in the XML file! But now I see the relation was there in your XML but not imported.

[apmon]

It works with hstore enabled (-j) but not without. I'll need to investigate what happened there, but I am traveling, so won't get to it until next week.