Filter duplicates

[steffenkleinle]

Fixes #478

[steffenkleinle]

Removed duplicates (1) of '3D-Webtools, Obermichelbach, Obermichelbacher Str., 8'
Removed duplicates (1) of 'A.S.S. Bau GmbH, Wilhermsdorf, Schelmleite, 8'
Removed duplicates (1) of 'AloeVera-Deutschland, Obermichelbach, Weichselleite, 10'
Removed duplicates (1) of 'Alpinsport Basis GmbH, Garmisch-Partenkirchen, Bahnhofstraße, 4'
Removed duplicates (1) of 'Arber Hohenzollern Skistadion Schneider Events, Regen, Eschenweg, 7 c'
Removed duplicates (1) of 'Arberland Apotheken, Zwiesel, Angerstr., 32'
Removed duplicates (3) of 'Augenoptik Schuhmann, Stein, Hauptstr., 7'
Removed duplicates (1) of 'Autohaus Konrad Schmidt, Cadolzburg, Seckendorfer Str., 4'
Removed duplicates (1) of 'Autoschilder Christoph Kroschke GmbH, Amberg, Pfalzgrafenring, 3'
Removed duplicates (1) of 'Ballsportwelt, Zirndorf, Schützenstr., 9'
Removed duplicates (1) of 'Bambini-Events, Roßtal, Gundekarstr., 34'
Removed duplicates (1) of 'Bauernhausmuseum Lindberg, Lindberg, Lohwaldweg, 16'
Removed duplicates (3) of 'Baumgartner GmbH&Co.KG Sachverständigenbüro, Cadolzburg, Greimersdorfer Dorfstr., 19'
Removed duplicates (1) of 'Bavaria Filmstadt, Grünwald, Bavariafilmplatz, 7'
Removed duplicates (1) of 'bavariaraft, Ohlstadt, Enzianstraße, 1'
Removed duplicates (1) of 'Bayerischer Localbahnverein e.V. - Localbahnmuseum, Bayer. Eisenstein, Bahnhofstr., 44'
Removed duplicates (1) of 'Bayerisches Rotes Kreuz, Amberg, Amselweg, 30'
Removed duplicates (1) of 'BDZ-Immobilien, Zirndorf, Maximilianstr., 2'
Removed duplicates (3) of 'Blümm Gisela Immobilien, Tuchenbach, Fasanenstr., 6'
Removed duplicates (3) of 'Bodymed-Center, Oberasbach, Bahnhofstr., 47'
Removed duplicates (1) of 'Brillen Wiegand, Zirndorf, Nürnberger Str., 17'
Removed duplicates (1) of 'Crooklyn, Amberg, Lederergasse, 2-4'
Removed duplicates (1) of 'Cucinella - Kochen, Grillen & Genießen, Wolfratshausen, Obermarkt, 29'
Removed duplicates (1) of 'Diakonie Hochfranken - Diakonie am Campus, Hof, Südring, 96'
Removed duplicates (3) of 'Die Hörberatung, Oberasbach, Am Rathaus, 14'
Removed duplicates (3) of 'Die Hörberatung, Zirndorf, Angerzeile, 18'
Removed duplicates (1) of 'Digital Satelliten Fernsehen, Oberasbach, Leonhardstr., 1a'
Removed duplicates (1) of 'DOMINO, Amberg, Fleischbankgasse, 2'
Removed duplicates (1) of 'Dorint Sporthotel Garmisch-Partenkirchen, Garmisch-Partenkirchen, Mittenwalder Straße, 59'
Removed duplicates (1) of 'Dreifaltigkeits Apotheke, Amberg, Mosacherweg, 3a'
Removed duplicates (5) of 'Einrichtungswerkstätte Sauber, Stein, Gewerbering, 18'
Removed duplicates (1) of 'Eis- und Rollsportclub Regen e.V., Regen, Poschetsrieder Str., 45'
Removed duplicates (1) of 'Eissportzentrum Regen / ARBERLAND Betriebs gGmbH, Regen, Poschetsrieder Str., 45'
Removed duplicates (1) of 'Elektro Reichel, Stein, Mühlstr., 50'
Removed duplicates (1) of 'Elektrogeräte-Stahlwaren-Waffenhandel-Rudolf Angerer, Veitsbronn-Siegelsdorf, Friedenstr., 14'
Removed duplicates (1) of 'Erlebnispark Geiersthal*, Geiersthal, Im Gewerbegebiet, 15'
Removed duplicates (1) of 'Erlebnisschifffahrt Brombachsee, Pleinfeld, Am Anger, 10'
Removed duplicates (1) of 'Fahrschule Baur, Zirndorf, Nürnberger Str., 46'
Removed duplicates (1) of 'FC Ingolstadt 04 Fußball GmbH, Ingolstadt, Am Sportpark, 1'
Removed duplicates (1) of 'Ferienunterkunft bei Fam. Friedlein, Roßtal, Heideckweg, 4'
Removed duplicates (3) of 'Fitness & Gesundheits-Club OLYMP, Wilhermsdorf, Ansbacher Str., 4'
Removed duplicates (1) of 'Friseur Bachmann, Roßtal, Rathausgasse, 1'
Removed duplicates (1) of 'Friseure Back, Langenzenn, Obere Ringstr., 4a'
Removed duplicates (1) of 'Gaisch GmbH, Stein, Hofwiesenweg, 2'
Removed duplicates (1) of 'Gärtnerei der JVA Bayreuth, Bayreuth, Markgrafenallee, 49'
Removed duplicates (1) of 'Gasthof Sonne, Seehausen, Bahnhofstraße, 22'
Removed duplicates (1) of 'Gemeinde Frauenau, Frauenau, Rathausplatz, 4'
Removed duplicates (1) of 'Gemeinde Haibach - Freibad, Haibach, Schulstraße, 1'
Removed duplicates (1) of 'Gemeinde Leiblfing - Freibad, Leiblfing, Schulstr., 6'
Removed duplicates (1) of 'Gemeinde Oberschneiding - Naturbad, Oberschneiding, Pfarrer-Handwercher-Platz, 4'
Removed duplicates (1) of 'Gemeinde Straßkirchen - Freibad Straßkirchen u. Schambach, Straßkirchen, Lindenstr., 1'
Removed duplicates (1) of 'Geuder Bäckerei, Veitsbronn, Am Dorfplatz, 1'
Removed duplicates (1) of 'Gläserne Scheune*, Viechtach, Rauhbühl, 3'
Removed duplicates (1) of 'Grenzglashütte Bayer. Eisenstein, Bayer. Eisenstein, Bahnhofstr., 48'
Removed duplicates (1) of 'Heiltherapeutische Privatpraxis, Uffing a. Staffelsee, Murnauer Straße, 15'
Removed duplicates (1) of 'Heizung, Sanitär Kutrowatz GmbH, Cadolzburg, Sudetenstr., 15'
Removed duplicates (3) of 'Helpi, Zirndorf, Herrleinstr., 1'
Removed duplicates (1) of 'Hochland-Kino, Garmisch-Partenkirchen, Ludwigstraße, 90'
Removed duplicates (1) of 'Hotel Knorz, Zirndorf, Volkhardtstr., 18'
Removed duplicates (1) of 'Hotel Reubel, Zirndorf, Banderbacher Str., 27'
Removed duplicates (1) of 'Integrativer Reit und Fahrverein Loisachtal e.V., Eurasburg, Ziegelstadel, 6'
Removed duplicates (1) of 'Intersport-Lange GmbH, Amberg, Marktplatz, 1'
Removed duplicates (1) of 'Johann Schmidt GmbH, Roßtal, Gewerbering, 34'
Removed duplicates (1) of 'Keramik Kunst & Kleckserei, Lenggries, Marktstraße, 14'
Removed duplicates (1) of 'Kino Center im Lamm, Garmisch-Partenkirchen, Forstamtweg, 1'
Removed duplicates (1) of 'Kochgalarie, Cadolzburg, Marktplatz, 9'
Removed duplicates (1) of 'Kreisjugendring Straubing-Bogen, Straubing, Leutnerstr., 15'
Removed duplicates (1) of 'Krümel & Monster, Langenzenn, Würzburger Str., 6'
Removed duplicates (1) of 'Landhotel & Restaurant Böld, Oberammergau, König-Ludwig-Straße, 10'
Removed duplicates (1) of 'Landkreis Dillingen a.d.Donau, Dillingen a.d.Donau, Große Allee, 24'
Removed duplicates (1) of 'Landkreis Regen - Hallenbad Viechtach, Viechtach, Jahnstraße'
Removed duplicates (1) of 'Landkreis Straubing-Bogen - Hallenbad Bogen, Straubing, Leutnerstr., 15'
Removed duplicates (1) of 'Lars Pfaffendorf Heizungsbau, Wilhermsdorf, Weinbergstr., 17'
Removed duplicates (2) of 'Legoland Deutschland, Günzburg, LEGOLAND Allee'
Removed duplicates (1) of 'Leos Modellbahnstube, Siegelsdorf, Fürther Str., 27a'
Removed duplicates (1) of 'Lichtspiele-Kino zum Wohlfühlen, Großhabersdorf, Bachstr., 15'
Removed duplicates (1) of 'Linden-Apotheke, Veitsbronn, Fürther Str., 11'
Removed duplicates (1) of 'L´Osteria Amberg, Amberg, Franziskanergasse, 5'
Removed duplicates (1) of 'Luna-Apotheke, Oberasbach, Rothenburger Str., 24'
Removed duplicates (1) of 'Malermeisterbetrieb Kölbl, Rohrenfels, Hauptstraße, 49'
Removed duplicates (1) of 'Markt Bodenmais, Bodenmais, Bahnhofstr., 56'
Removed duplicates (1) of 'Markt Burgebrach, Burgebrach, Hauptstraße, 3'
Removed duplicates (3) of 'Markt Cadolzburg, Cadolzburg, Rathausplatz, 1'
Removed duplicates (1) of 'Markt Gaimesheim, Gaimersheim, Marktplatz, 3'
Removed duplicates (1) of 'Markt Kipfenberg, Kipfenberg, Marktplatz, 2'
Removed duplicates (1) of 'Markt Kösching, Kösching, Marktplatz, 1'
Removed duplicates (1) of 'Markt Mallersdorf-Pfaffenberg - Frei- und Hallenbad, Mallersdorf-Pfaffenberg, Steinrainer Str., 8'
Removed duplicates (1) of 'Markt Mitterfels, Mitterfels, Burgstr., 1'
Removed duplicates (3) of 'Maroni Dirndl, Cadolzburg, Zu den Eichen, 3a'
Removed duplicates (1) of 'Martin-Behaim-Apotheke, Oberasbach, Meißener Str., 49'
Removed duplicates (1) of 'Mediendesign Eckert, Cadolzburg, Marktplatz, 12'
Removed duplicates (1) of 'Montevia, Lenggries, Bergbahnstraße, 1'
Removed duplicates (5) of 'MqSol Heizung, Cadolzburg, Am Sand, 8'
Removed duplicates (1) of 'NähArt !, Oberasbach, Dresdener Str., 29a'
Removed duplicates (1) of 'Noris Werbung GmbH, Oberasbach, Hainbergstr., 32'
Removed duplicates (1) of 'Oberpfälzer Freilandmuseum, Nabburg, Neusath, 200'
Removed duplicates (1) of 'Oma´s Ladl, Amberg, Viehmarkt, 4'
Removed duplicates (3) of 'Optello-optik, Oberasbach, Rothenburger Str., 62b'
Removed duplicates (1) of 'Pese August ohG, Veitsbronn, Am alten Sportplatz, 1'
Removed duplicates (1) of 'Pflanzenhaus Schöner, Zirndorf, Jahnstr., 14'
Removed duplicates (1) of 'Post Hotel Mittenwald, Mittenwald, Obermarkt, 9'
Removed duplicates (1) of 'Powderworld Cross Country Skischule & Outdoorsport, Moosham, Am Bergl, 4'
Removed duplicates (1) of 'Praxis für Allgemeinmedizin Dr. Doru Popa, Veitsbronn, Nürnberger Str., 3a'
Removed duplicates (1) of 'Pröller-Skilift-Gesellschaft, Sankt Englmar, Klinglbach, 1'
Removed duplicates (3) of 'Rollershop Veitsbronn, Veitsbronn, Fürther Str., 16'
Removed duplicates (1) of 'ro-ro motorradkommunikation, Stein, Wilhelmstr., 1'
Removed duplicates (1) of 'Rückert Dieter - Galerie Art, Zirndorf, Ackerstr., 33'
Removed duplicates (1) of 'Schild Automobile und Handels GmbH, Veitsbronn, Seukendorfer Str., 11'
Removed duplicates (1) of 'Schloß-Apotheke, Cadolzburg, Hindenburgstr., 29'
Removed duplicates (1) of 'Schuh-Sport Blenk, Bischofsmais, Bergstr., 8'
Removed duplicates (1) of 'Schuh-Sport Katzendobler, Bogen, Stadtplatz, 33'
Removed duplicates (1) of 'Schulverband Hunderdorf - Hallenbad, Hunderdorf, Sollacher Str., 4'
Removed duplicates (2) of 'Schwarzfischer GmbH, Roding, Bahnhofstr., 12'
Removed duplicates (1) of 'SG-elektronik, Zirndorf, Siegfriedstr., 19'
Removed duplicates (1) of 'Shirtworld, Zirndorf, Schützenstr., 9'
Removed duplicates (1) of 'Silvia´s Friseursalon, Seukendorf, Fürther Str., 6'
Removed duplicates (1) of 'Sixt Autovermietung, Bad Tölz, Albert-Schäffenacker-Str., 1a'
Removed duplicates (1) of 'Skilift Farchant, Farchant, Esterbergstraße, 50'
Removed duplicates (1) of 'Skischule Isarwinkel, Lenggries, Enzianstraße, 3'
Removed duplicates (1) of 'Sporrer Technik e.K., Zwiesel, Langdorfer Str., 2'
Removed duplicates (1) of 'Sport & Mode Pöschl, Bayer. Eisenstein, Hauptstr., 39'
Removed duplicates (1) of 'Sport & Mode Pöschl, Zwiesel, Langdorfer Str., 3'
Removed duplicates (1) of 'Sport + Moden Wild, Geiselhöring, Landshuter Str., 15'
Removed duplicates (1) of 'Sport-Garmisch Meyer, Garmisch-Partenkirchen, Bankgasse, 7'
Removed duplicates (1) of 'Sportshop Bielmeier, Viechtach, Schmidstr., 16'
Removed duplicates (1) of 'Stadt Dachau, Dachau, Konrad-Adenauer-Str., 1'
Removed duplicates (1) of 'Stadt Geiselhöring -Frei- u. Hallenbad, Geiselhöring, Stadtplatz, 4'
Removed duplicates (1) of 'Stadt Gundelfingen a.d.Donau, Gundelfingen a.d.Donau, Prof.-Bamann-Str., 22'
Removed duplicates (7) of 'Stadt Langenzenn, Langenzenn, Friedrich-Ebert-Str., 7'
Removed duplicates (2) of 'Stadt Regen, Regen, Stadtplatz, 2'
Removed duplicates (1) of 'Stadt Schlüsselfeld, Schlüsselfeld, Marktplatz, 5'
Removed duplicates (1) of 'Stadtwerke Bogen - Freibad, Bogen, Stadtplatz, 56'
Removed duplicates (1) of 'Steeger Wolfgang (Reikigroßmeister), Oberasbach, Coburger Str., 7'
Removed duplicates (5) of 'Suto-Thaimassage, Veitsbronn, Stockäckerstr., 21'
Removed duplicates (1) of 'traininks- Trainig für Körper und Seele, Oberasbach, Donauschwabenstraße, 1'
Removed duplicates (1) of 'Ulrich Dienstleistungen Reinigungs-Geräte-Service, Oberasbach, Hopfenweg, 3'
Removed duplicates (5) of 'Unger Systemberatung, Cadolzburg, Gonnersdorf, 27'
Removed duplicates (1) of 'VHS Geretsried, Geretsried, Adalbert-Stifter-Straße, 13'
Removed duplicates (3) of 'Vitamin-Coaching Karin Pilz, Oberasbach, Milbenweg, 7a'
Removed duplicates (1) of 'Waldwipfelweg GmbH Sankt Englmar, Sankt Englmar, Maibrunn, 2a'
Removed duplicates (1) of 'Werbetechnik Sokoll, Roßtal, Eibenstr., 3'
Removed duplicates (7) of 'Zara-Partyservice, Stein, Gewerbering, 11'
Removed duplicates (1) of 'Zauncenter Zwart GmbH, Oberasbach, Schlossgasse, 1'
Removed duplicates (1) of 'Zentrum für Umwelt und Kultur, Benediktbeuern, Zeilerweg, 2'
Removed duplicates (1) of 'Zugspitzbowling, Garmisch-Partenkirchen, Zugspitzstraße, 61'
Removed duplicates (1) of 'Zugspitz-Kegelbahn, Garmisch-Partenkirchen, Zugspitzstraße, 67'
Removed duplicates (1) of 'Fränkisches Freilandmuseum Fladungen, Fladungen, Bahnhofstraße, 19'
Removed duplicates (1) of 'Optik Horn, Würzburg, Domstraße, 28'
Removed duplicates (1) of 'Kletterhalle Birkenfeld, Birkenfeld, Weidenmühle, 1'
Removed duplicates (1) of 'Jugendzentrum MainHaus/Minigolf Projekt, Marktheidenfeld, Lengfurter Straße, 26'
Removed duplicates (1) of 'Mezger GmbH + Co KG, Schweinfurt, Werner-von-Siemens-Str., 6'
Removed duplicates (1) of 'Multum in Parvo Opernhaus, Mering, Augsburger Str., 48'
Removed duplicates (1) of 'Schwimmverein Würzburg 05 e. V., Würzburg, Oberer Bogenweg, 1'
Removed duplicates (2) of 'Sparkasse Mainfranken Würzburg, Würzburg, Hofstraße, 7'
Removed duplicates (1) of 'Landkreis Erlangen-Höchstadt, Erlangen, Nägelsbachstr., 1'
Removed duplicates (1) of 'Optik Huth & Optometrie, Würzburg, Wenzelstr., 17'
Removed duplicates (1) of 'Toilettenservice Kraus, Reichertshofen, Forststraße, 5'
Removed duplicates (1) of 'Cadolzburger Welt Lager, Cadolzburg, Am Farnbach, 21'
Removed duplicates (1) of 'Playmobil – Fun Park, Zirndorf, Brandstätterstr., 2-10'
Removed duplicates (1) of 'OBI Baumarkt Regnitztal GmbH & Co. KG, Erlangen, Neumühle, 1'
Removed duplicates (2) of 'Rist Textilhandel Zubehör GmbH, Marktheidenfeld, Luitpoldstraße, 6-10'

[steffenkleinle]

Moved from old step Encode

[steffenkleinle]

It would perhaps be possible to find more duplicates with the tradeoff of possibly removing false positives (grouping by coordinates + fuzzy matching e.g. with levenshtein distance). Tried it, does not really work well imo as there are some edge cases where names are really similar but not identical. Therefore I went with exact matching by name + postal code + street, because of #467 this works well.
Sometimes we may take the wrong data but since the user may view the wrong data without this as well, I don't think that is a big problem.

[maxammann]

Looks good!

[michael-markl]

it would be nice to also log specifics about each entry when we mark them as "duplicate", maybe log the complete objects....
I wonder why there are so many duplicates. Do you have an idea? Are the entries in that case most times completely the same or do they differ slightly here and there?

[steffenkleinle]

it would be nice to also log specifics about each entry when we mark them as "duplicate", maybe log the complete objects.... I wonder why there are so many duplicates. Do you have an idea? Are the entries in that case most times completely the same or do they differ slightly here and there?

Will improve the logging (and log differing attributes only perhaps?). So for once lots of stores are once from the LK and once from the eak platform (not sure what it's called?). Also there are slight differences most of the time like a whitespace in phone numbers, a little bit adjusted discount, websites without protocol, stuff like that.

[maxammann]

So for once lots of stores are once from the LK and once from the eak platform (not sure what it's called?)

Hm there should be exactly one data source per Landkreis/Kreisfreie Stadt. Either they use an excel list or Freinet.

[steffenkleinle]

So for once lots of stores are once from the LK and once from the eak platform (not sure what it's called?)

Hm there should be exactly one data source per Landkreis/Kreisfreie Stadt. Either they use an excel list or Freinet.

At least in the test app-daten.xml it is that way (e.g. just have a look at the first duplicate that is sorted out: 3D-Webtools, once with <anbieter_name>Landkreis Fürth</anbieter_name> and once <p_eak_agentur_id>766</p_eak_agentur_id>). Of course one cannot be sure for the production app-daten.xml since we don't have these attributes there yet but it seems a lot like it.

[maxammann]

At least in the test app-daten.xml it is that way (e.g. just have a look at the first duplicate that is sorted out: 3D-Webtools, once with \u003Canbieter_name>Landkreis Fürth\u003C/anbieter_name> and once \u003Cp_eak_agentur_id>766\u003C/p_eak_agentur_id>). Of course one cannot be sure for the production app-daten.xml since we don't have these attributes there yet but it seems a lot like it

Yes, that makes sense. So the duplication is is caused by duplicate data sources (freinret+something else).
Can you maybe find some of such cases? Then I would report them upstream.

[steffenkleinle]

Yes, that makes sense. So the duplication is is caused by duplicate data sources (freinret+something else). Can you maybe find some of such cases? Then I would report them upstream.

See #485