The OSMF Data Working Group has removed roughly 9'000 street and 1'000 place names from the Swiss OSM dataset, because there is very good reason to believe that they had been derived from address data from the GWR (https://www.housing-stat.ch/) which at the time was not legal for us to use and currently has the same status, more on that later.
Most affected is likely French-speaking and central Switzerland and naturally mainly such names that are not easily surveyable (which is why they were missing in the first place). Nothing was removed in the Canton Berne as the GWR data for there is essentially the same as the cantonal address data that we have access too.
The best course of action is simply to get off that couch and go out and survey as far as possible what was "lost", it should be possible from comparing http://qa.poole.ch/ch-roads/list-2017-08-01.html with http://qa.poole.ch/ch-roads/list-2017-08-04.html to determine how much your local area has been affected.
Quickly back to the current situation with the GWR. As you may know on July the 1st the revised GWR ordinance entered in to force, unluckily it seems that I was right in being skeptical in http://www.openstreetmap.org/user/SimonPoole/diary/41672 as, as it seems now, the main effect of the revision seems to have been to open up new business opportunities for SwissTopo but even that will not happen before 2020.
All in all this means that until further notice use of the GWR address data on both the SwissTopo WMS server and map.geo.admin.ch (and most other data there) is completely off limits.
Naturally additional political pressure on the federal counsel to live up to their announcement https://www.admin.ch/gov/de/start/dokumentation/medienmitteilungen.msg-id-66... with real instead of Fake-OpenData is more than welcome.
Simon
I quickly threw a spreadsheet together that gives an overview of the damage (column H), which is mainly where I suspected plus Ticino. 750 municipalities in total.
See http://he.poole.ch/GWR-redaction.ods
See http://qa.poole.ch/ for roads completely without a name, some of the now "missing" names are likely to be false positives (OSM has a surveyed name that doesn't agree with what is in the GWR), how best to handle these, see: http://wiki.openstreetmap.org/wiki/OSM_-_GWR_Street_and_Place_Names_Comparis...
Simon
I never use GWR except for a fun QA after survey (I already found 3 diff between ground reality and GWR). Of course I always encode ground reality found by survey, never GWR data
But I still doesn't understand the cleanup coverage
example belmont-broye (FR) On http://qa.poole.ch/ch-roads/list-2017-08-01.html 117 OSM Roads 107 GWR Roads On http://qa.poole.ch/ch-roads/list-2017-08-04.html 100 OSM Roads 98 GWR Roads
I doubt that the municipality has removed 9 roads. What happened ?
Given the large number of errors in osm I already found by survey there, I also doubt that someone used GWR for 17 roads. For at least 1 street, the (removed) name in osm was wrong... I remember it because it is on my todo list. so I doubt that someone use GWR to filling a wrong info. I found an affected road that I survey 11 days ago. the name is removed before a crossing and still exist after the crossing. I am not the author of the name of this road. But I survey it later to add housenumber, building type, sign, lamp, sidewalk, lit, ... I of course put a source=survey on the changeset. What todo to prevent a verified information from being deleted because someone made a mistake a few years ago ? Because if the named was not set, I would have added myself when I survey the whole road. It would be absurd to have to backup everything I see only in case... Besides my survey has not been censored in the history, the deletion of the name is for the current version and the version before mine.
For another road, it is much more annoying because after "cleanup", the road have a wrong name... I found it by chance. Is it necessary therefore to tag all the streets modified with a fixme="check name" ? Could DWG avoid doing revert so invisible ? no name at all is easy to find and survey. But a restored bad name is hard to find. We can't survey all street in the country just in case that a previous fix was lost. I feel that the cleaning was too radical. They made 27428 changes many of which are invisible in our comparison. How was the selection made? Are you talking with DWG ? How did you learn the revert ?
It is also amazing that openaddresses.io uses the export of addresses made by Fribourg but that these same data are not open for osm. This seems to be the same situation as you describe for Bern, no ?
Regards, Marc
Le 04. 08. 17 à 20:26, Simon Poole a écrit :
The OSMF Data Working Group has removed roughly 9'000 street and 1'000 place names from the Swiss OSM dataset, because there is very good reason to believe that they had been derived from address data from the GWR (https://www.housing-stat.ch/) which at the time was not legal for us to use and currently has the same status, more on that later.
Most affected is likely French-speaking and central Switzerland and naturally mainly such names that are not easily surveyable (which is why they were missing in the first place). Nothing was removed in the Canton Berne as the GWR data for there is essentially the same as the cantonal address data that we have access too.
The best course of action is simply to get off that couch and go out and survey as far as possible what was "lost", it should be possible from comparing http://qa.poole.ch/ch-roads/list-2017-08-01.html with http://qa.poole.ch/ch-roads/list-2017-08-04.html to determine how much your local area has been affected.
Quickly back to the current situation with the GWR. As you may know on July the 1st the revised GWR ordinance entered in to force, unluckily it seems that I was right in being skeptical in http://www.openstreetmap.org/user/SimonPoole/diary/41672 as, as it seems now, the main effect of the revision seems to have been to open up new business opportunities for SwissTopo but even that will not happen before 2020.
All in all this means that until further notice use of the GWR address data on both the SwissTopo WMS server and map.geo.admin.ch (and most other data there) is completely off limits.
Naturally additional political pressure on the federal counsel to live up to their announcement https://www.admin.ch/gov/de/start/dokumentation/medienmitteilungen.msg-id-66... with real instead of Fake-OpenData is more than welcome.
Simon
talk-ch mailing list talk-ch@openstreetmap.ch http://lists.openstreetmap.ch/mailman/listinfo/talk-ch
Am 05.08.2017 um 03:22 schrieb marc marc:
I never use GWR except for a fun QA after survey (I already found 3 diff between ground reality and GWR). Of course I always encode ground reality found by survey, never GWR data
But I still doesn't understand the cleanup coverage
example belmont-broye (FR) On http://qa.poole.ch/ch-roads/list-2017-08-01.html 117 OSM Roads 107 GWR Roads On http://qa.poole.ch/ch-roads/list-2017-08-04.html 100 OSM Roads 98 GWR Roads
I doubt that the municipality has removed 9 roads. What happened ?
No, that is the difference between what had names -before- the redaction and -after- so 9 names have been removed from OSM (not the roads themselves).
The mapper in question was uncooperative and didn't help with determining what needed to be removed and what not, so there are likely to be a couple of false positives (I did sport check a couple of the proposed redactions and they all looked correct so the number of errors is likely to be small).
...
For another road, it is much more annoying because after "cleanup", the road have a wrong name... I found it by chance.
That is to be expected, as typically it will have reverted to what is was before the name was added from the incompatible source, and besides having no name, a stretch of road having the wrong name is naturally possible (typically happens over municipality borders or when a street hasn't been surveyed to the end and it changes names somewhere along the road).
Is it necessary therefore to tag all the streets modified with a fixme="check name" ? Could DWG avoid doing revert so invisible ? no name at all is easy to find and survey. But a restored bad name is hard to find. We can't survey all street in the country just in case that a previous fix was lost. I feel that the cleaning was too radical. They made 27428 changes many of which are invisible in our comparison. How was the selection made? Are you talking with DWG ? How did you learn the revert ?
It is also amazing that openaddresses.io uses the export of addresses made by Fribourg but that these same data are not open for osm. This seems to be the same situation as you describe for Bern, no ?
openaddresses.io is for all practical purposes not legally usable (in any context not just OSM). I've pointed out a couple of such cases which they fixed after long discussions, but they continue to not vet and police their incoming sources (contrary to what OSM does) and I don't see why we or I should run after a competing project that continues to cheat in a big way (for example they also distribute illegally obtained address data for the Canton Aargau).
Simon
Hello,
Le 05. 08. 17 à 13:00, Simon Poole a écrit :
117 OSM Roads 107 GWR Roads 100 OSM Roads 98 GWR Roads I doubt that the municipality has removed 9 roads. What happened ?
No, that is the difference between what had names -before- the redaction and -after- so 9 names have been removed from OSM (not the roads themselves).
107 -> 98 GWR Roads : I thought it means that 9 names have been removed in the GWR database (independently of the revert in osm) and ask what can be the reason for this decrease of GWR Roads in GWR database for a municipality that hasn't announced any destroyed/merged routes ?
the number of errors is likely to be small).
I make a query to quantify my feeling about the quality of the revert
In Fribourg, way last modified by dwg since 2017-08-01 1163 have now no name -> easy to survey 221 still have a (supposed wrong) name -> what do you think if I add a fixme "name altered by DWG. survey needed" ? it 'll help to warn where a new survey is needed to fix a typo or a wrong road limit.
In Domdidier half of the roads are involved at least partly. 3/5 of the affected road with a wrong name have been survey by myself AFTER the problematic import. so I suppose that survey changeset done after the import are not taken into account. 1/5 of the affected road have now a correct name so the deleted name wasn't imported from GWR. so IMHO in Domdidier, the revert hit ratio is 20% problematic data removed <> 60% survey data destroyed <> 20% not imported data removed IMHO I find that an error rate 4 times higher than what was problematic is not as small as you think. I'm of course unable to do this at the whole country scale.
A much more important issue : what todo in the future to prevent a verified information by survey from being deleted because someone made a mistake a few months/years ago ? Because if the named of affected road was not set or wrong, I would have added/fix myself when I survey the whole road. Should we backup all changeset before upload ? It would be problematic to replay them a few months or years later without conflicting with the changes made in the meantime by the others,
I had a quick look at what stolenaddresses.io is doing wrt Fribourg and they are scraping a non-public arcgis server to generate the dataset.
Thanks for the check. I 'll make another threat in a few day about opendata done by Bern and how try go the same for Fribourg.
Regards, Marc
Am 13.08.2017 um 14:55 schrieb marc marc:
In Domdidier half of the roads are involved at least partly. 3/5 of the affected road with a wrong name have been survey by myself AFTER the problematic import. so I suppose that survey changeset done after the import are not taken into account. 1/5 of the affected road have now a correct name so the deleted name wasn't imported from GWR.
Could you give an example? In general the redaction process tries to avoid breaking non involved changesets. In any case we may be able to fix this up automatically if the reason is clear.
Simon
Le 13. 08. 17 à 15:07, Simon Poole a écrit :
Am 13.08.2017 um 14:55 schrieb marc marc:
In Domdidier half of the roads are involved at least partly. 3/5 of the affected road with a wrong name have been survey by myself AFTER the problematic import. so I suppose that survey changeset done after the import are not taken into account.
Could you give an example?
road Vy-d'Avenches this part have been reverted https://www.openstreetmap.org/way/237594016 hidden v3 seems to be mine done by survey in https://www.openstreetmap.org/changeset/49451984 You can see the correct typo (a "-" missing) in changeset comment, in addr:street on building and also in another part of this road that doesn't have been reverted version #8 of https://www.openstreetmap.org/way/128523419/history
another affected part of the same road https://www.openstreetmap.org/way/43590043 hidden v6 seems to be mine done by survey in https://www.openstreetmap.org/changeset/49477530
we may be able to fix this up automatically if the reason is clear.
I have the local knowledge and working to fix all affected street in Domdidier and can survey for typos.
But I think that changesets made by DWG need to be review/improve because if I found many surveyed datas lost in this village, it is likely that it also happen at the country scale.
Let me know if you want I freeze my fix while you're talking with DWG (I got no reply to the message I wrote to him a few days ago).
Regards, Marc
I'll take this up with Frederik asap, right now a number of relevant people are in Japan so things are a bit difficult and I can't check redacted changesets in a simple way either.
Simon
Am 13.08.2017 um 21:09 schrieb marc marc:
Le 13. 08. 17 à 15:07, Simon Poole a écrit :
Am 13.08.2017 um 14:55 schrieb marc marc:
In Domdidier half of the roads are involved at least partly. 3/5 of the affected road with a wrong name have been survey by myself AFTER the problematic import. so I suppose that survey changeset done after the import are not taken into account.
Could you give an example?
road Vy-d'Avenches this part have been reverted https://www.openstreetmap.org/way/237594016 hidden v3 seems to be mine done by survey in https://www.openstreetmap.org/changeset/49451984 You can see the correct typo (a "-" missing) in changeset comment, in addr:street on building and also in another part of this road that doesn't have been reverted version #8 of https://www.openstreetmap.org/way/128523419/history
another affected part of the same road https://www.openstreetmap.org/way/43590043 hidden v6 seems to be mine done by survey in https://www.openstreetmap.org/changeset/49477530
we may be able to fix this up automatically if the reason is clear.
I have the local knowledge and working to fix all affected street in Domdidier and can survey for typos.
But I think that changesets made by DWG need to be review/improve because if I found many surveyed datas lost in this village, it is likely that it also happen at the country scale.
Let me know if you want I freeze my fix while you're talking with DWG (I got no reply to the message I wrote to him a few days ago).
Regards, Marc _______________________________________________ talk-ch mailing list talk-ch@openstreetmap.ch http://lists.openstreetmap.ch/mailman/listinfo/talk-ch
Am 05.08.2017 um 03:22 schrieb marc marc:
... It is also amazing that openaddresses.io uses the export of addresses made by Fribourg but that these same data are not open for osm. This seems to be the same situation as you describe for Bern, no ? ...
Just to confirm: I had a quick look at what stolenaddresses.io is doing wrt Fribourg and they are scraping a non-public arcgis server to generate the dataset.
-
Hubert -
marc marc -
Simon Poole