elementsFullHistory: geometry type filter not applied on all returned features

[SlowMo24]

When using the fullHistory endpoint elements that do not fully adhere to the set filter may be returnd if the filter violation appears after an edit. E.g. here: curl -X POST "https://api.ohsome.org/v1/elementsFullHistory/geometry?bboxes=7.770778%2C47.306056%2C7.816097%2C47.324619&clipGeometry=false&filter=geometry%3Apolygon%20and%20id%3A6648645&properties=tags%2Cmetadata&time=2020-05-01%2C2020-05-07" -H "accept: application/json" the objects geometry gets destroyed rendering it a multilinestring rather than a polygon which is returned by the api but should be filtered.

[tyrasd]

this works as expected with the old types/keys/values filter syntax, so it must be either a regression in #86 or introduced with the addition of the filter parameter.

[tyrasd]

related code lines on OSHDB side of things: https://github.com/GIScience/oshdb/blob/89530f3/oshdb-api/src/main/java/org/heigit/ohsome/oshdb/api/mapreducer/MapReducer.java#L773-L774 – for contributions only one geometry (either the "before" or "after" state must match the given filter). That means the oshome API must do additional checks before sending such features out to the client.

[SlowMo24]

for contributions only one geometry (either the "before" or "after" state must match the given filter)

whats the reason for that behaviour and not applying the filter to both states?

[tyrasd]

it's because the contributions exist for finding out about edits to certain features. For example, say, you want to find out about all changes to forests. You would certainly be interested in finding out about when a features was changed from landuse=forest to landuse=industrial. There the tag filter matches only the "before" state. The same is happening here: the geometry type only needs to match one of the before-after pair, so one can find out about changes of the geometry type.

//edit: there is one slight difference though in how tag and geometry filters are applied: if a tag filter only matches the "before" or "after" state of a contribution, the type of the contribution would be DELETION or CREATION, respectively. While it is "only" a GEOMETRY_CHANGE for the geometry-type filters. The reason for this is that the geometry filters are applied in a processing step quite a bit after where the tag filters are handled. This structural limitation might be something to consider fixing in the future of the OSHDB, but it is something which we have to live with for now.

[SlowMo24]

that makes sense and your edit is in my opinion worth mentioning in the oshdb-docs! (although you know I have a different opinion on the contribution types: GIScience/oshdb#113)

to me it seems that also oshome-api cannot handle this but rather should update the docs to make the user handle this appropriately? Otherwise, as you have noted, we would obscure some states from the user. What about something .ike: geometry:XXX -> applying a geometry filter to the fullHistory or contribution endpoint will not ensure only data passing this filter will reach the client. Contributions may change an object from an object passing the geometry filter to one which is not. In that case both states are returned and further filtering has to take place on the client.

[tyrasd]

to me it seems that also oshome-api cannot handle this

My educated guess is that simply applying the same filter also on the geometry of the potentially-to-be-returned the geojson features should be sufficient to fix this here.

[KowatschFab]

I see.. so it still works for the old parameters as expected? Then I'd say it must be related to the filter yes, otherwise it could also be related to the combination of the elementsFullHistory and contributions processing in the ohsome API.

About having features in the contributions response that are outside of the filter @SlowMo24 : I don't see a need for action here, because just as @tyrasd explained in his example with the landuse feature: this should be the expected behavior. If I want to analyzse all the contributions related to a specific tag, then I would also like to know when a feature changed and potentially also to which specific tag. Same goes when the geometry of a feature changes and it gets moved outside of my area of interest.

So for elementsFullHistory this seems to be a bug that should be fixed. For contributions I understand it as the wanted behavior.

[SlowMo24]

I wasn't event thinking of the contribution endpoint yet but for me both endpoints are quite synonyms so I would expect them to return the same result. Adding a DELETION (as in 'valid-to') to the fullHistory if the geometry filter is no longer passed is fine. I just recommend to either make the contribution endpoint return the same (DELETION + GEOMETRY_CHANGE) or update the documentation to notify the user about this subtle difference.

[KowatschFab]

@tyrasd any progress on this issue here? The 1.4 milestone is due next week and you self-assigned this issue a couple of weeks ago, that's why I'm asking.

[tyrasd]

you self-assigned this

sorry, I must have clicked that by mistake (or because I assumed at first that it might be an upstream issue in the OSHDB). did you try the fix proposed in #109 (comment)?