extra tags not loading from json file

[azizghuloum]

Hello,

I have my pbf file loaded into nominatim. I extracted the entries for photon into a json file as follows:

java -jar photon.jar -nominatim-import -extra-tags ALL -host localhost -port 5432 -database nominatim -user nominatim -password password -json photon.json && gzip photon.json

I checked that the resulting file does contain the extra tags:

# zcat photon.json.gz | grep extra | head
 {"type":"Place","content":[{"place_id":317420,"object_type":"W","object_id":233138627,"categories":["osm.place.islet"],"rank_address":0,"importance":0.1333433333333333,"name":{"name":"Île Marguerite"},"extra":{"survey:date":"07/02/2007"},"centroid":[141.3781156413658,-66.78827175],"bbox":[141.3769155,-66.7875672,141.3792967,-66.7889463]}]}
 {"type":"Place","content":[{"place_id":317148,"object_type":"W","object_id":233138744,"categories":["osm.place.islet"],"rank_address":0,"importance":0.1333433333333333,"name":{"name":"Île Curie"},"extra":{"survey:date":"07/02/2007"},"centroid":[140.04214554756481,-66.6442665],"bbox":[140.0402396,-66.6434839,140.0447432,-66.6449062]}]}
 {"type":"Place","content":[{"place_id":317408,"object_type":"W","object_id":233135107,"categories":["osm.place.islet"],"rank_address":0,"importance":0.1333433333333333,"name":{"name":"Île Lattanzi"},"extra":{"survey:date":"07/02/2007"},"centroid":[139.95994216210897,-66.65320955000001],"bbox":[139.9583792,-66.6519952,139.9608684,-66.6542789]}]}
 {"type":"Place","content":[{"place_id":317179,"object_type":"W","object_id":233135106,"categories":["osm.place.islet"],"rank_address":0,"importance":0.1333433333333333,"name":{"name":"Îlot des Hydrographes"},"extra":{"survey:date":"07/02/2007"},"centroid":[139.9971133823899,-66.6704738],"bbox":[139.9962708,-66.6700533,139.998037,-66.6709346]}]}
...

I import that file into opensearch like this:

if [ -f /data/photon.json.gz ]; then
  echo "Importing /data/photon.json.gz"
  zcat /data/photon.json.gz | java -jar photon.jar \
    -nominatim-import -extra-tags ALL \
    -transport-addresses 'opensearch:9200' \
    -import-file -
  echo "Import done"
else
  echo "Nothing to import."
fi

It does import the data into opensearch, but no extra tags present. Here is a link to the opensearch dashboard:

http://172.105.91.247:5601/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'31053930-57ad-11f0-a309-b103e7b8bee0',view:discover))&_q=(filters:!(),query:(language:kuery,query:''))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))

And a screenshot showing some data, and again, no extra tags:

Since the import command line is simple enough, I don't see how I might've messed it up, but I could've.

Digging in the code, I spotted this line which looks wrong, but I don't think it affects my issue (or maybe it does).

photon/src/main/java/de/komoot/photon/CommandLineArgs.java

Line 155 in df01eda

public boolean isExtraTagsSet() { return this.extraTags == null; }

Any help is appreciated. Thanks in advance.

[azizghuloum]

Digging further and taking note that extra-tags is not saved/loaded in

photon/app/es_embedded/src/main/java/de/komoot/photon/Server.java

Lines 237 to 247 in df01eda

	public void saveToDatabase(DatabaseProperties dbProperties) throws IOException {
	final XContentBuilder builder = XContentFactory.jsonBuilder().startObject().startObject(BASE_FIELD)
	.field(FIELD_VERSION, DATABASE_VERSION)
	.field(FIELD_LANGUAGES, String.join(",", dbProperties.getLanguages()))
	.field(FIELD_IMPORT_DATE, dbProperties.getImportDate() instanceof Date ? dbProperties.getImportDate().toInstant() : null)
	.field(FIELD_SUPPORT_GEOMETRIES, Boolean.toString(dbProperties.getSupportGeometries()))
	.endObject().endObject();
	esClient.prepareIndex(PhotonIndex.NAME, PhotonIndex.TYPE).
	setSource(builder).setId(PROPERTY_DOCUMENT_ID).execute().actionGet();
	}

and

photon/app/es_embedded/src/main/java/de/komoot/photon/Server.java

Lines 258 to 284 in df01eda

	public DatabaseProperties loadFromDatabase() {
	GetResponse response = esClient.prepareGet(PhotonIndex.NAME, PhotonIndex.TYPE, PROPERTY_DOCUMENT_ID).execute().actionGet();
	if (!response.isExists()) {
	throw new UsageException("Cannot find database properties. Your database version is too old. Please reimport.");
	}
	Map<String, String> properties = (Map<String, String>) response.getSource().get(BASE_FIELD);
	if (properties == null) {
	throw new UsageException("Found database properties but no '" + BASE_FIELD +"' field. Database corrupt?");
	}
	final var dbProps = new DatabaseProperties();
	dbProps.setDatabaseVersion(properties.get(FIELD_VERSION));
	final String langString = properties.get(FIELD_LANGUAGES);
	dbProps.setLanguages(langString == null ? null : langString.split(","));
	String importDateString = properties.get(FIELD_IMPORT_DATE);
	dbProps.setImportDate(importDateString == null ? null : Date.from(Instant.parse(importDateString)));
	dbProps.setSupportGeometries(Boolean.parseBoolean(properties.get(FIELD_SUPPORT_GEOMETRIES)));
	return dbProps;
	}

even though it is part of the database properties:

photon/src/main/java/de/komoot/photon/DatabaseProperties.java

Line 23 in df01eda

private ConfigExtraTags extraTags = new ConfigExtraTags();

This might've been missed.

[azizghuloum]

Tags are dumped as "extra"

photon/src/main/java/de/komoot/photon/json/JsonDumper.java

Line 183 in df01eda

dbProperties.configExtraTags().writeFilteredExtraTags(writer, "extra", doc.getExtratags());

They are read from "extratags"

photon/src/main/java/de/komoot/photon/json/NominatimPlaceDocument.java

Lines 146 to 149 in df01eda

	@JsonProperty("extratags")
	void setExtratags(Map<String, String> extratags) {
	doc.extraTags(extratags);
	}

Right?

[azizghuloum]

Hacked it with sed:

zcat /data/tmp-photon.json.gz | sed 's/"extra":/"extratags":/' | java -jar photon.jar -nominatim-import -extra-tags ALL -transport-addresses 'opensearch:9200' -import-file -

Changing one or the other of the two lines in the previous comment should fix it.

[lonvia]

Thanks for investigating and finding the right cause. The json reader is wrong here. Patch coming and I'll do a 0.7.2 release with the fix shortly.

[azizghuloum]

Thank you @lonvia. Much appreciated.