Architecture

An overview of the architecture of esy-osmfilter is shown in Fig.2.

The OSM pbf-file is read with the esy-osm-pbf library. The filtering process is divided into a prefilter phase and a main filter phase. The customisable pre-filter can contain different filter rules for all OSM items types (nodes, ways, and relations).

alt text Fig.2: Working-flow of the esy-osmfilter library.

The internal block structure of OSM-pbf files allows to accelerate the prefilter process via internal parallization. The python multiprocessing library therefore calls multiple instances of the esy-osmfilter.run_filter method and unifies the results. The following OSM items are stored in the Data dictionary as a result:

  • nodes

  • ways + referenced way nodes

  • relations + referenced relation nodes+ member ways + referenced member way nodes

The prefiltering of the data with very permeable prefilter is recommended, especially for larger files. This allows to further accelerate the main filtering process for relevant items in the future. Therefore, the output Data dictionary can be stored to a pickle file, from which it can be reloaded at any time. Further, the output can be stored as JSON-file to make it human-readable.

In the main filtering phase, the user can apply a combination of white-filters and black-filters to identify specific items from Data. These items will be stored in the Elements dictionary, which can equally be written to a pickle-file for reuse and to a JSON-file for human-readability. The “Elements” dictionary receives only the actual items from the Data dictionary, which fulfill the main filter rules. So, specific OSM-items are stored to the Elements dictionary, but none of the referenced nodes or relation members. However, they can still be accessed from the Data dictionary by their OSM id.