Metadata Broker

Searching and finding within International Data Spaces

Searching for suitable data and components is a major challenge – not only in distributed systems such as International Data Spaces. Familiar search engines from the World Wide Web and their business models can and will not be used within International Data Spaces. Self-determined control of data with equal rights is diametrically opposed to the current development of central information gatekeepers. International Data Spaces therefore need decentralized solutions with equal rights for publishing and searching for resources and components.

The IDS Metadata Broker is a comprehensive connector addressing these issues. It provides the necessary interfaces for communicating with any other International Data Spaces connector. More specifically, it is capable of handling messages from IDS connectors, indicating a status update, such as new data being available. Self-descripting metadata of connectors are automatically indexed and can be optionally restricted by means of usage control policies. Search functionality encompasses a full-text search, filter options based on the International Data Spaces information model, and full queries using SPARQL, the standardized query language in the semantic web. Human user friendly access is granted via a web interface.

The IDS Metadata Broker supports every interaction defined in the process layer, the descriptions defined in the information layer, and the architectures listed in the system layer. A metadata broker can provide additional services when needed. These can then be described as well with machine-readable descriptions compliant with the IDS information model. For example, a broker can regularly execute heartbeats to detect inactive connectors. The storage of quality of service (QoS) metrics or payment models can be implemented as well.

Reference for search and registration functions within nternational Data Spaces

The Metadata Broker constitutes the reference implementation for registration and search functionality compliant with International Data Spaces. Therefore it also follows the generic connector architecture as described in the reference architecture model. With its modular structure consisting of multiple Docker containers, it separates its functionality into independent units which can be maintained separately. Here the core container is the central, in-house development, covering external interactions within International Data Spaces and the internal exchange of messages. A Fuseki triple store ensures persistent data storage in RDF together with the efficient full-text search capabilities of ElasticSearch.

Container Management: Docker
Message router: Apache Camel
Webseiten framework: React

Test environment

A sandbox instance is available to IDSA members through the IDS Lab at the Fraunhofer Institute for Software and Systems Engineering ISST. The respective latest version is available there as a test environment and for connector development. Contact us for an optimal access model which best fits to your requirements.