loading page

Corpus Processing Service: A Knowledge Graph Platform to perform deep data exploration on corpora.
  • Peter Staar,
  • Michele Dolfi,
  • Christooh Auer
Peter Staar
IBM Zurich Research Laboratory
Author Profile
Michele Dolfi
IBM Zurich Research Laboratory
Author Profile
Christooh Auer
IBM Zurich Research Laboratory
Author Profile


Knowledge Graphs have been fast emerging as the {\em de facto} standard to model and explore knowledge in weakly structured data. Large corpora of documents constitute a source of weakly structured data of particular interest for both the academic as well as the industrial world. Key examples include scientific publications, technical reports, manuals, patents, regulations, etc. Such corpora embed many facts that are elementary to critical decision making or enabling new discoveries. In this paper, we present a scalable cloud platform to create and serve Knowledge Graphs, which we named \textit{Corpus Processing Service}. Its purpose is to process large document corpora, extract the content and embedded facts, and ultimately represent these in a consistent knowledge graph that can be intuitively queried. To accomplish this, we use state-of-the-art natural language understanding models to extract entities and relationships from documents converted with our previously presented CCS platform. This pipeline is complemented with a newly developed graph engine which ensures extremely performant graph queries and provides powerful graph analytics capabilities. Both components are tightly integrated and can be easily consumed through REST APIs. Additionally, we provide user-interfaces to control the data ingestion flow and formulate queries using a visual programming approach. The CPS platform is designed as a modular microservice system operating on Kubernetes clusters. Finally, we validate the quality of queries on our truly end-to-end knowledge pipeline in a real-world application in the oil and gas industry. To date, the capabilities of CPS are successfully leveraged in more than 5 client engagements.

Peer review status:UNDER REVIEW

15 Sep 2020Submitted to Applied AI Letters
16 Sep 2020Assigned to Editor
16 Sep 2020Submission Checks Completed
17 Sep 2020Reviewer(s) Assigned