Concept Extraction and Tagging using AI
Research and Development pertaining to Biomedical and Chemical Research especially is very expansive, more than 1200+ articles and research papers are published yearly, some with immediate industrial applications and implications. But, Science is collaborative work where may works can and will contain similar concepts and approaches. Manually sifting through them can be an extremely hectic task and can slow down many integration and application procedures.
The Philippines
Client
ePublishing and Content Management Industry
CLIENT
Our clients is a content creation and management company operating out of the Philippines for the past 30+ providing their services for many corporations and conglomerates with massive success. They have a massive technical literature output covering more than ten different fields with more 120+ papers published yearly. Their ethos is intelligent publication; always seeking out the best of the new technologies that could provide not only an easy alternative but also something that could make a severe impact in the field. They commissioned us to create a robust model which can extract core concepts of an academic paper very easily without resorting to stringent and laborious reviews done by industry experts. These reviews also tend to be very costly and time consuming. By efficient concept extraction, the clients hoped to make a giant leap across fields of research easing referencing and industrial application.
CHALLENGE
A complicated model of this manner certainly has a number of clear challenges that have to be tackled, they are :
-
Complexity of Literature and Terminology - Both the biomedical and chemical fields are known for their expansive and complicated nature, this could be very hard to organize and explicate for the construction of the model.
-
Expert Arbitration - Most of these texts are generally perused by subject experts who have a concise knowledge of the techniques, theories, and the structure of these papers which can take years of education to master. Our goal of automating concept extraction should be a total one.
SOLUTION
Solution Summary:
-
A base data set is created by using ready made data sources, which contains the biomedical and chemical terminology in a objective sense.
-
Utilizing the sources as a base, we can create an auto-indexing tool, which can organize the articles to the needs of the client.
-
This indexing tool contains various steps such as document clustering, content localization, concept ranking, and etc.
- These steps ensure near-perfect concept extraction and seamless tagging.
With the challenges in mind, we have created flexible solutions to deal with them:
-
The source PDFs are organized and pre-processed to orient themselves for easy analysis.
-
Then the document is vectorized and where the context of terms are identified with its subject and is differentiated.
-
A machine learning model which is coupled with a word proximity model is run recursively to identify the key terms.
- They are then shortlisted and is used as the base for the operator response system.
- This response system contains various appraisal and clustering tools used for the auto-indexing and content tagging operations of our model.
- The model’s system also provides content filtration of low quality articles and manual self-check routines.
RESULT
A strong model like this can simplify the tasks of concept extraction by the multitude, there’s little to no room for misplaced concept tagging. We can also automate bulk extraction very easily which drastically increases the accuracy of articles analyzed. Our model can easily recommend 15-20 concepts per document, with a whopping 85% average selection rate. This is highly beneficial to both the industries and academia.
We have also constructed this model to be modular, which implies it can be tailored to use for other fields as well.