Advanced Search

Indexed by SCI、CA、РЖ、PA、CSA、ZR、etc .

Volume 34 Issue 5
Oct 2023
Turn off MathJax
Article Contents
Kai Ma, Miao Tian, Yongjian Tan, Qinjun Qiu, Zhong Xie, Rong Huang. Ontology-Based BERT Model for Automated Information Extraction from Geological Hazard Reports. Journal of Earth Science, 2023, 34(5): 1390-1405. doi: 10.1007/s12583-022-1724-z
Citation: Kai Ma, Miao Tian, Yongjian Tan, Qinjun Qiu, Zhong Xie, Rong Huang. Ontology-Based BERT Model for Automated Information Extraction from Geological Hazard Reports. Journal of Earth Science, 2023, 34(5): 1390-1405. doi: 10.1007/s12583-022-1724-z

Ontology-Based BERT Model for Automated Information Extraction from Geological Hazard Reports

doi: 10.1007/s12583-022-1724-z
More Information
  • Corresponding author: Rong Huang, huangrong@ctgu.edu.cn
  • Received Date: 25 Dec 2021
  • Accepted Date: 22 Aug 2022
  • Available Online: 14 Oct 2023
  • Issue Publish Date: 30 Oct 2023
  • Geological knowledge can provide support for knowledge discovery, knowledge inference and mineralization predictions of geological big data. Entity identification and relationship extraction from geological data description text are the key links for constructing knowledge graphs. Given the lack of publicly annotated datasets in the geology domain, this paper illustrates the construction process of geological entity datasets, defines the types of entities and interconceptual relationships by using the geological entity concept system, and completes the construction of the geological corpus. To address the shortcomings of existing language models (such as Word2vec and Glove) that cannot solve polysemous words and have a poor ability to fuse contexts, we propose a geological named entity recognition and relationship extraction model jointly with Bidirectional Encoder Representation from Transformers (BERT) pretrained language model. To effectively represent the text features, we construct a BERT- bidirectional gated recurrent unit network (BiGRU)-conditional random field (CRF)-based architecture to extract the named entities and the BERT-BiGRU-Attention-based architecture to extract the entity relations. The results show that the F1-score of the BERT-BiGRU-CRF named entity recognition model is 0.91 and the F1-score of the BERT-BiGRU-Attention relationship extraction model is 0.84, which are significant performance improvements when compared to classic language models (e.g., word2vec and Embedding from Language Models (ELMo)).

     

  • Conflict of Interest
    The authors declare that they have no conflict of interest.
  • loading
  • Bengio, Y., Ducharme, R., Vincent, P., 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3: 1137–1155
    Bojanowski, P., Grave, E., Joulin, A., et al., 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5: 135–146. https://doi.org/10.1162/tacl_a_00051
    Bouvrie, J., 2006. Notes on Convolutional Neural Networks, Neural Nets. http://web.mit.edu/jvb/www/papers/cnn_tutorial.pdf
    Cao, P., Chen, Y., Liu, K., et al., 2018. Adversarial Transfer Learning for Chinese Named Entity Recognition with Self-Attention Mechanism. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, October 31–November 4, Brusssels
    Chiticariu, L., Krishnamurthy, R., Li, Y. Y., et al., 2010. Domain Adaptation of Rule-Based Annotators for Named-Entity Recognition Tasks. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. October 9–11, 2010, Cambridge. 1002–1012. https://doi.org/10.5555/1870658.1870756
    Chiu, J. P., Nichols, E., 2016. Named Entity Recognition with Bidirectional LSTM-CNNS. Transactions of the Association for Computational Linguistics, 4: 357–370 doi: 10.1162/tacl_a_00104
    Deng, C., Jia, Y. T., Xu, H., et al., 2021. GAKG: A Multimodal Geoscience Academic Knowledge Graph. Proceedings of the 30th ACM International Conference on Information & Knowledge Management. November 1–5, 2021, Virtual Event, New York. 4445–4454. https://doi.org/10.1145/3459637.3482003
    Devlin, J., Chang, M. W., Lee, K., et al., 2018. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1810.04805. https://arxiv.org/abs/1810.04805
    Enkhsaikhan, M., Liu, W., Holden, E. J., et al., 2021. Auto-Labelling Entities in Low-Resource Text: A Geological Case Study. Knowledge and Information Systems, 63(3): 695–715. https://doi.org/10.1007/s10115-020-01532-6
    Fan, J., Shen, S., Erwin, D. H., et al., 2020. A High-Resolution Summary of Cambrian to Early Triassic Marine Invertebrate Biodiversity. Science, 367(6475): 272–277 doi: 10.1126/science.aax4953
    Fang, W. L., Ma, L., Love, P. E. D., et al., 2020. Knowledge Graph for Identifying Hazards on Construction Sites: Integrating Computer Vision with Ontology. Automation in Construction, 119: 103310. https://doi.org/10.1016/j.autcon.2020.103310
    Gayen, V., Sarkar, K., 2014. An HMM Based Named Entity Recognition System for Indian Languages: The JU System at ICON 2013. arXiv: 1405.7397. https://arxiv.org/abs/1405.7397
    Gers, F. A., Schmidhuber, J., Cummins, F., 2000. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10): 2451–2471. https://doi.org/10.1162/089976600300015015
    Ghahabi, O., Hernando, J., 2018. Restricted Boltzmann Machines for Vector Representation of Speech in Speaker Recognition. Computer Speech & Language, 47: 16–29. https://doi.org/10.1016/j.csl.2017.06.007
    Hashimoto, K., Miwa, M., Tsuruoka, Y., et al., 2013. Simple Customization of Recursive Neural Networks for Semantic Relation Classification. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 18–21
    Hochreiter, S., Schmidhuber, J., 1997. Long Short-Term Memory. Neural Computation, 9(8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
    Jauregi Unanue, I., Zare Borzeshi, E., Piccardi, M., 2017. Recurrent Neural Networks with Specialized Word Embeddings for Health-Domain Named-Entity Recognition. Journal of Biomedical Informatics, 76: 102–109. https://doi.org/10.1016/j.jbi.2017.11.007
    Lai, T., Ji, H., Zhai, C. X., et al., 2021. Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference. arXiv: 2105.13456. https://arxiv.org/abs/2105.13456
    Lawley, C. J. M., Raimondo, S., Chen, T. Y., et al., 2022. Geoscience Language Models and Their Intrinsic Evaluation. Applied Computing and Geosciences, 14: 100084. https://doi.org/10.1016/j.acags.2022.100084
    Li, P. F., Mao, K. Z., 2019. Knowledge-Oriented Convolutional Neural Network for Causal Relation Extraction from Natural Language Texts. Expert Systems with Applications, 115: 512–523. https://doi.org/10.1016/j.eswa.2018.08.009
    Lin, Y., Shen, S., Liu, Z., et al., 2016. Neural Relation Extraction with Selective Attention Over Instances. The 54th Annual Meeting of the Association for Computational Linguistics, August 7–12, Berlin
    Liu, Z. J., Yang, M., Wang, X. L., et al., 2017. Entity Recognition from Clinical Texts via Recurrent Neural Network. BMC Medical Informatics and Decision Making, 17(Suppl 2): 67. https://doi.org/10.1186/s12911-017-0468-7
    Lü, X., Xie, Z., Xu, D., et al., 2022. Chinese Named Entity Recognition in the Geoscience Domain Based on BERT. Earth and Space Science, 9(3): e2021EA002166 doi: 10.1029/2021EA002166
    Ma, K., Tan, Y. J., Tian, M., et al., 2022a. Extraction of Temporal Information from Social Media Messages Using the BERT Model. Earth Science Informatics, 15(1): 573–584. https://doi.org/10.1007/s12145-021-00756-6
    Ma, K., Tan, Y. J., Xie, Z., et al., 2022b. Chinese Toponym Recognition with Variant Neural Structures from Social Media Messages Based on BERT Methods. Journal of Geographical Systems, 24(2): 143–169. https://doi.org/10.1007/s10109-022-00375-9
    Ma, K., Tian, M., Tan, Y. J., et al., 2022c. What is This Article About? Generative Summarization with the BERT Model in the Geosciences Domain. Earth Science Informatics, 15(1): 21–36. https://doi.org/10.1007/s12145-021-00695-2
    Ma, X. G., 2019. Geo-Data Science: Leveraging Geoscience Research with Geoinformatics, Semantics and Open Data. Acta Geologica Sinica-English Edition, 93(S3): 44–47. https://doi.org/10.1111/1755-6724.14240
    Ma, X. G., Ma, C., Wang, C. B., 2020. A New Structure for Representing and Tracking Version Information in a Deep Time Knowledge Graph. Computers & Geosciences, 145: 104620. https://doi.org/10.1016/j.cageo.2020.104620
    Mikolov, T., Chen, K., Corrado, G., et al., 2013. Efficient Estimation of Word Representations in Vector Space. arXiv: 1301.3781. https://arxiv.org/abs/1301.3781
    Miwa, M., Bansal, M., 2016. End-to-End Relation Extraction Using LSTMS on Sequences and Tree Structures. arXiv: 1601.00770. https://arxiv.org/abs/1601.00770
    Nguyen, T. H., Grishman, R., 2015. Relation Extraction: Perspective from Convolutional Neural Networks. Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing. Association for Computational Linguistics, Denver
    Nieh, E. H., Schottdorf, M., Freeman, N. W., et al., 2021. Geometry of Abstract Learned Knowledge in the Hippocampus. Nature, 595(7865): 80–84. https://doi.org/10.1038/s41586-021-03652-7
    Oramas, S., Ostuni, V. C., Di Noia, T., et al., 2017. Sound and Music Recommendation with Knowledge Graphs. ACM Transactions on Intelligent Systems and Technology, 8(2): 1–21. https://doi.org/10.1145/2926718
    Palumbo, E., Monti, D., Rizzo, G., et al., 2020. Entity2rec: Property-Specific Knowledge Graph Embeddings for Item Recommendation. Expert Systems with Applications, 151: 113235. https://doi.org/10.1016/j.eswa.2020.113235
    Peng, N. Y., Dredze, M., 2016. Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning. arXiv: 1603.00786. https://arxiv.org/abs/1603.00786
    Pennington, J., Socher, R., Manning, C., 2014. Glove: Global Vectors for Word RepresentationProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha
    Peters, M. E., Neumann, M., Iyyer, M., et al., 2018. Deep Contextualized Word Representations. arXiv: 1802.05365. https://arxiv.org/abs/1802.05365
    Qiu, Q. J., Xie, Z., Wu, L., et al., 2018. DGeoSegmenter: A Dictionary-Based Chinese Word Segmenter for the Geoscience Domain. Computers & Geosciences, 121: 1–11. https://doi.org/10.1016/j.cageo.2018.08.006
    Qiu, Q. J., Xie, Z., Wu, L., et al., 2019a. BiLSTM-CRF for Geological Named Entity Recognition from the Geoscience Literature. Earth Science Informatics, 12(4): 565–579. https://doi.org/10.1007/s12145-019-00390-3
    Qiu, Q. J., Xie, Z., Wu, L., et al., 2019b. Geoscience Keyphrase Extraction Algorithm Using Enhanced Word Embedding. Expert Systems With Applications, 125: 157–169. https://doi.org/10.1016/j.eswa.2019.02.001
    Qiu, Q. J., Xie, Z., Wu, L. A., et al., 2019c. GNER: A Generative Model for Geological Named Entity Recognition without Labeled Data Using Deep Learning. Earth and Space Science, 6(6): 931–946. https://doi.org/10.1029/2019ea000610
    Qiu, Q. J., Xie, Z., Wu, L., et al., 2020a. Automatic Spatiotemporal and Semantic Information Extraction from Unstructured Geoscience Reports Using Text Mining Techniques. Earth Science Informatics, 13(4): 1393–1410. https://doi.org/10.1007/s12145-020-00527-9
    Qiu, Q. J., Xie, Z., Wu, L. A., et al., 2020b. Dictionary-Based Automated Information Extraction from Geological Documents Using a Deep Learning Algorithm. Earth and Space Science, 7(3): e2019EA000993. https://doi.org/10.1029/2019ea000993
    Qu, J. F., Ouyang, D. T., Hua, W., et al., 2018. Distant Supervision for Neural Relation Extraction Integrated with Word Attention and Property Features. Neural Networks, 100: 59–69. https://doi.org/10.1016/j.neunet.2018.01.006
    Radford, A., Narasimhan, K., 2018. Improving Language Understanding by Generative Pre-Training, preprint. https://gwern.net/doc/www/s3-us-west-2.amazonaws.com/d73fdc5ffa8627bce44dcda2fc012da638ffb158.pdf
    Radford, A., Wu, J., Child, R., et al., 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8): 9
    Santos, R., Murrieta-Flores, P., Calado, P., et al., 2018. Toponym Matching through Deep Neural Networks. International Journal of Geographical Information Science, 32(2): 324–348. https://doi.org/10.1080/13658816.2017.1390119
    Singhal, A., 2012. Introducing the Knowledge Graph: Things, not Strings. Google Blog. https://www.blog.google/products/search/introducing-knowledge-graph-things-not/
    Sun, C., Yang, Z. H., Wang, L., et al., 2021. Biomedical Named Entity Recognition Using BERT in the Machine Reading Comprehension Framework. Journal of Biomedical Informatics, 118: 103799. https://doi.org/10.1016/j.jbi.2021.103799
    Surdeanu, M., Tibshirani, J., Nallapati, R., et al., 2012. Multi-Instance Multi-Label Learning for Relation Extraction. The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. July 12– 14, 2012, Jeju Island. https://doi.org/10.5555/2390948.2391003
    Vaswani, A., Shazeer, N., Parmar, N., et al., 2017. Attention is All You Need. Advances in Neural Information Processing Systems, 30: 5998–6008
    Vincent, P., Larochelle, H., Bengio, Y., et al., 2008. Extracting and Composing Robust Features with Denoising Autoencoders. The 25th International conference on Machine Learning. July 5–9, 2008, Helsinki. https://doi.org/10.1145/1390156.1390294
    Wang, C., Hazen, R. M., Cheng, Q., et al., 2021. The Deep-Time Digital Earth Program: Data-Driven Discovery in Geosciences. National Science Review, 8(9): nwab027. https://doi.org/10.1093/nsr/nwab027
    Wu, S., Song, X. N., Feng, Z. H., 2021. MECT: Multi-Metadata Embedding Based Cross-Transformer for Chinese Named Entity Recognition. arXiv: 2107.05418. https://arxiv.org/abs/2107.05418
    Xu, Y., Mou, L. L., Li, G., et al., 2015. Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path. arXiv: 1508.03720. https://arxiv.org/abs/1508.03720
    Yao, L., Zhang, Y., Chen, Q. F., et al., 2017. Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data. Engineering Applications of Artificial Intelligence, 64: 432–439. https://doi.org/10.1016/j.engappai.2017.06.024
    Zeng, D., Liu, K., Lai, S., et al., 2014. Relation Classification via Convolutional Deep Neural Network. The 25th International Conference on Computational Linguistics: Technical Papers, March 25–28, Tokyo
    Zhang, W., Du, Y. H., Yoshida, T., et al., 2019. DeepRec: A Deep Neural Network Approach to Recommendation with Item Embedding and Weighted Loss Function. Information Sciences, 470: 121–140. https://doi.org/10.1016/j.ins.2018.08.039
    Zhang, X. Y., Ye, P., Wang, S., et al., 2018. Geological Entity Recognition Method Based on Deep Belief Networks. Acta Petrologica Sinica, 34(2): 343–351
    Zheng, S. C., Hao, Y. X., Lu, D. Y., et al., 2017a. Joint Entity and Relation Extraction Based on a Hybrid Neural Network. Neurocomputing, 257: 59–66. https://doi.org/10.1016/j.neucom.2016.12.075
    Zheng, S. C., Wang, F., Bao, H. Y., et al., 2017b. Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. arXiv: 1706.05075. https://arxiv.org/abs/1706.05075
    Zhou, C. H., Wang, H., Wang, C. S., et al., 2021. Geoscience Knowledge Graph in the Big Data Era. Science China Earth Sciences, 64(7): 1105–1114. https://doi.org/10.1007/s11430-020-9750-4
    Zhou, P., El-Gohary, N., 2017. Ontology-Based Automated Information Extraction from Building Energy Conservation Codes. Automation in Construction, 74: 103–117 doi: 10.1016/j.autcon.2016.09.004
    Zhou, P., Xu, J., Qi, Z., et al., 2018. Distant Supervision for Relation Extraction with Hierarchical Selective Attention. Neural Networks, 108: 240. https://doi.org/10.1016/j.neunet.2018.08.016
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(3)  / Tables(11)

    Article Metrics

    Article views(233) PDF downloads(42) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return