Reinterpreting Text Data Through Graph Machine Learning
Text classification is one of the most important and classical tasks in natural language processing (NLP). It serves as a core technology for a wide range of practical applications, including document categorization, news filtering, spam detection, sentiment analysis, and customer review evaluation.
In particular, since the widespread adoption of deep learning, convolutional neural network (CNN)-based models have become the mainstream approach to text classification.
However, CNN-based models have reached certain limitations. They tend to focus only on local information, often fail to capture the overall context of a document, struggle with short texts, and rely heavily on the quality of pre-trained embeddings.
To overcome these limitations, this article introduces a case study on text classification using graph-based models[1].
Graph machine learning models enable powerful and effective text classification
Background
Limitations of Traditional Text Classification Models
Traditional text classification models based on CNN, RNN, or BERT primarily rely on sequential word patterns and local context, but they fail to effectively leverage global word co-occurrence relationships across the entire corpus.
As a result, these models face limitations such as information loss, especially when classifying long documents or handling complex topics.
Core Idea
A Graph-Based Approach That Connects Documents and Words into a Unified Structure
In this approach, the entire corpus is structured as a single heterogeneous text graph composed of word nodes and document nodes.
A Graph Convolutional Network (GCN) is then applied to this graph to jointly learn embeddings for both documents and words.
In other words, the text classification task is reformulated as a node classification problem, offering a novel classification system that captures the global structure of the entire document set.
<Example of a Heterogeneous Text Network>
A heterogeneous text graph refers to a graph that contains different types of nodes. In the context of this study, the graph includes the following two types of nodes:
Word Nodes: Each unique word that appears in the corpus is represented as a single node
Document Nodes: Each document in the corpus (e.g., news articles, research abstracts) is also represented as a single node
Solution
Step 1. Generating the Input Feature Matrix
Node data: Document Nodes + Word Nodes
Without using any external pre-trained embeddings (such as word2vec or GloVe), one-hot encoding is applied to the node data (documents and words) to simply distinguish between nodes.
<Example of One-Hot Encoding>
This approach results in a feature matrix that enables the model to learn embeddings solely from neighboring node information and the graph structure during training.
In other words, meaningful representations can gradually be learned using only the connections and label information within the graph.
Step 2. Build a Heterogeneous text graph
Link data: Word-Document + Word-Word
The document-word links were created based on the occurrence of each word in the corresponding document within the corpus, and TF-IDF values were assigned as weights.
The word-to-word links were created based on co-occurring word pairs within the entire corpus using a sliding window approach, and weights were assigned only when the PMI (Pointwise Mutual Information) value was greater than zero.
<Example of Link Data>
This approach allows for the creation of the final network to be input into the model.
Step 3. Comparison of Text Classification Models
In this study, we compared traditional models such as CNN, LSTM, and fastText with the Text GCN model. For each dataset, we created input feature matrices and Heterogeneous text graph, which were then used as input data for the Text GCN model.
The Text GCN is structured as follows.
Input Feature Matrix: Initialized with one-hot vectors for both document and word nodes
Adjacency Matrix: Contains weighted edges based on TF-IDF and PMI values.
2-Layer GCN Model: The first GCN layer generates the initial embeddings, and the second layer performs classification through softmax.
In other words, this study presents a new approach by reinterpreting the text classification problem using a graph-based framework. By constructing documents and words into a single graph and learning through GCN, we compared the performance using a model that incorporates global word relationships, which were often overlooked by traditional deep learning models.
Result
Solving Text Classification Problems with Graph Machine Learning
In this study, comparison experiments were conducted using five benchmark datasets.
20NG(20 Newsgroups): A multi-class news classification dataset
R8(Reuters): Economic-related news
R52(Reuters): Economic-related news with more detailed news categories
Ohsumed: MEDLINE-based dataset of medical research abstracts
MR(Movie Review): Movie review sentences, used for sentiment analysis
Unlike traditional models, Text GCN achieves high performance without external embeddings, particularly demonstrating strong results with limited training data. While existing models focused on local information, Text GCN improved accuracy by reflecting the global document-word structure.
In particular, the Text GCN model achieved higher accuracy (86.34% and 68.36%) than the best-performing models among the existing models, SWEM (85.16%) and Graph-CNN-C (63.86%), on the 20NG and Ohsumed datasets.
Text GCN demonstrated superior performance over traditional models across various benchmarks, particularly maintaining high accuracy in environments with limited data.
As a representative example of successfully applying graph machine learning to text analysis, it not only provides valuable insights for researchers aiming to integrate graph theory into data analysis and natural language processing but also suggests that it can be a powerful solution in various fields such as cyber security, news analysis, and policy document classification, where interrelationships between documents are crucial.
CYRAM Inc.
1660-4230 | netminer@cyram.com
References
[1] Liang Yao, Chengsheng Mao, Yuan Luo. (2019). Graph Convolutional Networks for Text Classification. AAAI Technical Track: Natural Language Processing. Vol. 33 No. 01: AAAI-19, IAAI-19, EAAI-20.