Social Media Analytics -- Content Analysis

The information overload in social media becomes increasingly noticeable. This multitude of information raises the demand of content analysis, especially, information retrieval: extracting useful relevant information from a set of document. Lecture 3 demonstrated the main components of content analysis, which are document comparison, text classification and text clustering.

Document comparison

Document is constituted by numerous words in various orders, no doubt, it can be represented, analyzed and comparing by some particular words. The most representative word can be defined as the ones with high Term Frequency & Inverse Document Frequency (TF-IDF), the product of Term Frequency (number of times this word appears in this document) and Inverse Document Frequency
(). TF-IDF value increases proportionally to the frequency of a word in the document, but is offset by the occurrences of the term in the corpus, which attenuates the effect of terms that occur too often in almost every document.

Text classification

1.     Approaches

Two document classification approaches are listed as follows.

1)    Hierarchical categorization

Hierarchical categorization is a tree distribution, each document belongs to a unique category associated with it, and the categories are arranged hierarchically, as Figure 1. The most distinct drawback of this method could appear, when some elements not fit distinctly in one category.

Fugure1. Example of a hierarchical category

2)    Tag Categorization

Tag-based categorization is opposed to Hierarchical categorization, which means documents can be assigned to numerous related categorical tags.

2.     Technique

The text classification of new data can be implemented by a classifier, trained by using labeled data. Naïve Bayed text classification, the most popular sorting technique, classifies documents based on the frequency of representative words.

Text clustering

Text clustering can be briefly defined as grouping the similar objects in the data, by measuring their similarity using similarity functions. This clustering technique shows an outstanding contribution in the text domain, where the objects are of different granularities (such as documents, paragraphs, terms, etc.), especially in improving retrieval and supporting browsing.

Clustering can by generally divided, based on its results, into hard (one item to one cluster) and soft (one item to multiple clusters), K-Means Clustering and Model-based clustering are the most representative algorithms of these two clustering categories, respectively.

Reference：

[1] Klieber, P, Document classification through data mining social media networks, November, 2009.

[2] Aggarwal, CC, Zhao, Y, & Yu, PS, On text clustering with side information, ICDE Conference, 2012.