Data-driven versus analyst-driven analysis

When we approach unstructured text data, there are two basic types of analysis we can use: data-driven and analyst-driven. In some contexts, “data-driven” refers to how an organization approaches strategic decision making. But in this case, we’re talking about an exploratory approach to analyzing comments and feedback. Essentially, data-driven analysis approaches use statistical methods and unsupervised machine learning techniques to explore text data and discover insights. Some examples of these techniques include automated keyword extraction, text clustering, and automated taxonomy generation.

Analyst-driven approaches, on the other hand, use supervised learning techniques where other inputs from human analysts are used to customize the analysis. Often, the analyst-driven approaches take what’s revealed by the data-driven exploratory analysis and then go beyond this to customize a structure that’s meaningful to the specific text dataset you’re working with. For example, an analyst may prepare a customized sentiment analysis scale or create a hand-crafted taxonomy to classify the attitudes of customers and the themes in their feedback surveys.

Data-driven analysis for customer surveys

To really see how data-driven analysis techniques can improve the analysis of text-based feedback, we’ll walk through some of the data-driven analysis capabilities found within Megaputer’s PolyAnalyst™ analysis platform that were used in a prepared project. This project analyzed customer feedback for a series of toothpaste products. A sample of the raw data is shown below.

The dataset for the project contained approximately 1,200 survey records. The records contained 4 structured data fields—the gender of the reviewer, the toothpaste product reviewed, a Likert-scale rating of the product provided by the reviewer, and a Likert-scale rating of the reviewer’s buying intention after testing the product. These records also contained an unstructured data field for customers to make suggestions on what to improve for the toothpaste product they tested.

After loading the data, several steps of data cleansing and pre-processing were performed to ready the column of text for analysis. Data cleansing is a highly important task, especially for text analysis. Typically, this involves intuitive things like spell checking and de-duplication. However, less obvious is the need for abbreviation expansion and standardizing terms by mapping unknown synonyms.

At this point, we typically begin the data-driven analysis, to reveal the various themes and patterns in the text of the customer reviews. In PolyAnalyst™, there are multiple analysis features that assist analysts with this process. We’ll now go over a few of the data-driven analysis methods and the type of results you can obtain by performing these operations on your text data.

Automated Keyword and Phrase Extraction

With the advent of Natural Language Processing (NLP), keyword extraction has become a go-to method among analysts for identifying important terms from plain text. At their core, keyword extraction algorithms count the frequency of specific words that appear in the text. In PolyAnalyst™, basic keyword statistics are derived from the well-known Open American National Corpus (OANC). Once commonly used words are removed (i.e., a, an, the, etc.), the counts of each word are tabulated. The results can be displayed as a word cloud, where the color and size of a word reflects its significance and frequency, respectively, in the dataset.

Some of you may be thinking, “Wait a minute, what is significance?” Significance is another metric used to reveal the importance of a keyword despite its frequency in the dataset. For example, if our analysis is based on a giant corpus of generic data where we expect to see the keyword baseball 10 times in 1000 documents, then it would be significant to see baseball show up 200 times in the same amount of documents. Even though the keyword “register” might show up twice as many times as baseball, this doesn’t necessarily mean it is more significant. Perhaps the keyword register is actually expected to appear 500 times in the same amount of documents. Furthermore, a keyword like baseball is even more significant when we provide our analysis model a relevant baseline, such as a corpus of data related to hygiene products. In that event, if baseball showed up even 10 times in a toothpaste survey data, that would certainly be worth some further investigation.

But in our case,  baseball didn’t show up in our customer reviews on toothpaste products. Here are the main results of automated keyword extraction:

Text Clustering

Clustering is a helpful technique that sorts related records into groups based on their common themes.  In PolyAnalyst™, a modified version of the Suffix Tree Clustering algorithm is used to search for sets of records with shared phrases, and then these clusters are merged together based on the percent of records containing the phrases. Users can specify various options, including the clustering method (K-means, pLSA, etc.), and the Text Clustering function automatically generates a classification model based on the clusters of terms within the text column you select. The output provides a report of the identified clusters.

Here is the view of the results from text clustering in PolyAnalyst™.

These clusters can give you a sense of some of the words that may go together in various categories and subcategories. For example, in the top left panel of the image, we see that the records in Cluster 2 contain the themes taste and strong, whereas Cluster 5 contains records with topics like cap, tube, and messy. Since Cluster 5 is highlighted, we see the other keywords that were commonly found in the cluster (lower left panel), and on the right, we can see the records in that cluster and can view the text of a selected record in the text panel.

We may observe that many of the clusters here contain sets of words that we’ve already discovered from automated keyword extraction, which confirms the themes are present in the customer reviews. So, from this, it seems like the following themes are among the top aspects being discussed by customers: taste, texture, packaging (i.e., cap, tube), and after effects (i.e., clean feeling, aftertaste).

Auto Taxonomy Generation

Another common method for data-driven analysis is automatic taxonomy construction (ATC). In this case, the system uses NLP algorithms to process unstructured text and generate a hierarchical classification of the content. This unsupervised method of discovering the various concepts and themes in the text is also highly transparent because the resulting structure is based on the frequency of terms present in the data. ATC is typically a good starting point for identifying additional topics arising from the text, prior to an analyst crafting a customized classification structure.

In PolyAnalyst, automated taxonomies can be generated in multiple ways. The Auto Taxonomy feature can be used directly on the unstructured text data to extract themes, or a taxonomy can be created using the results of keyword extraction. Typically, both approaches are useful to perform when exploring the text data, particularly because the results can vary between methods.

Here are the auto taxonomy results for the toothpaste product survey comments, both with and without the automated keyword extraction inputs.