Behind the Screens: Using AI to Dissect Social Media Persuasion

In the initial data cleaning phase, LLMs can scan through the whole dataset to pinpoint social media posts that pertain to financial advice. Traditional cleaning techniques for this type of work might revolve around matching the text to specific keywords referencing financial advice. If a post contained one or multiple words from a preselected group of keywords, we would classify such posts as containing financial advice content. One of the benefits of LLMs is they move away from preselection biases and classify the posts based on the prompt that is indicated. We opted not to utilize this specific approach in our study because we sourced our data from specific financial influencers selected for their popularity with the intent of assessing their broader communication patterns.

It is possible to feed a dataset of text to an LLM to indicate if the text contains what could be described as financial advice. Given the technical limitations of the current architecture of LLMs, we have to feed the AI with a social media post from the database and the classification prompt one by one to be classified. If done manually, this process would take a significant amount of time. Instead, we can automate this process by connecting to the LLM Application Programming Interface (API), an interface that enables us to interact directly with AI models. This connection allows for the automatic submission and labeling of social media posts, speeding up the study by processing large amounts of data quickly and effectively. 

This process allows for a better cleaning phase since context is crucial for text classification. Modern LLMs based on a specific neural network architecture called transformers are built to consider the context of the entire text.¹ For example, if we use “invest” as a keyword, we could classify the following two statements as financial advice: 

1. “Invest in crypto” 
2. ”Musicians invest emotion” 

LLMs can take the context of the entire text and classify it correctly: “Invest in crypto” as financial advice and ”Musicians invest emotion” not as financial advice. 

This process is not exempt from errors since we have just introduced a possible data bias by relying on the AI’s training data to classify for us. Such training data could be disproportionately inclined toward certain classifications. An example of this bias could occur if the LLM was disproportionately trained on a tech-centric dataset. The LLM might be more adept at classifying texts related to technology investments as financial advice while also misclassifying statements about investments in other sectors, such as agriculture or education.

This is a known limitation and AI companies are constantly trying to minimize this risk.

LLM application is also useful in the second part of natural language processing: classifying data and extracting insights. The behavioral application for this particular case is that we can prompt the LLM to identify and classify specific persuasive communication techniques within the data. Using AI in this manner, we can loop over the millions of words containing financial advice from social media and quantify the persuasive techniques such as authority, scarcity, and social proof used by influencers. 

We apply the same methodology to classify the construal of social media advice. Construal refers to the level of abstraction or specificity with which information is presented in a message. Low construal refers to thinking that focuses on concrete, specific details and immediate actions. Meanwhile, high construal involves abstract, general principles and long-term goals. Here are two hypothetical examples to help you get a better idea of construal in this context:

1. Low-construal: "Start by setting aside $200 each month into a low-cost index fund. Use automatic transfers from your checking to your investment account on the first of each month to ensure consistency."
2. High-construal: "Focus on building a diverse investment portfolio to secure your financial future. Prioritize investments that align with your long-term financial goals and risk tolerance."

Low-construal messages on social media are viewed as more credible, reducing the perceived riskiness of the information within, resulting in an increased potential for persuasion.² The findings of our research are yet to be publicly available.

Some limitations of this methodology are the training biases of LLMs as previously mentioned, as well as the necessity of connecting to an AI model available through an API. This means that you must share your data with a third party—and in some cases, when the text to be analyzed contains sensitive information, this approach is not viable. It is also important to mention that each call to the API will have a small cost depending on the length of the text and the model being used. The cost varies depending on the provider (in our case, the cost per post classified was a fraction of a cent) but can start to be significant with bigger datasets and large numbers of classifications.   

In summary, we found a way for AI to classify social media posts based on the persuasive techniques used by influencers and the level of construal in a more streamlined and precise manner than traditional methodologies. This allowed us to analyze behavior at a great scale within a digital environment. The specific findings from the mentioned study will be shared later this year with the complete report published by one of our clients. 

This is just one application of how AI can be used to solve challenges using behavioral science. If you are interested, reach out to discuss how we can tailor these strategies to achieve success in your field.