How to analyse sentiment and emotion in call transcripts

Step by step

1. 1

   Get a transcript with speaker names

   Sentiment means little unless you know who said it. Export the transcript with speakers and timestamps.

2. 2

   Split it into speaker turns

   Score each turn on its own. Very long turns can be split into sentences; these models read up to 512 tokens.

3. 3

   Pick a model

   cardiffnlp/twitter-roberta-base-sentiment-latest gives Negative, Neutral or Positive in English. j-hartmann/emotion-english-distilroberta-base gives anger, disgust, fear, joy, neutral, sadness or surprise.

4. 4

   Score and aggregate

   Average per speaker, and look at how the customer's sentiment moves from the start to the end of the call.

5. 5

   Review the extremes

   Read the most negative turns yourself. Sarcasm, jargon, apologies and polite complaints are easy for a model to misread: an agent's "I'm sorry about that" often scores as negative.

Choosing a model

• cardiffnlp/twitter-roberta-base-sentiment-latest: English, 3 labels, trained on about 124M tweets, CC BY 4.0 licence.
• j-hartmann/emotion-english-distilroberta-base: English, Ekman's 6 basic emotions plus neutral; the model card reports 66% evaluation accuracy.
• tabularisai/multilingual-sentiment-analysis: 5 labels from Very Negative to Very Positive in 23 languages, but a CC BY-NC 4.0 licence, so not for commercial use.

Score each speaker turn and average per speaker

pip install transformers torch. The model downloads from Hugging Face on first run.

from collections import defaultdict
from transformers import pipeline

turns = [
    ("Customer", "We've had the export bug for two weeks and nobody has called us back."),
    ("Agent", "I'm sorry about that. I've escalated it and I'll call you tomorrow with a fix date."),
    ("Customer", "Thanks, that helps. The rest of the product has been great for our team."),
]

sentiment = pipeline("text-classification", model="cardiffnlp/twitter-roberta-base-sentiment-latest", truncation=True)

# Turn each prediction into a score from -1 (negative) to +1 (positive).
sign = {"negative": -1, "neutral": 0, "positive": 1}
per_speaker = defaultdict(list)
for speaker, text in turns:
    p = sentiment(text)[0]
    score = sign[p["label"].lower()] * p["score"]
    per_speaker[speaker].append(score)
    print(f"{speaker:9} {p['label']:8} {p['score']:.2f}  {text[:50]}")

for speaker, scores in per_speaker.items():
    print(speaker, "average", round(sum(scores) / len(scores), 2), "first", round(scores[0], 2), "last", round(scores[-1], 2))

Add emotions

from transformers import pipeline

emotion = pipeline("text-classification", model="j-hartmann/emotion-english-distilroberta-base", top_k=None, truncation=True)
scores = emotion("We've had the export bug for two weeks and nobody has called us back.")[0]
print(sorted(scores, key=lambda s: -s["score"])[:3])

Common mistakes

• Scoring the whole call as one block; the average hides the moment a customer got upset.
• Ignoring the licence: some popular models don't allow commercial use.
• Using a model trained on tweets as a final judgement on a person's call.

Questions