Sentiment analysis seminar

Hi ho, people.  I attended the sentiment analysis seminar again this week, but I was helping lead the discussion this time, on current efforts in sentiment annotation, particularly the Multi-Purpose Question-Answering (MPQA) corpus.  We covered these papers:

The first paper mainly covers the basic effort in MPQA sentiment annotation—how to break down the problem so as to achieve consistent annotation and how to measure inter-annotator agreement.  The basic paradigm that Wiebe and her team use is to view sentiment annotation as being about a mapping between stretches of text and the “private states” of an opinion-holder.  A private state is simply a property of an opinion-holding entity that is not independently verifiable. In other words, a private state is a description of subjectivity.

We can thus develop an ontology of private states and the holders of said states and how they are reflected in text.  There are two major categories of private state expressions: expressive (implicit) subjectivity and direct (explicit) subjectivity.  “Bob squashed the hated insect” is a statement in which the hatedness of the insect is clearly a subjective statement, but it’s not directly attributed to Bob—even though in the context it could be Bob’s opinion.  “Bob claims he hates insects,” on the other hand, is a statement directly made by Bob.

It was interesting to watch the reactions of some of my fellow classmates.  Most of them come from either straight-up CS or hardcore statistical NLP, so fine-grained philosophical distinctions of subjectivity are not day-to-day staples in their work, and some of them expressed on the course mailing list and in class that reading the papers required some amount of mentality shift.  Philip went through some of the techniques of linguists in making these distinctions, including how linguists change the contexts of statements in order to establish tests for the linguistic properties of statements.

Sentiment analysis seems to be one of those places where a stronger bridge between linguistics and applied NLP can be made.

The second paper followed much in the same vein as the first, except that it emphasized the extent to which current NLP techniques cannot yet handle some of the distinctions in the MPQA annotation.  In particular, semantic role labeling—a family of existing techniques for establishing grammatical dependencies—cannot be used to directly infer some of the participants in a private state expression.  For example, when the holder of an opinion is only implied, the semantic role labeling as we currently conceive it will never find it.

The last reading dealt with some additions to the MPQA made by Theresa Wilson, particularly in the addition of target/topic annotations to the MPQA as well as subdividing opinion types into “attitudes” like “sentiment” and “arguing”.

Having fooled around a bit with the MPQA myself, I had the opportunity to show to the class a little bit of what it looked like, and what the challenges of using a somewhat inconsistent standoff annotation format could be.  Philip also took the opportunity to try out some collaborative annotation of a small passage of text with hilarious results—in the amount of arguing it took to decide the subjectivity of even small stretches of text.  Assigning subjectivity is too subjective! In that sense, the high inter-annotator agreement in the original MPQA effort seems somewhat surprising, which had been pointed out on the mailing list before the class.


2 Responses to “Sentiment analysis seminar”

  1. Janyce Wiebe Says:

    Hello – I happened upon this message board. Your class is finished I realize, so perhaps none of you will see this. Many people have carried out the MPQA annotations over the years. People do need to be trained for them to reach good agreement. I can assure you that the agreement results we have published are sound.

    Janyce Wiebe

  2. Hi, there. Pleased to note that someone of note discovered this blog even though I haven’t really been updating it—no posts in more than a year. It wasn’t a class blog, just my personal notes, no one else in the class reads it.

    I don’t think anyone was questioning the soundness of the result. I’ve been trying my hand at running a sentiment annotation task and indeed it is not easy, not at all, and doing it in a detailed manner requires extensive training.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: