Well, I’m appointed in the department of Computational Linguistics and Phonetics (COLI), but in the way of academia, there are a series of bodies involved in my employment, and their relationships are something I don’t entirely understand yet.  I am apparently part of the Multi-modal Computing and Innovation Cluster of Excellence (MMCI), which is a joint project of a bunch of groups, including the Max Planck Institute and the Deutsche Forschungsinstitut für Künstliche Intelligenz (DFKI, the German Research Institute for Artificial Intelligence).  My supervisor is Vera Demberg, who is a Junior Research Group Leader appointed as part of the MMCI package.

I chose to accept Vera’s offer back in the summer and come here because Vera is very active in bridging the gap between the formal sort of linguistics which is near and dear to my heart (as some of you know), the psycholinguistics that I’ve always wanted to get my fingers into, and the more practical-minded statistical efforts to make systems that represent the world in a robust way, which is where I focused my PhD dissertation work.  So it was a close match of congruent interests.

Right now, we’ve been working on defining our actual research project and goals, which has actually been a lot of fun.  There are plusses and minuses to starting a postdoc when there wasn’t already an active project in place (Vera has other projects less technologically-oriented).   During my graduate career, I had multiple opportunities to define projects both for my thesis and for grant- and internship-driven work, and for the most part it worked out well, so on balance I’m pretty happy to be there at the beginning.  Another advantage is that I don’t have to reverse-engineer someone else’s peculiar code.  Well, for now, at least.

I’ve had some catching up to do on some recent developments in psycholinguistic and representational frameworks, and we’ve particularly been focusing on surprisal-based measures of cognitive load.  Surprisal in computational psycholinguisics has developed a burgeoning literature over the past several years, particularly stemming from foundational work by people like John Hale, now at Cornell.  As an information-theoretic measure, surprisal is one way to bridge the gap between formal representation and statistical robustness which other measures of cognitive load (and statistical modeling) do not do as well.  As long as we have a conditional probability in the denominator, we can find the surprisal at particular points in a string (assuming incremental parsing).  Then it becomes a matter of testing the predictions experimentally.

However, we’ve been looking at opportunities to apply these to various kinds of real-time information retrieval and user interface tasks, particularly with transcribed/ASR speech.  But, and here’s the catch, we want to augment the typically syntax-based surprisal measures with some kind of additional formal semantics, which will then allow domain-dependence in our applications.

We’re hoping not to build an entire experimental and processing pipeline from scratch, so we’ve been casting about for resources and collaborators, as well as looking for students from within the Saarland fold.  I suppose I’m biased, but the current state of psycholinguistics, statistical modeling, and syntactic/semantic formalism is coming finally toward an interesting convergence points where we can start to model natural natural language processing activities, so to speak, so I already see lots of exciting opportunities.  And I’ve only been here a month.

Living at about 50% German immersion has certainly forced my high-school German to improve.  Particularly my renting vocabulary.

Anyway, I finally found myself a place in the very heart of Saarbrücken with an excellent, unbeatable view.  When I get around to it, I might post pictures.  Ordered a ton of furniture; probably ill-advised, but I got tired of living completely like a student.  Even then, I mostly got the cheap stuff that will be easy to get rid of, since this is just a postdoc job.  So I have to go through much of the paperwork all over again to change my address with the city resident register—this is rather alien to people from English-speaking countries, the idea that the government has proof-positive of your whereabouts, and no, this is for all residents, not just foreigners—change my address with the bank, and so on.

So as for Saarbrücken itself, it offers a pretty high standard of living, and is not very expensive to live in.  Rents are less than half of what one finds in Washington DC.  It’s a city of about 200,000, but it serves a much larger metropolitan region, as well as being the main metropolitan centre for all of the Saarland, a good chunk of the Palatinate (the Pfalz in Rheinland-Pfalz), and a big chunk of Lorraine.  I have so far managed to make day trips to places in the Saarland and Lorraine; the farthest I’ve gone in Trier, which is well worth visiting.  I shall be back.

Now y’all are probably wondering what I’m doing at work, aside from filling out forms.  That, I will get to in the next post, if I have time to post it.  Unfortunately, with no permanent address, getting a stable Internet connection has been a challenge, and I’m blogging from work (and it’s pretty late here).

But I am getting ahead of myself, because you’ll now correctly wonder what I am going to be doing with my life.  Well, the first thing I am doing, once I have fully deconstructed my time in Maryland, is to move to Germany.  Yep, I am starting a post-doctoral position in Germany next month, at the University of the Saarland, working on incremental parsing formalisms with Dr. Vera Demberg, once all the paperwork is in order.  Change of continent, change of focus.

I’m more than a little sad to leave the DC area, of course.  The first three years I was here, I was but a tourist; if it had ended then, I’d have thought, “Huh, that was interesting.  The fourth and fifth years, the DC are became “familiar” to me.  These past couple of years, though, have left me with some amount of roots and personal attachment to the place.  It’s home, but the way of the world is that those of us with intended careers in research have to follow the jobs.  Even when times are good, I can’t just pick a city and expect to live there.  Doubly so for DC, where I am not a citizen and not eligible for most of the directly-attached government jobs. So I leave Maryland without the illusion that I would be coming back and settling down here.

But…a trip to Europe to pick up on a kind of work I used to do, and want to keep alive as part of my repertoire.  If I have to leave the land of the Smithsonian, there are few better ways I could have done it.  An opportunity to refresh my German too.

Now back to sorting through seven years worth of excess paper and packing and all those annoying details…

 THE DISSERTATION DEFENSE FOR THE DEGREE OF Ph.D. IN COMPUTER SCIENCE FOR

Asad Basheer Sayeed

Will be held:

DATE:                    Wednesday August 3, 2011 at 12:00 p.m.

LOCATION:           Room 3258 A.V. Williams Bldg.

TITLE:        A Distributional and Syntactic Approach to Fine-Grained Opinion Mining

ABSTRACT:  This thesis contributes to a larger social science research program of analyzing the diffusion of IT innovations. We show how to automatically discriminate portions of text dealing with opinions about innovations by finding {source, target, opinion} triples in text. In this context, we can discern a list of innovations as targets from the domain itself. We can then use this list as an anchor for finding the other two members of the triple at a “fine-grained” level—paragraph contexts or less.

We first demonstrate a vector space model for finding opinionated contexts in which the innovation targets are mentioned. We can find paragraph-level contexts by searching for an “expresses-an-opinion-about” relation between sources and targets using a supervised model with an SVM that uses features derived from a general-purpose subjectivity lexicon and a corpus indexing tool. We show that our algorithm correctly filters the domain relevant subset of subjectivity terms so that they are more highly valued.

We then turn to identifying the opinion. Typically, opinions in opinion mining are taken to be positive or negative. We discuss a crowdsourcing technique developed to create the seed data describing human perception of opinion bearing language needed for our supervised learning algorithm. Our user interface successfully limited the meta-subjectivity inherent in the task (“What is an opinion?”) while reliably retrieving relevant opinionated words using labour not expert in the domain.

Finally, we developed a new data structure and modeling technique for connecting targets with the correct within-sentence opinionated language. Syntactic relatedness tries (SRTs) contain all paths from a dependency graph of a sentence that connect a target expression to a candidate opinionated word. We use factor graphs to model how far a path through the SRT must be followed in order to connect the right targets to the right words. It turns out that we can correctly label significant portions of these tries with very rudimentary features such as part-of-speech tags and dependency labels with minimal processing. This technique uses the data from the crowdsourcing technique we developed as training data.

We conclude by placing our work in the context of a larger sentiment classification pipeline and by describing a model for learning from the data structures produced by our work. This work contributes to computational linguistics by proposing and verifying new data gathering techniques and applying recent developments in machine learning to inference over grammatical structures for highly subjective purposes. It applies a suffix tree-based data structure to model opinion in a specific domain by imposing a restriction on the order in which the data is stored in the structure.

Examining Committee:

COMMITTEE CHAIR:                          Dr. Amy Weinberg

Dean’s Representative:                      Dr. William Idsardi

Committee Members:

Dr. Jordan Boyd-Graber

Dr. Hal Daume III

Dr. Donald Perlis

EVERYONE IS INVITED TO ATTEND THE PRESENTATION PORTION OF THIS DEFENSE

This article is a draft of something that I’ve been working on for several years.  It’s a kind of mini-linguistics-thesis inside my computer science degree.   I had to stop significant activity on it in March 2009 in order to devote the bulk of my time to experiments for my dissertation and other projects on which I was working at the time, so it was left in a (in my opinion) nearly ready state, but just needs a couple of more solid months of work, or a couple of years of hacking at it in my non-existent spare time.  But, you know, dissertation…

So I was faced with a choice of hiding my light under a bushel (or whatever the saying is) while I waited for that solid time-period or letting the world see it before it became a little too out-of-date.  I choose the latter.

What it’s about: The abstract (and article) is written in syntax-ese, but it’s the culmination of some thoughts I’ve had about languages that allow for long-distance extraction of noun phrase constituents from finite clauses…there I go with the syntax-ese again.  But let me put it this way: in some languages you can turn:

I now see that the jelly donuts were tasty.

into something like:

(Not English) The jelly donuts I now see that _____ were tasty.

But English does something weird as well: preposition stranding in questions:

What is he going on about ____?

In our article, Amy and I suggest that these facts are related, but current versions of generative syntax have removed the mechanism to express relationships of this kind.  These were called “escape hatches” in the past, and we propose a way to express them that is consistent with the terms of current theory.

I suspect/hope that it will be controversial.

Update 11:20 – 6 downloads and it’s hardly been 20 minutes.  That was fast.

In a nutshell: Melody’s thread is yet another rehash of the old methodological arguments against linguistic (particularly syntactic) theory that are destined to be visited on every generation. Multiple times. Forever and ever—it is simply a fact one must accept that people are going to believe that Google is a sort of linguistic counterexample engine.  I am in the peculiar position of someone who works with Big Corpora as his bread-and-butter and dissertation topic and so on—but remains quite skeptical of the ability of this work to provide us with particularly interesting insights as to the human capacity for language in itself.

But the main point I want to make, briefly, is on the linguistics blogosphere itself.  Is it just me, or is it wildly unrepresentative of the linguistics field as a whole?  Maybe it’s because I live very near to/participate in the Maryland hothouse of unreconstructed generative grammarians (Philip Resnik excepted, heh), but, um, it doesn’t seem to reflect the other “hothouses” (Carleton U and U of Ottawa) to which I’ve belonged as well, nor does it reflect my brushes past other real-life linguists and other departments.  On the occasions that I have read Language Log, it and its commentariat have tended to take positions a lot closer to Melody’s than the part of mainstream syntactic theory.

Aside from an obvious accusation of “anecdote” and “sample bias”, let me throw out another possible explanation that might actually tie together a number of issues: the fact that a lot of syntactic theorists, both faculty and students, tend to come from humanities (lit. and philosophy) backgrounds, and that it is not really surprising that the linguistic blogosphere is pretty saturated by Big Corpus and neo-empiricists and so on—and why a Google (heh) search for “minimalist linguistics blog” and various terms like that don’t tend to turn up much.

Again, perhaps I missed the Big Syntax Blog out there, but I’m pretty connected and well-read *cough* online, and I’d be surprised if I had truly missed it.

Now as to why syntacticians tend to have this background, why the technically-oriented ones might drift to the Big Corpus side of things, and what this all means for the field, well, those are interesting questions indeed.  It seems to be the case, for example, that a lot of syntactic theorists are getting jobs in English departments rather than, say, applied math or logic positions.  (More anecdotal experience.)

And as to what it all means, well, it means that syntactic theory is susceptible to criticism from the camp on the opposite side, the “European-style” logicians and formal grammarians—a criticism to which I am much more sympathetic than the claims of post-Chomskyans/neo-empiricists. (And about which I intend to write a post in the not-to-distant future!) But it also means that, regardless of who is right about these matters, syntax is not growing its base in places that it needs to grow its base, insofar as academic blogs are potential incubators of future collaborators and grad students.  And I believe that they are these days to a goodly extent.

And unfortunately it kind of also means that a lot of syntacticians will only be dimly aware that these issues are being revisited, even if the arguments aren’t really all that different from the ones that have been made in the past.  I am definitely sympathetic to people who might think we’ve been here and done that, like, 50 years ago.  So it goes.

Recent events have caused me to reconsider firing up this blog again.  Although I’ve been having a ball on Twitter. In the meantime:

Not long after I named this blog, I realized that the title of this blog has a certain, um, unfortunate acronym.  But I quickly decided that I wouldn’t consider changing it.  It just amuses me to think that someone might one day say to themselves, “Asad is writing BS!” and my spirit self would think “Yes, yes he is.” So, I think it’s perfect.

For those not as familiar with theoretical syntax and wondering what a “derivational cycle” might be…hoo, boy.  One of the criticisms of theoretical linguistics of the so-called “Chomskyan” variety (that UMD linguistics practices with gusto) is that it has its head in the formalistic clouds, far away from language, but never far enough away that it can be described with a great deal of mathematical precision.  Cyclicity in syntax is both a prime example of this, and one of the most important and IMO convincing and interesting aspects of the approach.

But one simple way of thinking about it is that there are definite limitations to the scope of question words in a sentence, and that these limitations happen in “cycles” roughly—but not strictly—defined by nested clauses.  Making the case requires a lot of examples and reams of PhD theses, but here’s an illustrative pair:

• Why was the man sleeping in the boat?
• What boat was the man sleeping in?

We can extend both questions by adding another clause-embedding, in a sense recursively (to appeal to CS sensibilities).

• Why did you tell the reporter that the man was sleeping in the boat?
• What boat did you tell the reporter that the man was sleeping in?

In the “why” case, the shorter question asks the reason for sleeping in the boat, but the longer question no longer allows that interpretation.  We are instead forced to interpret that it was asking why “you [told] the reporter” about it.  In other words, introduction of the “that” seems to have had an effect of “blocking” the question from applying to the later clause.

But not so for “What boat”!  There is therefore something special about the introduction of a clause boundary in English that blocks some interpretations of questions but not others.  We can extend these examples further, and into other languages.  As clauses can be nested further, we can suggest that these phenomena are therefore in some sense cyclic or recursive, yet apply to very abstract human faculties of interpretation.

This particular class was a review of Chomsky’s classic Barriers monograph (in a nutshell, how clause boundaries act as barriers to certain interpretations) followed by a review of Lasnik and Saito’s work (that’s UMD’s Howard Lasnik) on “proper government”, which elaborates on some of the conditions that permit barriers to form through characteristics of abstract variables called “traces”.  Each of these, however, would take me hours to summarize, so I won’t, at this point.

• Wilson and Wiebe 2003, Annotating Opinions in the World Press. (PDF)
• Ruppenhofer, Somasundaran, and Wiebe 2008, Finding the Sources and Targets of Subjective Expressions. (PDF)
• Wilson, PhD Thesis, 2008, Fine-Grained Subjectivity and Sentiment Analysis. (PDF, we covered only chapter 7)

The first paper mainly covers the basic effort in MPQA sentiment annotation—how to break down the problem so as to achieve consistent annotation and how to measure inter-annotator agreement.  The basic paradigm that Wiebe and her team use is to view sentiment annotation as being about a mapping between stretches of text and the “private states” of an opinion-holder.  A private state is simply a property of an opinion-holding entity that is not independently verifiable. In other words, a private state is a description of subjectivity.

We can thus develop an ontology of private states and the holders of said states and how they are reflected in text.  There are two major categories of private state expressions: expressive (implicit) subjectivity and direct (explicit) subjectivity.  “Bob squashed the hated insect” is a statement in which the hatedness of the insect is clearly a subjective statement, but it’s not directly attributed to Bob—even though in the context it could be Bob’s opinion.  “Bob claims he hates insects,” on the other hand, is a statement directly made by Bob.

It was interesting to watch the reactions of some of my fellow classmates.  Most of them come from either straight-up CS or hardcore statistical NLP, so fine-grained philosophical distinctions of subjectivity are not day-to-day staples in their work, and some of them expressed on the course mailing list and in class that reading the papers required some amount of mentality shift.  Philip went through some of the techniques of linguists in making these distinctions, including how linguists change the contexts of statements in order to establish tests for the linguistic properties of statements.

Sentiment analysis seems to be one of those places where a stronger bridge between linguistics and applied NLP can be made.

The second paper followed much in the same vein as the first, except that it emphasized the extent to which current NLP techniques cannot yet handle some of the distinctions in the MPQA annotation.  In particular, semantic role labeling—a family of existing techniques for establishing grammatical dependencies—cannot be used to directly infer some of the participants in a private state expression.  For example, when the holder of an opinion is only implied, the semantic role labeling as we currently conceive it will never find it.

The last reading dealt with some additions to the MPQA made by Theresa Wilson, particularly in the addition of target/topic annotations to the MPQA as well as subdividing opinion types into “attitudes” like “sentiment” and “arguing”.

Having fooled around a bit with the MPQA myself, I had the opportunity to show to the class a little bit of what it looked like, and what the challenges of using a somewhat inconsistent standoff annotation format could be.  Philip also took the opportunity to try out some collaborative annotation of a small passage of text with hilarious results—in the amount of arguing it took to decide the subjectivity of even small stretches of text.  Assigning subjectivity is too subjective! In that sense, the high inter-annotator agreement in the original MPQA effort seems somewhat surprising, which had been pointed out on the mailing list before the class.

Couple this with other sources of annotation down the pipeline, and it’s at the very least a long series of the tedious and repetitive kind of programming, not the fun kind.

Now, of course, some portion of my nonexistent readership will pipe up: what about UIMA?  UIMA is very nice in theory, but it is a big Javafied over-engineered mess (to be blunt).  I and teammates used it last year for an information extraction project that is now on ice, and it was one of those cases where the cure was worse than the disease. Modularization and reusability and generality and safety and all that software engineering is all very nice and elegant in theory, but it imposes a steep and unacknowledged price in programmer usability, especially since we were trying to integrate it into an annotation pipeline heavily dependent on tools I had already created in Python.

(Someone really needs to rein in software engineers.  UIMA reminded me of the UML, whose problems were apparent in the 90s for similar reasons.  Note in the Wikipedia article the problem of OO orthodoxy in UML.  As scripting languages have evolved and become popular, UML becomes even more difficult to conceive as a practical way of translating design to implementation.)

So until someone comes up with a nice, friendly way of mangling multiple NLP-related annotation streams together in Python and Perl and whatever, without expecting enormous Eclipse workbenches and UML-ish software-engineering doodads, I guess I’m just going to stick to combining annotations via irritating ad hoc scripts.