A quantitative defense of linguistic methodology by Jon Sprouse & Diogo Almeidaa

It is often argued that the current divide between linguistic theory and other domains of language research can be traced to the unreliability of linguistic methodology, and the resulting unreliability of linguistic theory. Linguists rely upon informal acceptability judgment experiments as the primary source of linguistic data. These experiments tend to employ small samples, often composed of non-naïve participants, and generally forego the use of distracter items and inferential statistics in the analysis of the results. We present resampling analyses of a large acceptability judgment dataset that provides new quantitative evidence that traditional linguistic methodology is extremely reliable with very small samples, usually at the level of the individual participant. We also review the evidence that critics of informal linguistic experiments provide to justify their concerns, which shows that the evidence provided by critics turns out to mostly corroborate the reliability of informal judgments, and not undermine it. We conclude that there is no empirical, logical, or statistical reason to think that the informal experiments routinely performed by linguists are unreliable. In fact, we show evidence that these experiments might be, in some circumstances, much more powerful than formal experiments with naïve participants. Given the lack of evidence of problems with traditional linguistic methodology, we hypothesize that one potential reason for the recurrence of this debate is that the phenomena critics are particularly interested in often elicit effects that are very small and hard to detect in formal acceptability judgment tasks. This suggests that critics may be mistaking a property of specific phenomena for a property of the methodology. Taken as a whole, these results suggest that broad criticisms of linguistic theory based on the reliability of linguistic data are unfounded, and that methodological concerns should not influence the relationship between linguistic theory and other domains of language research.

Looks like a good new resource, if they keep it going. Two useful posts so far.

NASSLLI 2010 is Open for Registration!

Fourth North American Summer School in Logic, Language, and Information NASSLLI 2010

June 20-26, 2010

http://www.indiana.edu/~nasslli/

The North American Summer School in Logic, Language, and Information (NASSLLI) is a summer school with classes in the interface between computer science, linguistics, and logic.

After previous editions at Stanford University, Indiana University, and UCLA, NASSLLI will return to Bloomington, Indiana, June 20–26, 2010. The summer school, loosely modeled on the long-running ESSLLI series in Europe, will consist of a number of courses and workshops, selected on the basis of the proposals. Courses and workshops meet for 90 or 120 minutes on each of five days, June 21–25, and there will be tutorials on June 20 and a day-long workshop on June 26. The instructors are prominent researchers who volunteer their time and energy to present basic work in their disciplines. Many are coming from Europe just to teach at NASSLLI.

NASSLLI courses are aimed at graduate students and advanced undergraduates in wide variety of fields. The instructors know that people will be attending from a wide range of disciplines, and they all are pleased to be associated with an interdisciplinary school. The courses will also appeal to post-docs and researchers in all of the relevant fields.

We hope to have 100-150 participants. In addition to classes in the daytime, the evenings will have social events and plenary lectures. Bloomington is a wonderful place to visit, known for arts, music, and ethnic restaurants. All of this is within 15 minutes walking from campus. We aim to make NASSLLI fun and exciting.

• Leiter on King’ College London firing Shalom Lappin
• Shalom’s statement

The journal Nature Chemical Biology has an editorial that is well worth reading and pondering for other fields as well:

• 2010. On citing well. Nature Chemical Biology 6:2. p.79. doi:10.1038/nchembio.310.

Unfortunately, the editorial is behind a paywall. Here are the main points in excerpts.

The appearance of new ideas and discoveries in the scientific literature is a reflection of ongoing scientific progress. Individual articles are nodes of scientific knowledge, but citations of published work link together the concepts, technologies and advances that define scientific disciplines. Though information technology and databases have helped us to better manage the expanding scientific literature, the quality of our citation maps still hinges on the quality of the bibliographic information contained in each published paper. Because article citations are increasingly used as metrics of researcher productivity, the citation record also affects individual scientists and their institutions. As a result, all participants in the scientific publication process need to ensure that the citation network of the scientific literature is as complete and accurate as possible.

Many factors may stand in the way of good citation practices. [...] Each research group has its own referencing habits, and some may feature their own work too prominently or rely on familiar references without a critical examination of whether a particular citation is the most appropriate in the given context. Some researchers may not cite ‘old’ papers either because these are incorrectly viewed as being out of date or because inertia inevitably may encourage authors to cite the articles that show up more frequently in searches or that have appeared recently.

Researchers understandably are motivated, in both professional and personal ways, to have their scientific contributions recognized through citation by their peers. The community also values the accurate assignment of credit and precedence for scientific discoveries. As a result, even an accidental omission of a necessary citation may create an uncomfortable situation for a paper’s authors. More problematic, however, are cases where authors deliberately omit relevant citations. Because perceived novelty can be an important factor in determining where a manuscript is published, some authors may be tempted to avoid citing earlier or concurrent work from their own laboratories to enhance the apparent advance of a submitted study. In other cases, some authors may consider ongoing scientific disagreements, personal conflicts or competition a sufficient justification for omitting citations of work by others. Clearly authors need to do everything they can to avoid accidental omission of key references, and should never exclude relevant citations for nonscientific reasons. In turn, all scientists, independent of their roles as authors, referees or editors, need to renew their commitment to guaranteeing that literature citations correctly assign credit for ideas and discoveries and are placed thoughtfully in manuscripts and published papers.

Though editors and referees can help, authors are ultimately responsible for the information contained in their published papers. We recommend that authors take several important steps to increase the quality of their citation lists. First, principal investigators need to teach young scientists the appropriate ways to select manuscript references and mentor them in the ethical dimensions of citation. Second, authors need to put as much care into selecting and accurately citing references as they devote to the rest of their manuscripts. As part of this process, authors should perform comprehensive literature searches as they write and revise manuscripts, so as to identify relevant work that may need to be cited. Before including references in their citation lists, all authors should have read and discussed the candidate references to ensure that they are the most relevant choices and are called out at the appropriate point in the paper.

The responsibility for maintaining and enhancing the citation network of a discipline resides with all participants: authors, referees, editors and database managers. Thoughtful attention during the writing and review processes remains the first and best approach for ensuring citation quality and the appropriate assignment of credit in published papers. Yet new publishing and database tools that lead us to an interactive multidimensional scientific literature will become essential. As publishers move toward integrating functionality such as real-time commenting on published papers and creating ‘living manuscripts’ that preserve the snapshot of a research area through the lens of a published paper, while permitting forward and backward linking, the scientific literature is poised to become a richer environment that will support future scientific progress.

At S&P, we are fully committed to these goals, but could surely work harder to improve the citation practices. One of our criteria for evaluating submissions is “contextualization of research”, as spelled out in our inaugural editorial:

Are the main research questions contextualized in terms of earlier related work? Does the paper adequately cite related work? Could the impact of the paper be improved through modifications that would show the relevance of the results to future work in the same or other fields?

Advice to authors: by contextualizing results appropriately, the author not only increases the worth of the paper to the audience, but also makes the job of the editors and reviewers easier. It will be much easier for us to be sure that a paper should be published if we can clearly see what previous work it betters. Authors would do well to flag, both in the abstract and early on in the paper, the relationship of the paper to earlier proposals, and to indicate in broad terms what the relative advantages of the new approach are. Of course, it is then incumbent on the author to make sure that all such claims are fully justified in the main text of the article.

Another aspect of this is that once the relevant citations have been chosen, the bibliographic detail given in the article needs to be as complete and clear as possible:

• full first names of authors and editors
• both volume and issue numbers for journal articles
• page numbers for everything that appeared on numbered pages
• DOIs for every work that has a DOI (important both for easy access by readers to the cited literature and for all kinds of automated processes, present and future)
• URLs for unpublished manuscripts and other obscure sources
• conference proceedings formatted as specified in the Unified Style Sheet for Linguistics

S&P strives towards citing well, which requires continuous attention from authors, reviewers, and editors.

• Rick Nouwen: Two kinds of modified numerals, doi:10.3765/sp.3.3.

Keep on rolling.

Today marks the start of what we hope will be a banner year for our journal. We just published two articles:

• a main article by Sigrid Beck on “Quantifiers in than-clauses”
• a commentary by Uli Sauerland on “Embedded Implicatures and Experimental Constraints: A Reply to Geurts & Pouscoulous and Chemla”.

We have three more main articles in production, which should all appear quite soon:

• Donka Farkas & Henriëtte deSwart: “The semantics and pragmatics of plurals”
• Thony Gillies: “Iffiness”
• Rick Nouwen: “Two kinds of modified numerals”

We also have two or three more commentaries in various stages of submission/production, and are always soliciting commentaries on any of our main articles.

Finally, there are four papers under current review, we’re expecting revised versions of a number of manuscripts, and we’re awaiting several manuscripts that have been promised to us.

All in all, the journal is ramping up phenomenally and this will be the year that the quantity of our output will reach the levels of the other three main journals in our field.

Please help us make the journal more widely known and please submit your work to us. You’ll get excellent reviewing and editorial service and your work will look great thanks to our superior typography and it will be published free of charge and openly accessible to everyone.

The latest version is now available as a pre-print:

• von Fintel, Kai & Anthony S. Gillies. 2010. Must … stay … strong! URL http://mit.edu/fintel/fintel-gillies-2010-mss.pdf. Final pre-print, to appear in Natural Language Semantics.

We are grateful to NLS‘s reviewers and many colleagues who have given us useful feedback over the years (!) that we have been crafting this article.

Since the article is appearing in a Springer journal, we had to sign away our copyright. But MIT has negotiated fairly good terms with Springer that make their policies more or less compatible with our Open Access Policy. We are allowed to make the final pre-print openly accessible, including through MIT’s DSpace archive (where it will appear once the publisher’s version is out on the streets).

Here’s a reproduction of a figure from a recent article in Mind and Language by Pietroski et.al.:

And here is a reproduction a figure from a recent article in Semantics and Pragmatics by Chemla:

No comment.

When one looks at a sentence like Grijpstra is playing the drums again, it makes a lot of sense to think that the sentence conveys two propositions: that Grijpstra is playing the drums and that Grijpstra has played the drums before. The first proposition is asserted as the new information, while the second proposition somehow has the status of old news, as being taken for granted, as presupposed. And so, it makes a lot of sense to think that it is a good idea to build a system of semantic interpretation that assigns to a sentence two separate propositions, its assertion and its presupposition. Perhaps the best known such system was developed by Lauri Karttunen and Stanley Peters in their seminal 1979 article “Conventional Implicature” (why they used that term rather than “presupposition” is an obscure story of its own).

After 52 pages of carefully devising a two-dimensional architecture for presuppositions and after giving an appendix with painstaking logical rules for their system, Karttunen & Peters drop a bombshell in the form of a one-page “note”. They note that their system cannot deal with the sentence Someone managed to succeed George V on the throne of England, which they observe is an odd thing to say. Their system correctly predicts that the sentence asserts that someone succeeded George V to the throne, but it incorrectly predicts that the presupposition is that it was difficult for someone to do that. If their predictions were correct, there should be nothing wrong with the example, since someone did succeed George V and since doing so was difficult (in fact, impossible) for anyone else.

Karttunen & Peters conclude that their two-dimensional system needed fixing by finding a “way of linking the choice of a person who is implicated to have difficulty to the choice of a person who is asserted to have succeeded”. They expected that this deficiency would be “remedied through further research” but that the task would not be “a trivial one”. In fact, they point out that “the problem arises directly from the decision to separate what is communicated in uttering a sentence into two propositions”. So, their note and their example spelled doom for two-dimensional approaches to presupposition.

One possible defect in their example is that it uses the presupposition trigger manage and it turns out to be quite hard to figure out what manage presupposes. Karttunen & Peters cite an article by Linda Coleman (“The case of the vanishing presupposition”) in the proceedings of the first annual meeting of the Berkeley Linguistic Society, which shows that the presupposition of manage is rather elusive. It turns out that Coleman’s proposal is perhaps the first place something like the Strongest Meaning Hypothesis was proposed. She argues (roughly) that manage has a set of progressively weaker meanings and that in any given occurrence the strongest sensible meaning is perceived. (Coleman’s paper is quite properly cited — although with the wrong title — in the seminal paper on the Strongest Meaning Hypothesis by Mary Dalrymple, Makoto Kanazawa, Yookyung Kim, Sam Mchombo, and Stanley Peters, the same Peters as in Karttunen & Peters.)

There is a perhaps not unreasonable reaction to the particular example using manage that would explain its oddness without dooming the two-dimensional set-up. Perhaps, manage simply presupposes that its complement is something that is difficult to do, that is, it simply presupposes something about the verb phrase without connecting to the subject at all. Perhaps, what is odd about their example is that succeed George V on the throne of England is simply not difficult (for the relevant people) or the question of difficulty just doesn’t arise for that property.

In any case, the problem can be illustrated with other examples. In fact, there is another famous example in this story. When Irene Heim in her 1983 WCCFL paper “On the Projection Problem for Presuppositions” turns to the problem, she uses a different example sentence (without any comment as to why): “Karttunen & Peters (1979) point out a difficulty with sentences like A fat man was pushing his bicycle. Their rules assign to [this sentence] a presupposition that they admit is too weak: that some fat man had a bicycle. On the other hand, a universal presupposition that every fat man had a bicycle would be too strong. What one would like to predict is, vaguely speaking, a presupposition to the effect that the same fat man that verifies the content of [the sentence] had a bicycle. But it is neither clear what exactly that means nor how it could be worked into K&P’s theory.”

In the same year 1983, in his book Quantification and syntactic theory, Robin Cooper says that a one-dimensional representation like there is an x: x succeeded George V & PRESUPPOSED (it was difficult for x to succeed George V) would be better than the two-dimensional Karttunen & Peters analysis. Ever since, similar proposals continue to be made (for example, quite recently Paul Dekker’s article “A Multi-Dimensional Treatment of Quantification in Extraordinary English” in L&P — Dekker uses a representation very much like Cooper’s but then proceeds to give a two-dimensional semantics, which suggests that the main problem with K&P’s system was that it had two unconnected representations of their sentence rather than that there are two meanings associated with the sentence).

But wait a minute. What would it even mean to say about the sentence that it claims that someone succeeded George V, about whom it is presupposed that he found it difficult to do so? How can it be that something is supposedly already being taken for granted about someone who is not even introduced into the discourse until the very sentence is uttered? Heim recognizes this problem and suggests that one way this could be is if it was already presupposed that everyone had this property (since then whoever is introduced indefinitely by the sentence would be entailed to verify that presupposition). But this is clearly not so in her fat man example. It is not reasonable to think that it would be presupposed that every fat man has a bicycle and so it is unreasonable to think that about some indefinite fat man we already have the presupposition that he has a bicycle. Heim’s conclusion is that in effect there is no presupposition here, even though there is a presupposition trigger; she gets rid of the presupposition by the process she dubbed “local accommodation”. In the end, it is as if the sentence asserts of some fat man that he has a bicycle and that he pushed it. The Case of the Vanishing Presupposition, indeed. These are clearly very tricky issues, so it’s no surprise that the George V example and the Fat Man example continue to be debated.

One recent development: Karttunen & Peters’ problem of binding across dimensions was adduced by Chris Potts in his work on conventional implicatures (this time, the term was more appropriately used). He argued that while K&P’s problem was indeed a problem for them since they wanted to use a two-dimensional system to treat something that is better treated in a single dimension, there are indeed ingredients of meaning (conventional implicatures) that are located in a separate dimension. And lo and behold, Potts says, there what K&P predict, that there can’t be binding across dimensions, is actually correct. For example, he observes, a quantifier cannot bind into an appositive relative clause: No reporter believes that Ames, who is often the subject of his columns, is a spy. The sentence doesn’t have a reading where the quantifier no reporter binds the pronoun his in the appositive. So, Potts says, it makes sense to treat appositives as operating in a second dimension, and K&P’s binding problem becomes a virtue of the analysis.

A wrinkle: as observed by Danny Fox in our seminar a couple of weeks ago, sentences like Every candidate thinks that his wife, who is of course his biggest supporter, will vote for him or No candidate suspects that his wife, who is after all his biggest supporter, will vote against him actually seem to have readings where binding into the appositive occurs. It seems that once the anchor of the appositive (his wife) contains a bound variable, then the appositive can as well. If these observations hold up, appositives would also appear to have to be treated in one dimension, perhaps the way that Philippe Schlenker will sketch at NELS 40 in a month.