Linguistic InfRastructure for Interoperable ResourCes and Systems

Semantic roles set

Annotation Guidelines

Tagset Evaluation

Publications

Semantic roles: definitions, explanations, examples

/agent/
Definition	Participant in an event who initiates and carries out the event intentionally or consciously, and who exists independently of the event.
-- Source	Adapted from: Dowty (1989), EAGLES, SIL, Sowa (2000) and UNL
Explanation	An agent may be animate, or only seemingly or perceived, as animate; this is so that cases of nonhuman agency such as a robot, an institution, or internally controlled forces and machines will not be excluded from the definition, e.g. “GM offers rebates on its new models”.
Example	[Libya Agent,e1&e2] has showne1 [interest Theme,e1] in and takene2 [steps Theme,e2] [to acquiree3 [weapons of mass destruction (WMD) Theme, e3] and their delivery systems Purpose, e1&e2] (FrameNet).

/partner/
Definition	Participant in an event who is intentionally or consciously involved in carrying out the event, but who is not the principal agent of the event, but and who exists independently of the event.
-- Source	Adapted from: FrameNet, PropBank (‘Second Party’) and UNL
Explanation	The semantic role of partner differs from agent principally in that the participant in question is performing the action in accompaniment to the agent, and is not the primary focus. Sometimes partner is also known as ‘accompaniment’ i.e. SIL and Sowa (2000). UNL distinguishes between a co-agent and a partner, but this distinction is felt to be unclear: a co-agent is simply another agent.
Example	[On 19 December 2003 Time, e1], [Libyan leader col. Muammar Gadhafi Agent,e1] [publicly Manner, e1] confirmede1 [his commitmente2 [to disclosee3 and dismantlee4 [WMD programs Patient, e3&e4] [in his country Location, e3&e4] Theme,e1] Purpose, e2] [following [a nine-month period Duration, e5] of negotiationse5 [with US and UK authorities Partner,e5] Reason, e1]. (FrameNet)

/cause/
Definition	Participant in an event (that may be animate or inanimate) that initiates the event, but that does not act with any intentionality or consciousness; it exists independently of the event.
-- Source	Adapted from: SIL (‘Causer’) and Sowa (2000) (‘Effector’)
Explanation	Except for the lack of intentionality of the participant, this semantic role is very similar to that of the agent and in fact shares all its other properties. The role of cause can often be identified with verbs of initiation, or causation, such as: ‘to cause’, ‘to produce’, ‘to start’, ‘to originate’, ‘to occasion’, ‘to generate’.
Example	[Signing the protocol Cause, e1] would ensuree1 [IAEA Beneficiary, e1] [oversight over Libya's nuclear transition from weapons creation to peaceful purposes Reason, e1]. (FrameNet)

/instrument/
Definition	Participant in an event that is manipulated by an agent, and with which an intentional act is performed; it exists independently of the event.
-- Source	Adapted from: EAGLES (‘Implement’), SIL, Sowa (2000) and UNL
Explanation	Some (Loos et al. 2004) would define instrument more simply as an inanimate object, which is used to implement an event. However, the semantic role of instrument can clearly also be animate (e.g. “The woman was dragged by her horse [instrument] for several meters”, or even “John [instrument] threw himself at the door in a rage”), and can also take part in a state as well as an event (e.g. “The tarpaulin is tied down with rope [instrument]”).
Example	[In 2003 Time, e1], Libya admittede1 [its previous intentions to acquiree2 [equipment Theme, e2; Instrument, e4] needede3 [to producee4 [biological weapons (BW) Result, e4] Purpose, e3] Theme, e1]. (FrameNet)

/patient/
Definition	Participant in an event that undergoes a change of state, location of condition, that is causally involved or directly affected by other participants, and exists independently of the event.
-- Source	Adapted from: EAGLES, Sowa (2000) and UNL (‘Object’)
Explanation	Patient is distinguished from the semantic role of theme principally in that it is affected or changed by the event.
Example	[White women Agent, e1&e2] servee1 [tea and coffee Theme, e1] , and then washe2 [the cups and saucers Patient, e2] [afterwards Time, e2] . (PropBank)

/pivot/
Definition	Participant in a state that is characterised as being in a certain position or condition throughout the state, and that has a major or central role or effect in that state. A pivot is more central to the state than a participant in theme role.
-- Source	Webster New Collegiate Dictionary (1976)
Example	[Vicar Marshall Agent, e1; Pivot, e2] admitse1 [to mixed feelingse2 [about this issue Theme, e2] Theme, e1]. (FrameNet)

/theme/
Definition	Participant in a state or event that in an event, it is essential to the event taking place but it does not have control over the way the event occurs and is not structurally changed by the event; and in a state, it is characterised as being in a certain position or condition throughout the state, and it is essential to the state being in effect; but it is not as central to the state as a participant in pivot role.
-- Source	Adapted from: EAGLES, Sowa (2000) and UNL (‘Object’)
Explanation	Theme is distinguished from the semantic role of patient principally in that it is not affected or changed by the event.
Example	[One man Agent, e1] wrappede1 [several diamonds Theme, e1] [in the knot of his tie Final_Location, e1]. (PropBank)

/beneficiary/
Definition	Participant in a state or an event that is advantaged or disadvantaged by the event or state.
-- Source	Adapted from: EAGLES, SIL, Sowa (2000) and UNL
Explanation	The roles of beneficiary and recipient differ in that the role of recipient is the ultimate target of an action, whereas beneficiary is not. Also, the recipient cannot take part in a state.
Example	[U.S. Trust Agent, e1] [recently Time, e1] introducede1 [certain mutual-fund products Theme,e1] , which allowe2 [it Beneficiary, e2] [to servee3 [customers Beneficiary, e3] Purpose, e2]. (PropBank)

/source/
Definition	Participant in an event that is the (non-locative, non-temporal) start point of an action. The source exists independently of the event.
-- Source	Adapted from: Sowa (2000)
Example	[Eaton Beneficiary, e1] earnede1 [from continuing operations Source, e1]. (PropBank)

/goal/
Definition	Participant in an event that is the (non-locative, non-temporal) end point of an action. The goal exists independently of the event.
-- Source	Adapted from: Sowa (2000)
Explanation	The goal differs from a beneficiary in that there need not be a clear benefit and only events can have a participant towards whom the action is directed.
Example	[The executive Agent, e1] recallse1 [[Mr. Corry Agent, e2] whisperinge2 [to him and others Goal, e2] Theme, e1]. (PropBank)

/result/
Definition	Participant in an event that comes into existence through the event. It indicates a terminal point for the event: when it is reached, then the event does not continue.
-- Source	Adapted from: Sowa (2000)
Explanation	Result is the completed point of a process, and unlike goal is dependent upon the event for its existence.
Example	[Within the past two months Duration, e1] [a bomb Patient, e1; Cause, e2] explodede1 [in the offices of the El Espectador in Bogota Location, e1], [destroyinge2 [a major part of its installations and equipment Patient, e2] Result, e1] (PropBank)

/reason/
Definition	Participant that represents the set of facts or circumstances explaining why a state exists or an event occurs.
-- Source	Adapted from: Sowa (2000) (‘Matter’) and UNL
Explanation	The role of reason can be distinguished from that of purpose, because purpose indicates the objective or goal of an agent that acts intentionally. However, the role of agent is not applicable to states. Reason is also different from manner and method in that reason describes why the event is being carried out, while manner and method describe how it is being carried out.
Example	[Elisa Hollis Agent, e1] launchede1 [a diaper service Result, e1] [last year Time, e1] [because [State College , Pa. Pivot, e2] didn't havee2 [one Theme, e2] Reason, e1]. (PropBank)

/purpose/
Definition	Participant that represents the set of facts or circumstances that describe what an agent wishes or intends to accomplish by performing some intentional action.
-- Source	Adapted from: EAGLES and UNL
Explanation	The notion of purpose implies intentional action. This property of purpose makes it impossible to have this semantic role in a state. The role of purpose differs from that of reason in that purpose describes the aims of an agent, whereas reason indicates why the event is carried out or the state is true. Contrast the following examples: (a) “The authorities extradited him to the US for drug trafficking [reason]” and (b) “The authorities extradited him for trial [purpose] in the US”.
Example	[Two steps Theme, s1] ares1 [necessary Attribute, s1] [to translatee1 [this idea Patient, e1] [into action Result, e1] Purpose, s1] (PropBank)

/time/
Conceptual Domain	/initialTime/ /finalTime/
Definition	Participant that indicates an instant or an interval of time during which a state exists or an event took place.
-- Source	Adapted from: EAGLES, SIL, Sowa (2000) and UNL
Explanation	Just as with location, time is divided into three other subroles: beginning, end and duration.
Example	[Right now Time, e1] [[about a dozen Amount, e1] laboratories Agent, e1&e2], [in the U.S. , Canada and Britain Location, e1] , are racinge1 [to unmaske2 [other suspected tumor-suppressing genes Theme, e2] Purpose, e1]. (PropBank)

/manner/
Definition	Participant that represents the way or style of performing an action, or the degree/strength of a cognitive or emotional state
-- Source	Adapted from: EAGLES, SIL and UNL
Explanation	It should be noticed here that the role of manner differs from instrument in that manner describes an event as a whole, whereas instrument characterises one of the components of event or state; manner is an abstract thing whereas instrument is a concrete one (the latter also differentiates instrument from means and method). The role of manner includes secondary effects (quietly, loudly), and general descriptions comparing events or states (in the same way). It may also indicate salient characteristics of theme, experiencer, agent, etc., e.g. coldly, deliberately, eagerly, carefully, etc.
Example	[These rate indications Theme, s1] ares1 n't [directly Manner, s1] comparables1.(PropBank)

/medium/
Definition	Participant that represents the physical setting, entity or channel used by an agent or agents in an event.
-- Source	Adapted from: Sowa (2000)
Explanation	As the medium always implies the presence of an agent because it defines the physical or abstract means of conveying the action, this semantic role is not relevant for states. There is some discussion about the difference between the semantic role of medium and that of instrument. To illustrate the problem, consider the example: “Joanna peeked at John through the binoculars”. Here ‘the binoculars’ could be said to be the instrument used to perform the action of peeking, whereas ‘through the binoculars’ could be said to be the medium. The main difference is that while medium is only ever a semantic role for a participant in an event, instrument can be present for a state as well.
Example	[They Pivot, s1; Agent, e1] coulds1 seee1 [the 23 pairs of chromosomes Theme, e1] [in the cells Location, e1] [under a microscope Medium, e1]. (PropBank)

/means/
Definition	Participant in an event that represents a procedure for performing the action in terms of component steps, . or a method by which an intentional act is performed by an agent. A means does not necessarily exist independently of the event.
-- Source	Adapted from: UNL
Explanation	The role means differs from instrument in that means describes means and methods for doing something, which may have no existence independent of the event, while an instrument describes previously existing tools. This distinction is exemplified by: (a) “I sliced the cucumber in 1/8th inch slices with a knife [instrument]” and (b) “I sliced the cucumber in 1/8th inch slices by marking the intervals with a ruler [means]”. Sometimes this distinction is not very clear-cut: (c) John ate the sushi with chopsticks [instrument]” and (d) John ate the sushi using chopsticks [means]”.
Example	[Sears Agent, e1] blanketede1 [the airwaves Patient, e1] [with ads about its new pricing strategy Means, e1] (FrameNet)

/setting/
Definition	Participant that represents a set of circumstances of the occurrence of an or a state.
-- Source	Adapted from: EAGLES (‘Scene’), UNL (‘Condition’) and FrameNet (‘Circumstances’).
Example	[A number of medical and agricultural research centers Pivot, s1; Intsrument, e1] hads1 [the potential Attribute, s1] to be usede1 [in BW research Setting, e1].(FrameNet) Here comee1 [the ringers Agent, e1&e2] [from above Initial_Location, e1], makinge2 [a very obvious exit Theme, e2] [while [the congregation Pivot, s1] iss1 [at prayer Setting, s1] Time, e1&e2] (FrameNet)

/location/
Conceptual Domain	/initialLocation/ /finalLocation/
Definition	Participant that represents the place where an event occurs, or a state that is true.
-- Source	Adapted from: EAGLES (‘Place’), SIL (‘Locative’) and Sowa (2000)
Example	[Here Location, s1] iss1 [an example Theme, s1]. (FrameNet) [They Patient, e1] aren't acceptede1 [everywhere Location, e1] (FrameNet) [The stairs Theme, s1] are locateds1 [next to the altar Location,s1] (FrameNet)

/initialLocation/
Broader Concept	/location/
Definition	Participant that indicates the place where an event begins or a state becomes true.
-- Source	Adapted from: EAGLES, Sowa (2000) (‘Origin’) and UNL (‘Initial Place’)
Example	Here comee1 [the ringers Agent, e1&e2] [from above Initial_Location, e1], makinge2 [a very obvious exit Theme, e2] [while [the congregation Pivot, s1] iss1 [at prayer Setting, s1] Time, e1&e2] (FrameNet)

/finalLocation/
Broader Concept	/location/
Definition	Participant that indicates a place where an event ends or a state becomes false.
-- Source	Adapted from: EAGLES, Sowa (2000) (‘Destination’) and UNL (‘Final Place’)
Example	[One man Agent, e1] wrappede1 [several diamonds Theme, e1] [in the knot of his tie Final_Location, e1]. (PropBank)

/path/
Definition	Participant that indicates an intermediate place or state or trajectory between two locations, or in a designated space.
-- Source	Adapted from: Sowa (2000)
Example	[Father McKenna Agent, e1] movese1 [through the house Path, e1] [praying in Latin Manner, e1] (PropBank)

/distance/
Definition	Length or extent of space.
-- Source	Adapted from: WordNet
Example	[Libya Agent, e1] pledgede1 [to eliminatee2 [[ballistic missiles Pivot, s1] capables1 of travelinge3 [more than 300km Distance, e3] Patient, e2] Theme, e1]. (FrameNet)

/initialTime/
Broader Concept	/time/
Definition	Participant that indicates when an event begins or a state becomes true.
-- Source	Adapted from: EAGLES, Sowa (2000) (‘Origin’) and UNL (‘Initial Place’)
Example	“Harry teaches on Fridays from 10:45 [initialTime] to 12:30”

/finalTime/
Broader Concept	/time/
Definition	Participant that indicates when an event ends or a state becomes false.
-- Source	Adapted from: EAGLES, Sowa (2000) (‘Destination’) and UNL
Example	“Harry teaches on Fridays from 10:45 to 12:30 [finalTime]”

/duration/
Definition	Length or extent of time.
-- Source	Adapted from: WordNet
Example	“Terry jogged for 2 hours [distance]”

/amount/
Conceptual Domain	/distance/
Definition	Participant denoting a quantity of something else than time or space.
-- Source	Adapted from: PropBank (‘Extent’), FrameNet (‘Amount’), UNL (‘Quantity’), Sowa (2000) (‘Amount’/’Measure’) and EAGLES (‘Quantity’).
Example	[The ruble Theme, s1] iss1 n't worths1 [much Amount, s1]. (PropBank)

/attribute/
Definition	Property of an entity or entities.
-- Source	Adapted from: FrameNet (‘Parameter’), UNL, Sowa (2000) and EAGLES.
Example	[A number of medical and agricultural research centers Pivot, s1; Intsrument, e1] hads1 [the potential Attribute, s1] to be usede1 [in BW research Setting, e1]. (PropBank)

/frequency/
Definition	Number of occurrences of an event within a given time span.
-- Source	Adapted from: WordNet.
Example	[President Zia of Pakistan Agent, e1] [repeatedly Frequency, e1] statede1 [that [fresh Soviet troops Patient, e2] were being insertede2 [into Afghanistan Final_Location, e2] Theme, e1] (PropBank)

LIRICS Annotation Guidelines for Semantic Roles

This document is designed to be read in conjunction with the following LIRICS deliverables:

http://lirics.loria.fr/doc_pub/D4.1_LiricsSemanticAnnotationAnalysis_011205.pdf

http://lirics.loria.fr/doc_pub/D4-2_2007-04-04.pdf

The guidelines assume the use of GATE as an annotation tool throughout.

1. Installing GATE in Windows (with semantic role annotation schemas)

For the annotation of semantic roles, we will be using the GATE annotation tool (see: http://gate.ac.uk for further details). The instructions given in this section are for installing GATE for the Windows operating system. The easiest way to install GATE is to download the most current ‘snapshot’, the most current version of the tool: (http://www.gate.ac.uk/download/snapshots/). Click on the file which ends with “-ALL.zip” (e.g. at the time of writing, “gate-4.0-a1-build2674-ALL.zip”) and unpack the contents in a suitable location on your hard disk (i.e. C:\Program Files\).

When you have GATE, you will also need to make sure that you have the most recent version of Java (JDK 6, at the time of writing) on your machine. You can get this from: http://java.sun.com/javase/downloads/index.jsp. Make sure that you delete all previous versions of Java from your machine, both JDK and runtime environments, as well. Once JDK 6 is properly installed on your machine, you should be able to run GATE by double clicking on gate.bat in the bin directory in the home directory of GATE. If the program still does not run, it is because you need to update your “Path” environment variable. Go to Control Panel, choose System ® Advanced ® Environment Variables. Make sure that the User variable CLASSPATH is set to “.” (a single dot) and add the path to Java bin (usually something like: “C:\Program Files\Java\jdk1.6.0\bin”) to the System variable Path, making sure that you separate it from any other text in the Path with a semicolon “;”.

Once all this is done, GATE should now run. However, it will run with the default annotation schemas as provided with the package. In order to set up GATE for semantic role annotation, you will need to update the gate.jar file (which can be found in the bin directory). The easiest way of doing this is to open it using WinRAR, then carry out the following two operations:

(1) Replace the old creole.xml with the new file provided by LIRICS in the gate.jar directory: \src\gate\resources\creole\.

(2) Create a new directory called “LIRICS” in the gate.jar directory: \src\gate\resources\creole\schema\ and copy SemanticRoleSchema.xml and SemanticAnchorSchema.xml provided by LIRICS into it.

Now when you run GATE and load the PropBank or FrameNet text for annotation, you will find that the correct annotation sets are displayed in the annotation tool.

There is some general documentation for GATE available on the website (http://gate.ac.uk/documentation.html), but be warned in advance, it is not very helpful!

2. Semantic role annotation

There are two types of annotation label which have been added to GATE for the annotation of semantic roles for the LIRICS project: SemanticAnchor and SemanticRole.

Before even beginning to annotate the semantic roles in a particular sentence, you must first identify the kind of SemanticAnchor to which it belongs or which it modifies. We do this because we define a semantic role as the type of relationship that a participant plays in some real or imagined situation; therefore every semantic role must be ‘anchored’ to this situation. In this document, we will show how recognising the different types of situation will narrow down the list of potential semantic roles, which can be identified, because some semantic roles will only occur within a particular type of situation.

SemanticAnchor has three possible situation types: event, state and fact. The SemanticAnchor is usually, but not always the label for the verb in a sentence (which generally also denotes or gives a strong indication of the type of situation). Rather than explain this in more detail at this point, it would be quicker to give some examples:

event: ‘John ran for five hours’ – here the event is that of running, so in this sentence we would mark up the word ‘ran’ as a SemanticAnchor of type event.

state: ‘Mary likes roses’ – here the state is indicated by a cognitive feeling or emotion. In this sentence we would mark up the word ‘likes’ as a SemanticAnchor of type state.

fact: ‘Wisdom is rare’ – here the fact is indicated by a universal application of the content in a way that does not change over time; it is not a temporary state, but something that is, was and always will claim to be true. So in this sentence, we would mark up the word ‘is’ as a SemanticAnchor of type fact.

Compare and contrast these three types in the following sentences: ‘The windowpane was broken with a hammer’, which is an event; ‘The windowpane was broken’, which is a state; and ‘All windowpanes are broken eventually’, which is a fact.

It can be quite tricky to distinguish events and states sometimes, but a simple rule of thumb is that events are indicated either by the process of some action, or by a change of something from one state into another, whereas states are indicated by no change in condition, or by some perceptive or cognitive state.

Once you have identified the SemanticAnchor in your sentence or clause, you will also need to give it a unique id (simply number these sequentially for the moment – this should be done for you automatically in a future version of the tool). The id is important as it will allow us to link every semantic role (annotated by SemanticRole) with the situation (annotated by SemanticAnchor) in which it participates.

Next you will need to annotate the semantic roles for the identified anchors in your sentence. You will annotate the participants in the sentence or clause with the SemanticRole element, then choose which SemanticAnchor this participant belongs to (by filling in the ‘anchor’ option with the same id number of the appropriate SemanticAnchor).

2.1. Examples

Once you know the type of SemanticAnchor for the sentence or clause, if the type of SemanticRole of a participant is not immediately obvious, you can begin to narrow down the possibilities for choosing the type of SemanticRole by looking at examples for the potential semantic roles in the following tables:

event

EXAMPLE

agent partner cause instrument patient theme beneficiary goal source result reason purpose manner medium means time initialTime finalTime duration setting location initialLocation finalLocation distance path frequency amount

John [agent] built the house. John built the house with Stephen [partner]. The wind [cause] broke the window. He opened the door with the key [instrument]. John painted the door [patient]. He talked about politics [theme] for hours. John sold the car for a friend [beneficiary]. He sent the letter to Mary [goal]. The researcher got his ideas from a book [source]. The tournament ended in disaster [result]. Harry couldn’t run because he was drugged [reason] Harry was drugged to make him pliable [purpose]. John ran quickly [manner] down the street. The student heard the news on the radio [medium]. He emptied the bucket by knocking it over [means]. He is teaching at five [time]. The class starts at eight [initialTime]. The class ends at nine [finalTime]. The class lasts for two hours [duration]. Libya employed chemical weapons in the conflict [setting]. She was cooking in the kitchen [location]. She pulled away from the curb [initialLocation]. The race finishes in Tilburg [finalLocation]. Terry jogged for two miles [distance] Terry jogged along the river [path]. They washed the car twice a week [frequency]. He walked to the door for the second time [amount].

state

EXAMPLE

attribute pivot instrument setting theme beneficiary reason time initialTime finalTime duration manner location initialLocation finalLocation distance amount

John has blue eyes [attribute] like my mother. Mary [pivot] believes in God. The tarpaulin is tied down with a rope [instrument]. The statue stood in the melee [setting]. Mary believes that Carlos loves her [theme]. Mary is cruel to her friends [beneficiary]. Tilburg is known for its beauty [reason]. Dinner is at six o’clock [time]. His mood was good from Friday [initialTime] … to Sunday [finalTime]. For months [duration] the train was late. The house was surprisingly [manner] clean. The statue stood in the garden [location] for years. The sea is very deep from here [initialLocation] … to there [finalLocation]. For miles [distance] the view was serene. She loved him quite a lot [amount].

Facts are a rather under-explored oddity; both events and states can often be ‘converted’ to facts by moving to the more generic case. E.g.

EVENT: The dog [agent] chased the cat [patient] ® FACT: Dogs [agent] chase cats [patient]

STATE: John [pivot] loves Mary [theme] ® FACT: Men [pivot] love women [theme]

It is therefore unsurprising to find that every semantic role possible in events and states can also occur in facts. Here are just a few examples.

fact

EXAMPLE

agent attribute pivot theme beneficiary reason purpose time initialTime finalTime duration manner location initialLocation finalLocation distance amount

Whales [agent] eat plankton. Roses are red [attribute]. Roses [pivot] are red. Roses are a type of flower [theme]. Cows give milk for men [beneficiary]. Birds fly because they have wings [reason]. Laws exist to protect citizens [purpose]. The dodo lived in Australia 300 years ago [time]. The 2^nd World War lasted from 1939 [initialTime] … to 1945 [finalTime]. The 2^nd World War lasted for 6 years [duration]. Hedgehogs make love carefully [manner]. The tiger lives in Africa [location]. Man is said to originate in Africa [initialLocation]. We all end in the grave [finalLocation]. A marathon is 26 miles [distance] long. A litre is 1.76 pints [amount].

As we can see, this only helps a bit (and hardly at all in the case of event). You can of course always read the detailed description of each role given in the document D4.2, but there are other means to pin point the type of role: namely by looking at its specific features.

2.2. Decision trees

To help distinguish particularly confusing cases, we have attempted to provide decision trees for some of the semantic roles with rather subtler differences. We will assume that some roles are self-evident and need no further clarification, namely: amount, attribute, manner, instrument, beneficiary, time, initialTime, finalTime, duration, frequency, location, initialLocation, finalLocation and distance.)

agent: animate participant that causes event in an intentional, active way, that exists independently and is unaffected (directly) by the event.

· Distinguished from partner by whether it is the main participant in the event or not; if it is, then it is an agent; if it is not, then it is a partner. In the case where this is unclear, two or more participants can be labelled as agent (main participants) separately.

· Distinguished from cause by whether it is an intentional participant in the event or not; if it is, then it is an agent; if it is not, then it is a cause.

· Distinguished from result, reason, purpose, manner, medium and means by whether it is an animate participant or not; if it is, then it is an agent; if it is not, then it is something else.

cause: participant that causes event in an unintentional, non-active way, that exists independently of the event.

· Distinguished from instrument, beneficiary, source, goal, theme and patient by whether it is the causal participant of the event or not; if it is, then it is a cause; if it is not, then it is one of the other options.

· Distinguished from result and purpose by whether it is a participant that exists independently of the event or not; if it is, then it is a cause; if it is not, then it is either result or purpose.

setting: non-locative, non-temporal participant that frames the event in some scene or subject, or abstract location.

· Distinguished from location by whether it is a participant that defines a physical location or not; if it does not, then it is a setting; if does, then it is location.

theme: passive, unaffected, non-causal, non-central participant that is essential to, but has no control over, the event or state.

· Distinguished from patient by whether it is a participant that is affected by the event or not; if it is not, then it is a theme; if it is, then it is a patient.

· Distinguished from pivot by whether it is a participant that has the most central role or not; if it is not, then it is a theme; if it is, then it is a pivot.

· Distinguished from result and purpose by whether it is a participant that exists independently of the event or not; if it is, then it is a theme; if it is not, then it is either result or purpose.

· Distinguished from goal by whether it is a participant that is the end point of the event or not; if it is not then it is a theme; if it is, then it is goal.

· Distinguished from instrument by whether it is a participant that is used by another participant to bring about the event or not; if it is not, then it is a theme; if it is, then it is an instrument.

goal: participant that is the non-locative, non-temporal end point of an event.

· Distinguished from finalLocation by whether it is a participant that is a locative end point or not; if it is not, then it is a goal; if it is, then it is a finalLocation.

· Distinguished from finalTime by whether it is a participant that is a temporal end point or not; if it is not, then it is a goal; if it is, then it is a finalTime.

· Distinguished from result by whether it is a participant that exists independently of the event or not; if it is, then it is a goal; if it is not, then it is either a result or a purpose (see below).

· Distinguished from purpose by whether it is a participant that exists independently and is not causal of the event; if it is, then it is a goal; if it is not, then it is a purpose.

· Distinguished from beneficiary by whether it is a participant that is clearly advantaged or disadvantaged by the event; if it is not, then it is a goal; if it is, then it is a beneficiary.

source: participant that is the non-locative, non-temporal start point of an event.

· Distinguished from initialLocation by whether it is a participant that is a locative start point or not; if it is not, then it is a source; if it is, then it is a initialLocation.

· Distinguished from initialTime by whether it is a participant that is a temporal start point or not; if it is not, then it is a source; if it is, then it is a initialTime.

· Distinguished from reason by whether it is a participant that is causal of the event or not; if it is not, then it is a goal; if it is, then it is a reason.

reason: participant that is the causal, independently pre-existing motive for the event taking place.

· Distinguished from result by whether it is a participant that is causal and independently existing of the event or not; if it is, then it is a reason; if it is not, then it is a result.

· Distinguished from purpose by whether it is a participant that is the independently post-existing motive for the event or not; if it is not, then it is a reason; if it is, then it is a purpose.

result: participant that is the non-causal, non-independently existing product of the event.

· Distinguished from purpose by whether it is a participant that is causal and independently existing of the event or not; if it is not, then it is a result; if it is, then it is a purpose.

means: participant that is the procedure or the method by which the event can take place.

· Distinguished from medium by whether it is a participant that is a channel used for the event to take place or not; if it is not, then it is a means; if it is, then it is a medium.

2.3. Practicalities of annotating the text

1. Articles, demonstratives and prepositions:
Should be included in the annotation of the semantic role. E.g.

[Dow Jones agent] [said event] [[it agent] would [end event] [the offer patient] [by Friday time] theme]

2. Multiple role labelling:
Participants can sometimes be labelled with more than one role. E.g.

[She agent] [dried event] [the baby patient] [under the hand-dryer location, instrument]

3. Copula + adjective:
States of being that are qualified by the use of an adjective should be annotated together as a SemanticAnchor of type state. E.g.

…[the ruble pivot] [is worth state] [little amount]
[Mary pivot] [was pregnant state]

4. Noun phrases as SemanticAnchors:
Under certain conditions (for example, ellipsis), noun phrases will also be seen as SemanticAnchors. E.g., in the example below, ‘having mixed feelings’ is implied:

[Vicar Marshall agent(“admits” event), pivot(“mixed feelings” state)] [admits event] [to [mixed feelings state] [about this issue theme(“mixed feelings” state)] theme(“admits” event)]

5. Semantic roles with embedded events:
As we have already shown in the examples given in 1. and 4. above, whole clauses can play a semantic role to a top level semantic anchor, while themselves also having multiple levels of embedding in them of other semantic anchors with their own respective semantic roles. The relationship between different levels is often a complicated one. In the example shown in 4. for instance, Vicar Marshall is both the agent of the “admits” event and the pivot of the “mixed feelings” state.

6. Idiomatic verb phrases:
In some cases the SemanticAnchor spans rather more than just the main verb of a sentence or clause, e.g. in the idiomatic verb phrase “to put in a performance”, you would mark up the whole as an anchor of type event.

7. Discontinuous annotation:
Often one semantic role may be split over discontinuous pieces of text. In this case, mark up the two pieces of text as the same kind of semantic role, anchored to the same SemanticAnchor. E.g.

[The soreness theme], [he agent] [admits event] [went away after the shot theme]

8. Misleading semantic markers:
Sometimes looking for clues for the type of semantic roles in the words will be misleading, or will cause you to overlook an important semantic role. E.g.

[Those hopes theme] were [dashed event] [when the stock market put in a relatively quiet performance reason, time]

In the above example, the word “when” would seem to indicate the semantic role type time; but if you stopped at that, then you would have missed the fact that the clause is actually a reason for the dashing of the hopes as well. The word “when” is overloaded here as it also signals a function similar to the word “because” in this context. Be sure that you think carefully about the underlying meaning of the sentence when annotating semantic roles and not just the surface form of the words.

9. Comments:
Don’t be afraid to add comments where you think that a role is not appropriate, but, at the same time, try not to overuse comments, as they will all need to be checked manually!

Tagset Evaluation

1. Completeness:

1.1. Theoretically: comparing to the tagsets defined in other projects

PropBank	FrameNet	UNL	SIL	Sowa	EAGLES	Proposed for LIRICS
ArgM_Extent	Amount	Quantity	-	Amount, Measure	Quantity	Amount
ArgM_Location	Location/ Place	Place	Locative	Location	Place	Location
Arg_from, Start State, Source	Source	Source	Source	Origin	Source	Source
Goal, End State	Goal	Goal, Final State	Result	-	Goal	Result, Goal
Causer/Forcer	Cause	-	Causer	Effector	-	Cause
Agent (verb-specific, e.g. Teller, Cutter, etc.)	Agent	Agent	Agent	Agent	Agent	Agent
ArgM_Cause	Reason	Reason	-	Matter	-	Reason
ArgM_Purpose	Purpose	Purpose	-	-	Purpose	Purpose
Benefactive	Beneficiary	Beneficiary	Beneficiary	Beneficiary	Beneficiary	Beneficiary
Experiencer (verb-specific, e.g. Feeler, Believer, etc.)	Experiencer	-	Experiencer	Experiencer	Experiencer	Pivot/Patient
Stimulus	Phenomenon	-	-	-	Stimulus	Theme
Entity, Thing, Object, Theme	Entity	Object	Patient	Patient	Object, Patient, Theme	Theme/Patient
Instrument	Instrument	Instrument	Instrument	Instrument	Instrument	Instrument
Medium, Source	Medium	-	-	Medium	-	Medium
ArgM_Manner	Means	Means/ Method	-	-	-	Means/ Method
ArgM_Manner	Manner	Manner	Manner	-	Manner	Manner
ArgM_Time	Time	Time	Time	Point in Time	Time	Time
100%	100%	77.8%	66.7%	72.2%	72.2%	100%

Mapping between VerbNet, PropBank, FrameNet and LIRICS *

VerbNet	PropBank	FrameNet	Lirics
Agent	Arg0, Arg1	Agent,  Cause, Speaker,  Cognizer, Communicator, Ingestor, Deformer, Creator, Cook, Arguer, Arraign_authority, Artist, Assailant, Assessor, Author, Authorities, Authority, Buyer, Carrier, Cause_to_shine, Claimant, Copy, Defender, Destroyer, Donor, Driver, Emitter, Employee, Employer, Entity, etc.	Agent
Actor	Arg0	Avenger, Communicator , Item, Items, Participants, Partners, Wrongdoer	Agent
Actor1	Arg0	Arguer1, Avenger, Communicator, Interlocutor1, Item_1, Part_ 1,   Participant, Participant_1, Partner,  Partner_ 1,  Parts,  Side, Speaker, Wrongdoer	Agent
Actor2	Arg1, Arg2	Addressee, Arguer2, Injured_Party, Injured_party, Injury, Instrument,  Interlocutor2, Item2, Offender, Part_2, Participant2, Participant_2, Partner2, Partner_2 Side2	Partner
Attribute	Arg1, Arg2	Attribute, Dimension, Extent, Feature, Phenomenon, Profiled_attribute, Reason, Side_2, Standard_attribute	Attribute
Beneficiary	Arg1, Arg2, Arg3, Arg4	Audience, Beneficiary, Benefitted_party, Goal, Purpose, Reason, Studio	Beneficiary
Cause	Arg0, Arg1, Arg2, Arg3	Addressee, Agent, Arguer, Cause, Communicator, Content, Donor, Evaluee, Event, Evidence, Indicator, Inspector, Internal_cause, Judge, Manipulator, Medium, Reason, Speaker, Stimulus, Support, Topic	Cause/Reason
Destination	Arg1, Arg2, Arg5	Addressee, Body_part, Context, Facility, Goal, Impactee, Location, Material, Medium, New, Path_end, Payer, Place, Recipient, Victim	Final_Location
Experiencer	Arg0, Arg1,	Addressee, Agent, Cognizer, Communicator, Experiencer, Impactee, Judge, Perceiver, Perceiver_agentive, Perceiver_passive, Reader, Speaker, Victim	Pivot
Extent	Arg2	Difference, Size_change	Amount/Distance
Instrument	Arg2	Agent, Change_agent, Fastener, Heating_instrument, Hot_Cold_source, Impactor, Injuring_entity, Instrument, Liquid, Means, Medication, Medium, Theme, Treatment	Instrument
Location	Arg1, Arg2, Arg3, Arg4, Arg5	Action, Area,  Container,  Evaluee,  Experiencer,  Fixed_location, Goal,  Goal_area,  Ground,  Impactee,  Location, Path,  Place,  Product,  Resistant_surface,  Road,  Sound_source,  Source,  State_of_affairs, Tested_property,  Time, Undesirable_location,  Undesirable_situation	Location
Material	Arg1, Arg2, Arg3	Components, Ingredients, Initial_entity, Medium, Original, Resource, Undergoer	Source
Patient	Arg0, Arg1, Arg2	Action, Addressee, Affliction, Attribute, Body_part, Child, Containing_object, Dryee, Eclipsed, Employee, Entity, Executed, Experiencer, Expressor, Flammables, Food, Food_item, Goal, Grantee, Hair, Impactee, Individuals, Ingestibles, Ingredients, Initial_entity, Item, Items, Member, Object, Original, Parts, Patient, Self_mover, Sleeper, Source, Speaker, State_of_affairs, Target, Theme, Undergoer, Victim, Whole, Whole_patient	Patient
Patient1	Arg0, Arg1	Concept_1, Connector, Fastener, Item, Item_1, Part_1, Whole_patient	Pivot
Patient2	Arg2, Arg3	Concept_2, Containing_object, Goal, Item_2 ,Part_2	Patient
Predicate	Arg1, Arg2	Action, Category, Charges, Containing_event, Contrast, Finding,  Location_of_event, Manifestation_of_bias, Manner, Medium, Message, Name, Offense,  Purpose, Reason, Role, Side_, State, Topic	-
Product	Arg1, Arg2, Arg4	Category, Copy, Created_entity, Entity, Final_entity, Flamables, Invention, New_idea, Produced_food, Product, Production, Result, Text	Result
Proposition	Arg1,  Arg2	Act, Action, Assailant, Attribute, Content, Field, Goal, Goods, Item_, Means, Message, Parameter, Path, Position, State_of_affairs, Task, Text, Topic, Value_, Value_, Victim	-
Recipient	Arg1, Arg2, Arg3	Addressee, Audience, Authorities, Buyer, Cognizer, Hearer, Recipient	Goal
Stimulus	Arg1,	Emotion, Emotional_state, Phenomenon, Text	Theme
Theme	Arg0, Arg1, Arg2	Accused, Act, Action, Activity, Addressee, Affliction, Air, Attribute, Audience, Baggage, Beam, Behavior, Benefit, Body_part, Charges, Chosen, Co-participant,  Co-resident, Competition, Content, Contents, Copy, Cotheme, Defendant, Dispute,  Emission, Emotion, Employee, Entity, Evader, Evaluee, Event, Eventuality, Excreta, Fact,  Feature, Figure, Fine, Fluid, Focal_entity, Focal_participant, Formal_realization, Goal, Goods, Grantee, Ground, Impactor, Indicated_entity, Individual, Individuals, Information, Institution, Instrument, Intermediary, Invention, Issue, Item, Light, Mass_theme, Material, Means, Medium, Message, Misdeed, New_idea, New_leader, Noisy_event, Numbers, Object, Old, Old_leader, Old_order, Parameter, Patient, Pattern, Performance, Phenomenon, Precept, Precipitation, Process, Production, Projectile, Property, Proposition, Protagonist, Punishment, Purpose, Pursuer, Resident, Resouce, Resource, Road, Role, Self_mover, Sensory_attribute, Side_1, Sides, Skill, Sleeper, Social_event, Sought_entity, Sound, State_of_affairs, Student, Subject, Task, Text, Theme, Topic, Traveler, Undergoer, Undertaking, Undesirable_event, Undesirable_situation, Unwanted_characteristics, User, Value, Vehicle, Victim, Whole	Theme
Theme1	Arg0, Arg1	Cause, Container, Fastener, Mother, Old, Phenomenon_1, Profiled_item, Theme	Theme
Theme2	Arg1, Arg2, Arg3	Child, Containing_object, Contents, Cotheme, Effect, Location, New, Phenonemon_2, Road, Standard_item	Theme
Time	ArgM_Time	Time	Time
Topic	Arg1, Arg2	Act, Behavior, Communication, Content, Eventuality, Indicated, Information, Issue, Message, Proposition, State_of_affairs, Subject, Text, Theme, Topic, Trigger, Unconfirmed_content	Theme
Asset	Arg1, Arg3	Asset, Category, Measurement, Result, Value	Amount
Source	Arg2, Arg3	Source, Role, Source, Source, Source, Source, Source, Source, Source, Source, Victim, Patient, Source, Source, Victim, Source, Source, Source, Path_start, Path_start, Path_start, Source, Path_start, Source, Addressee, Seller, Source, Source, Source, Teacher, Source, Source, Firearm, Old, Ingestibles, Source, Excreter, Source, Source_emitter, Source, Source, Defendant, Position, Field, Source, Source, Means, Instrument, Evidence, Means, Teacher, Institution, Course, Material	Source, Initial_location
-	-	Setting, ContainingEvent	Setting
-	-	Means	Means
-	ArgM_Manner	Manner	Manner
-	ArgM_Purpose	Purpose	Purpose

*Source

SemLink   http://verbs.colorado.edu/semlink/

Edward Loper, Szu-Ting Yi, and Martha Palmer (2007). Combining Lexical Resources: Mapping between PropBank and VerbNet. In Proceedings of the 7th International Workshop on Computational Linguistics, Tilburg, the Netherlands

1.2 Empirically: tag occurrences in annotated data

Data category	English	Dutch	Italian	Spanish
Total roles identified	1795	1326	454	1356
/agent/	311 (17.3%)	186 (14%)	60 (13.2%)	258 (19%)
/partner/	5(0.3%)	9 (0.7%)	2 (0.4%)	3 (0.2%)
/cause/	39 (2.2%)	33 (2.5%)	2 (0.4%)	43 (3.2%)
/instrument/	10 (0.56%)	7 (0.5%)	7 (1.5%)	4 (0.3%)
/patient/	186 (10.4%)	137 (10.3%)	51 (11.2%)	119 (8.8%)
/pivot/	104 (5.8%)	85 (6.4%)	51 (11.2%)	154 (11.4%)
/theme/	501 (27.9%)	331 (25%)	117 (25.6%)	315 (23.2%)
/beneficiary/	40 (2.02%)	19 (1.4%)	7 (1.5%)	63 (4.7%)
/source/	16 (0.9%)	31 (2.3%)	7 (1.5%)	2 (0.1%)
/goal/	18 (1%)	13 (1%)	13 (2.9%)	5 (0.4%)
/result/	66 (3.7%)	54 (4.1%)	14 (3.1%)	24 (1.8%)
/reason/	36 (2%)	14 (1.1%)	9 (2%)	43 (3.2%)
/purpose/	49 (2.7%)	18 (1.4%)	7 (1.5%)	24 (1.8%)
/time/	135 (7.5%)	106 (8%)	13 (2.9%)	65 (4.8%)
/manner/	39 (2.2%)	27 (2%)	18 (4%)	44 (3.2%)
/medium/	4 (0.2%)	1 (0.1%)	2 (0.4%)	8 (0.6%)
/means/	8 (0.4%)	6 (0.5%)	0	2 (0.1%)
/setting/	47 (2.6%)	48 (3.6%)	16 (3.5%)	28 (2.1%)
/location/	41 (2.3%)	66 (5%)	24 (5.3%)	34 (2.5%)
/initial_location/	2 (0.1%)	1 (0.1%)	2 (0.4%)	5 (0.4%)
/final_location/	6 (0.3%)	10 (0.8%)	7 (1.5%)	43 (3.2%)
/path/	20 (1.1%)	9 (0.7%)	0	0
/distance/	1 (0.06%)	0	1 (0.2%)	0
/amount/	27 (1.5%)	19 (1.4%)	11 (2.4%)	17 (1.3%)
/attribute/	72 (4%)	88 (6.6%)	6 (1.3%)	45 (3.3%)
/frequency/	12 (0.7%)	8 (0.6%)	0	9 (0.7%)

2. Reliability and usability: inter-annotator agreement

Task	Kappa	Disagreement ratio	Cases of confusion
semantic anchors	0.77	0.15	state vs event
semantic roles	0.68	0.25	Agent vs Cause, Attribute vs Manner, Beneficiary vs Goal, Instrument vs Means, Purpose vs Reason, Theme vs Result, Location vs Setting, Theme vs Pivot, Theme vs Patient
Semantic Role	Kappa	Disagreement ratio	Cases of confusion
Agent	0.87	0.1	Theme; Pivot; Patient; Cause
Amount	0.77	0.2	Instrument; Source; Manner
Attribute	0.71	0.29	Theme; Manner; Result; Setting
Beneficiary	0.81	0.19	Patient; Goal; Theme
Cause	0.64	0.36	Agent; Theme; Patient
Final Location	0.98	0.02	Setting
Frequency	0.94	0.06	Amount; Attribute
Goal	0.64	0.36	Beneficiary; Theme; Result
Instrument	0.3	0.72	Patient; Means
Initial Location	0.9	0.1	Setting
Location	0.92	0.08	Setting
Manner	0.89	0.11	Attribute; Setting
Means	0.57	0.43	Patient; Manner; Instrument
Medium	0.76	0.24	Patient; Source; Setting
Partner	0.8	0.19	Patient; Theme
Path	0.76	0.23	Goal; Result
Patient	0.73	0.25	Theme; Result; Instrument; Agent
Pivot	0.65	0.33	Theme; Agent; Patient
Purpose	0.76	0.23	Theme; Reason
Reason	0.81	0.19	Theme; Purpose
Result	0.77	0.22	Theme; Patient; Goal
Setting	0.68	0.32	Manner; Location; Attribute
Source	0.52	0.48	Reason; Setting; Agent
Time	0.99	0.01	Manner; Setting; Theme
Distance	1.00	0	-
Theme	0.67	0.28	Pivot; Result; Patient

Publications

Volha Petukhova and Harry Bunt
`LIRICS semantic role annotation: design and evaluation of a set of data categories.' In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC 2008). Marrakech, May 28-30, 2008.

Volha Petukhova, Harry Bunt and Amanda Schiffrin
`Defining Semantic Roles.' In Proceedings Seventh International Workshop on Computational Semantics (IWCS-7). Tilburg, January 10-12, 2007, pp. 362-365.

Harry Bunt and Amanda Schiffrin
`Methodologial aspects of semantic annotation'. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006). Genova, May 24-26, 2006.

Harry Bunt and Laurent Romary
'Requirements on multimodal semantic representations'. In Proceedings of ISO TC37/SC4 Preliminary Meeting. Seoul, 2002, pp. 59–68.