The share of the reproduction paper is based on analyses of as well as in insights into current methods and problems—plus the certainty that is added is sold with validating past outcomes.
- Relevance: Is it paper highly relevant to COLING?
- Readability/clarity: could be the paper well-written and well-structured?
- Data/code access: could be the data/code (as appropriate) open to the investigation community or perhaps is here a compelling explanation offered why it is not feasible?
- Analysis: In the event that paper surely could reproduce the outcomes of the earlier in the day work, does it demonstrably formulate exactly what must be filled in to do therefore? If it wasn’t in a position to reproduce the outcomes of early in the day work, does it demonstrably recognize just what information ended up being missing/the likely factors?
- Generalizability: Does the paper rise above replicating the outcomes regarding the initial to explore if they may be reproduced an additional environment? Instead, in instances of non-replicability, does the paper talk about the wider implications of this outcome?
- Informativeness: To exactly exactly what degree does the analysis reported within the paper deepen our knowledge of the methodology utilized or perhaps the nagging issue approached? Will the given information into the paper assistance professionals along with their selection of technique/resource?
- Significant contrast: as well as determining the experimental results being replicated, does the paper motivate why these specific email address details are a target that is important reproduction and exactly what the long term implications are of these having been reproduced or been discovered to be non-reproducible?
- Overall suggestion: there are lots of good submissions contending for slots at COLING 2018; essential can it be to feature this 1? Will people learn a complete lot by scanning this paper or seeing it presented? Please be decisive—it is much better to vary from other reviewers rather than grade every thing in the centre.
Documents in this track provide a brand new language resource. This might be a corpus, but additionally could possibly be an annotation standard, device, an such like.
- Relevance: Is it paper highly relevant to COLING? Will the resource presented likely be of good use to our community?
- Readability/clarity: Through the means the paper is written, are you able to inform how a resource had been produced, the way the quality of annotations (if any) ended up being examined, and just why the resource must certanly be of great interest?
- Originality: Does the resource fill a need into the current assortment of available resources? Keep in mind that originality could possibly be when you look at the range of language/language variety or genre, within the design associated with the annotation scheme, when you look at the scale for the resource, or nevertheless other parameters.
- Resource quality: what sort of quality control had been performed? If appropriate, ended up being inter-annotator contract calculated, and when therefore, with appropriate metrics? Otherwise, the other assessment ended up being conducted, and exactly how acceptable were the outcomes?
- Resource accessibility: could it be simple for researchers to down load or otherwise access the resource to be able to put it to use in their own personal work? As to what degree could work predicated on this resource be shared? answers to add: Yes, We have verified
- Metadata: perform some writers explain whoever language use is captured when you look at the resource also to exactly what populations experimental outcomes based from the resource might be generalized to? In instance of annotated resources, will be the demographics associated with the annotators also characterized?
- Significant comparison: could be the brand new resource situated with regards to current work with the industry, including comparable resources it took motivation from or improves on? Can it be clear what exactly is unique about the resource?
- General suggestion: there are lots of submissions that are good for slots at COLING 2018; essential can it be to feature this 1? Will people discover a complete great deal by looking over this paper or seeing it presented? Please be decisive—it is way better to vary from other reviewers than to grade everything at the center.
A situation paper presents a challenge to mainstream reasoning or a futuristic brand new eyesight. It might start a brand new area or unique technology, propose changes in existing research, or offer a unique collection of ground guidelines.
- Relevance: Is this paper highly relevant to COLING?
- Readability/clarity: Is it clear exactly exactly what the career is the fact that paper is arguing for? Will be the arguments because of it laid call at a way that is understandable?
- Soundness: Are the arguments presented within the paper coherent and relevant? Could be the eyesight well-defined, with success requirements? (Note: it must be feasible to offer a score that is high even though you don’t buy into the place taken by the writers)
- Imagination: How bold or novel may be the place drawn in the paper? Does it express well-thought through and imaginative ground that is new?
- Range: How scope that is much brand brand new research is exposed by this paper? Exactly What effect could it have on current areas and concerns?
- Significant contrast: could be the paper well-situated with regards to past work, both place documents (taking exactly the same or opposing side for a passing fancy or comparable problems) and appropriate theoretical or work that is experimental?
- Substance: Does the paper have sufficient substance for the paper that is full-length? Could be the problem sufficiently crucial? Would be the arguments adequately varied and thoughtful?
- General suggestion: there are numerous submissions that are good for slots at COLING 2018; essential can it be to feature that one? Please be decisive—it is way better to vary from other reviewers than to grade every thing at the center.
- Relevance: could be the paper highly relevant to COLING?
- Readability/clarity: may be the paper generally speaking an easy task to follow and well structured?
- Organization: Does the paper organize the literature that is relevant a narrative and recognize typical strands of inquiry?
- Scope: Does the paper recognize a fairly concentrated area to review?
- Thoroughness: because of the location identified to survey, does the paper address most of the literature that is relevant? May be the literary works reviewed represented accurately?
- Outlook: Does the paper determine areas for future work and/or ninjaessays obviously mention just what is certainly not yet managed in the literary works surveyed?
- Context: Does the paper situate research that is current within its historic context? (We don’t expect papers in the first place Pa?ini, yet during the same time something that only cites work from 2017 most likely does not capture just exactly how current work pertains to the larger image.)
A study paper provides an organized summary of the literary works up to now for a certain subject that assists the reader comprehend the kinds of concerns being expected about the subject, the many approaches which have been used, the way they relate with one another, and just exactly exactly what further research areas they open. A conference-length survey paper should be about a adequately concentrated subject that it could try this effectively with within the web web page limits.