These recommendations are based on the “Handreichung für den Begutachtungsprozess der DHd-Jahrestagungen” prepared by Manuel Burghardt, Lisa Dieckmann, Svenja Guhr, Nils Reiter, Walter Scholger, Timo Steyer, Peer Trilcke, Ulrike Wuttke on behalf of DHd (Digital Humanities in the German-speaking area), and were expanded and adjusted by Anne Baillot, Walter Scholger and Toma Tasovac. [Download PDF]
1. General recommendations for reviewers
- Before accepting the review, please consider potential conflicts of interest or biases that may hinder your partiality.
- Read the expanded review criteria (see below) in full and use them as the basis for your evaluation. Please make the relationship between the score (number of points you are giving) and the criteria this score is based on as explicit as possible.
- Please provide constructive feedback that clearly points out existing weaknesses to the authors and offers concrete suggestions for improvement.
- Refrain from belittling or derogatory formulations in your review. The review should critically evaluate the submission, but always be constructive in tone.
- Please include full bibliographic references or DOIs/URLs to further information that you feel would improve the quality of the submission.
- If you have objections to the writing style, cite specific examples/the passages in question.
- Even if you personally disagree with basic premises of the submission, please critically question whether rejection of the same can be justified with objective arguments. Likewise, please do not reject a submission simply because you are not personally interested in the relevant subject area, but rather make use of the option to decline review in such cases.
- Please provide appropriate feedback on contributions for which you have no critical comments. In such cases, emphasize the particular strengths of the contribution.
- The focus of the review is the contribution. Especially in the case of early career scholars, you may consider their disciplinary and academic background. In general, the bibliography attached to the contribution can also be taken into account in order to assess the level of innovation of the proposal.
- Personal and/or political opinions expressed by authors (e.g., on social media) are not part of the review process and should not be considered.
- Reviewers should reflect on their own potential unconscious biases and, as much as possible, exclude them from the review process.
- The review process is understood as a dialogue between authors, reviewers and the program committee. Please be aware that you may be asked to clarify statements in your review or to add additional information.
2. Bias criteria for declining review
Please check the following list of bias criteria thoroughly before accepting any review as well as when conducting a review. Criteria A to C should lead to declining a review. In the case of criteria D to F, you as the reviewer should make a decision on a case-by-case basis.
This list does not cover all biases scenarios. Each reviewer is encouraged to thoroughly and conscientiously reflect upon their own biases regarding a submission and, when in doubt, decline to review.
Exclusion criteria
A. Close relationships: first-degree relationships such as close family, marriage, civil partnership, romantic partners, etc.
B. Existing or planned collaboration: current or planned close scientific collaboration, e.g. joint publications, joint project implementation.
C. Existing, imminent, or recent professional hierarchical relationship: official subordination or a supervisory relationship up to three years after termination of the relationship, as well as imminent or planned official dependence or supervisory relationship (from teacher-student relationship up to and including the postdoc phase).
Case-by-case decision
D. Past close cooperation: significant scientific cooperation within the last 1.5 years, e.g. joint publications, joint project realization.
E. Competition: preparation of a proposal or implementation of a project on a closely related research topic.
F. Distant kinship or other close personal relationship: kinship relationships that do not fall under A), other personal ties or conflicts.
3. CfP DH2023 – Evaluation of Submissions
Evaluation Criteria
The following evaluation criteria, based on the Call for Papers will be applied when reviewing for DH2023:
- Innovation (25%): Is the submission an innovative contribution to DH subject area(s)?
- State-of-the-art (20%): Is the state of the art adequately presented (including a bibliography)?
- Methodology (20%): Is the research methodology comprehensively described and adequately reflected upon?
For reviewing of Workshop proposals: Are the didactic methodology and procedure of the workshop described understandably and realistically? - Comprehensiveness (5%): Is the submission formulated in a comprehensible manner?
- Formalities (5%): Does the submission meet all formal criteria (abstract length, citation style, formatting etc.)?
- Recommendation (25%): Reviewers’ general recommendation for acceptance
Rating scale (Points)
10 – Excellent work and a meaningful contribution
08 – Good and meaningful work
06 – Solid content
04 – Evident weaknesses in content
02 – Only a minor contribution
00 – Questionable work
Example: In addition to the textual review, a reviewer assigns the following points: Innovation: 6, State-of-the-art: 6, Methodology: 8, Comprehensiveness: 8, Formalities: 10, Recommendation: 8. The total review score of the submission is therefore 72%. As was the case in previous DH conferences, the final review scores are given as a percentage, with the optimal review score being 100%.
4. Elaboration on the evaluation criteria
As a general rule, a good submission describes the research problem that is addressed, refers to and describes the current state of research, explains the method(s) that are used, and clearly communicates the results of the conducted research.
1) The submission is an innovative contribution to the subject area of the DH (25% of total score).
- The contribution provides added scientific value to a subject area within the Digital Humanities and thus contributes to the overall state of knowledge.
- The contribution has a noticeable degree of novelty for the DH community and makes this explicit. It should not have been previously published in this form (including in another
language). Repetitive or replication studies may be accepted, but may require clarifying why replication was important/relevant in this case. - Research results or methods from other disciplines outside of the DH fields may be new to the DH community, despite having been published before: in this case, the contribution
should clearly identify the “discipline of origin” and the novelty aspect for DH.
2) The state of the art of research is adequately presented (20% of total score).
- The paper presents the current state of research and thus demonstrates why and how the paper advances the overall state of knowledge (and represents “an innovative contribution
to the DH subject area” as stated in item 1). - The state of research should be presented in a structured manner so that explicit research gaps and desiderata can be identified in the context of the submission.
- The contribution must include a bibliography that fulfills scholarly quality criteria.
3) The research methodology is comprehensively described and appropriately reflected, or, in case of a workshop, the didactic methodology and schedule of the workshop are comprehensively and realistically described (20% of total score).
- A comprehensively described research methodology is part of a comprehensively formulated contribution (covered by criterion 4, below). Here, you should focus on the meaningful, appropriate, and consistent presentation of the application of the chosen methodology.
- Methodological errors or inadequacies (if applicable) also fall under this criterion.
- Adequate reflection on the research methodology requires that, at a minimum, the reasons for the specific choice of methodology are explicitly stated. Depending on the contribution
format, the reflection may be selective (such as in the case of posters or panels) or more detailed. - In the case of software, development methodology may replace research methodology where appropriate.
- In the case of workshops, the contribution should include a description of the didactic goals and methods as well as a feasible schedule.
4) The submission is formulated in an understandable manner (5% of total score).
- When evaluating comprehensibility, consider the interdisciplinary nature of the community: discipline-specific technical terms or concepts should be explained in a generally
understandable way for the broader DH community and thus be comprehensible. - The contribution should be comprehensible on its own without having to consult additional sources.
5) The submission meets all formal criteria (5% of total score).
- Are the formal requirements defined in the Call for Papers fulfilled in an appropriate way, i.e.: length of submission, adherence to format requirements (figures, tables, etc.), citation
style, and bibliography? - All sources cited are correctly referenced and traceable.
6) General recommendation for acceptance (25% of total score).
- Should the contribution be accepted as a whole?
- Are there arguments for or against acceptance that do not fall into the other categories?
- Does the contribution enrich the meeting and DH research in general?
We hope that these recommendations and the underlying arguments we elaborated on here will support you during the review process. Our sincerest thanks for your willingness to serve as a reviewer for DH2023, we really appreciate your service to our community!