What is the purpose of Peer-Review in DLESE?
What are the criteria for review?
What is an educational resource?
How will portal sites be reviewed?
How will resources be chosen for review?
Who decides what will be removed from the collection?
Can the review process be streamlined for resources that have undergone another type of quality review?
Who will conduct the reviews?
How can I become a reviewer?
Below we answer a number of questions commonly asked about the DLESE Peer-Review process. If you have a question that is not answered below, please contact us
The purposes of the peer review process are to a) ensure DLESE resources have scientifically accurate and current content; b) advance the quality of teaching and learning about the Earth system by providing feedback to developers of sites so they can improve their resources; c) provide documentation of the high quality of resources; and d) provide recognition to authors of exemplary resources through a publication reference number or Digital Object Identifier (DOI).
Support for peer review of resources in DLESE has existed since its inception. The DLESE Community Plan (Manduca and Mogk, 2000) calls for the library to contain high quality resources and recommends peer review to ensure that creators of high-quality resources achieve academic recognition equivalent to a published paper for their efforts. This position was overwhelmingly supported by results of a survey on academic recognition in which almost 90% of the respondents endorsed peer review to secure the reputation of DLESE as a high-quality library and the individual resources as worthy of academic recognition (Butler et al., 2000). These same respondents indicated that they would be willing to perform reviews. NSF (Colwell, 1999; Leinen, 2000) has maintained from the inception that peer review is essential to ensure that high-quality resources are accessible to the public and educators at all levels.
"Any educator-indeed, any citizen-can enter the library, easily find what they need, and be assured that it has been reviewed."
Margaret Leinen, Associate Director NSF Geosciences, Spring 2000 AGU Meeting
The Digital Library for Earth System Education: a National Science Foundation perspective
"Now being developed, the digital library will provide instructional materials, connect teachers with others who teach similar courses, and provide students with real-time data and information. The teaching resources will be peer reviewed. We're excited that Earth science can offer some of the most dynamic, real-world, visual data at all levels of learning."
Rita Colwell, Director NSF, Fall 1999 AGU Meeting
Complexity and Connectivity: A New Cartography for Science and Engineering Keynote Speech
This peer review process will assign a Digital Object Identifier (DOI), the equivalent of a reference for printed journal articles, to all resources successfully completing the peer review process. This will allow contributors to officially publish their work on the web. In addition, relevant reviewer comments will be linked to each reviewed resource as part of an annotation collection, further enriching the resource.
The review process focuses on the accuracy and currency of the scientific content, overall quality of the resource (layout, completeness, documentation, relevance to the Earth system, uniqueness, etc.), and pedagogical development. Community groups of scientists and educators providing input on the development of the library have indicated that scientific accuracy and currency are the most critical elements of a resource. Any resource that does not meet these criteria will require revisions to remain in the library. However, high quality science content is also not sufficient to warrant publication in the library; a resource must also have qualities that lend itself to effective teaching. A resource given a rating of 1 will be removed from the catalog, a 2 or 3 should be revised, and a 4 will be highlighted as an exemplary resource. Learn more...
We distinguish different types of education resources in the library as reference, lab activity, audio, visual etc. Some resources are specifically designed to provide a framework for instruction. We call these instructional resources . Reviewers will be asked to take into account the type of resource when forming their responses. For example, a laboratory activity should include some instructional guidance within the activity or as a supplement. On the other hand, a well-written documentary on a field expedition or a research article may be included as a stand-alone resource that can be incorporated into a lesson plan. Reviewer comments can be provided with all resources, potentially providing ideas for how to effectively use each resource for teaching.
Judgment by the Managing Editor and Associate Editor(s) will be used when addressing resources that contain science content but are not specifically intended for use in the classroom (e.g., science news items). These types of resources may require only content and metadata reviews, but no pedagogical review. Additionally, a separate but complementary approach for review will be invoked for data sets. The criteria for data set review are still under development by the Data Access Working Group.
Portal sites, such as the US Geological Survey site or Windows to the Universe present a challenge for review due to their size and breadth of content. At this stage of the review process development, we will forego review of these sites until they can be broken down into more manageable elements. With experience in reviewing smaller resources and thematic collections, we expect to develop an effective strategy for review of portals.
Key criteria for early review will be the ability to contact the owner/author of the resource and the owner/authorÕs interest and willingness to participate in the review. Another criteria for prioritizing reviews are requests from the community to review a resource because of its high or poor quality. The review load can be managed by prioritizing reviews based on importance of a resource or collection, perceived quality, ability to potentially streamline the review process due to previous reviews, and ability to assemble reviewers to conduct reviews of a resource or an entire collection. The Managing Editor and Associate Editors will refine priorities based on information from the contributors as well as reviewers.
The Associate Editors will make their recommendation for retention or removal from the library to the Managing Editor, based on well-documented information from the peer reviews. These recommendations will be accepted in most instances; however, the Managing Editor has the right to request further review in instances where the recommendations of the peer reviewers are not conclusive.
The Managing Editor has the authority to decide whether a streamlined review can be conducted on a resource or collection based on the strength of the previous review process and associated documentation. An example case might be a collection of resources that have been thoroughly field-tested with appropriate educational audiences by the owner/creator or collection builder, but the scientific review process was not as rigorous. These resources could be accepted after performing a scientific content review, while additional pedagogical review might be waived. The Managing Editor and relevant Associate Editor(s) would consider each case individually.
Given the broad range of science, pedagogy, and user levels in the library, there will need to be discipline- or grade-level-specific reviewers to perform content and pedagogical reviews (e.g., K-4, Middle School, High School/Intro Undergrad, Grad/Professional, and Data). The Associate Editors will identify a team of 2-3 reviewers with appropriate pedagogical and scientific credentials to perform a review. Reviewers will be asked to review multiple resources so that they can gain some perspective on the process as it applies to different resources. This has been found to improve the quality of reviews overall. Great effort will be made to ensure the Associate Editors and reviewers are only assigned a finite number of resources to review and that those resources are closely aligned with the individualÕs expertise. While a structured approach, this process seeks to entrain as many community members in the review process as possible to lighten the load on each individual and to embrace the broad expertise within our community.
We welcome all community members interested in being a reviewer to contact the Managing Editor. We will ask for some background information on your education and scientific expertise so that we can properly assign resources within your expertise and interests.
DLESE Visiting Committee Report
DLESE Peer-Review Opportunities
DLESE Quality Review
DLESE Peer-Review Criteria