Designing Intelligent Review Forms For Peer Assessment: A Data-Driven Approach

This research paper employs a data-driven, explainable, and scalable approach to the development and application of an online peer review form in computer science and engineering courses. Crowd-sourced grading through peer review is an effective evaluation methodology that 1) allows the use of meaningful assignments in large or online classes (e.g. assignments other than true/false, multiple choice, or short answer), 2) fosters learning and critical thinking in a student evaluating another’s work, and 3) provides a defendable and non-biased score through the wisdom of the crowd. Although peer review is widely utilized, to the authors’ best knowledge, the form itself and associated grading process have never been subjected to data-driven analysis and design. We present a novel, iterative algorithm by first gathering the most appropriate review form questions through intelligent data mining of past student reviews. During this process, key words and ideas are gathered for the positive and negative sentiment, flag word, and negate word dictionaries. Next, we revise our grading algorithm using simulations and perturbation to determine robustness (measured by standard deviation within a section). Using the dictionaries, we leverage sentiment gathered from reviewer detailed comments as a quality assurance mechanism to generate a crowd comment “grade”. This grade supplements the weighted average of the other review form sections. This result of this innovative process is a peer assessment package (robust algorithm leveraging crowd sentiment) based on actual student work that provides a semi-automated grader that can be used by a professor to confidently assign and grade any assignment in any size class in any environment.

Zachariah Beasley, Les A Piegl, and Paul Rosen. Designing Intelligent Review Forms For Peer Assessment: A Data-Driven Approach. ASEE Annual Conference & Exposition, 2019.

@inproceedings{beasley2019designing,
  title = {Designing Intelligent Review Forms for Peer Assessment: A Data-driven
    Approach},
  author = {Beasley, Zachariah and Piegl, Les A and Rosen, Paul},
  booktitle = {ASEE Annual Conference & Exposition},
  year = {2019},
  abstract = {This research paper employs a data-driven, explainable, and scalable
    approach to the development and application of an online peer review form in computer
    science and engineering courses. Crowd-sourced grading through peer review is an
    effective evaluation methodology that 1) allows the use of meaningful assignments in
    large or online classes (e.g. assignments other than true/false, multiple choice, or
    short answer), 2) fosters learning and critical thinking in a student evaluating
    another’s work, and 3) provides a defendable and non-biased score through the wisdom of
    the crowd. Although peer review is widely utilized, to the authors’ best knowledge, the
    form itself and associated grading process have never been subjected to data-driven
    analysis and design. We present a novel, iterative algorithm by first gathering the most
    appropriate review form questions through intelligent data mining of past student
    reviews. During this process, key words and ideas are gathered for the positive and
    negative sentiment, flag word, and negate word dictionaries. Next, we revise our grading
    algorithm using simulations and perturbation to determine robustness (measured by
    standard deviation within a section). Using the dictionaries, we leverage sentiment
    gathered from reviewer detailed comments as a quality assurance mechanism to generate a
    crowd comment “grade”. This grade supplements the weighted average of the other review
    form sections. This result of this innovative process is a peer assessment package
    (robust algorithm leveraging crowd sentiment) based on actual student work that provides
    a semi-automated grader that can be used by a professor to confidently assign and grade
    any assignment in any size class in any environment.}
}