Search Results for author:
Found 41 papers, 14 papers with code
On the Detection of Reviewer-Author Collusion Rings From Paper Bidding
A major threat to the peer-review systems of computer science conferences is the existence of "collusion rings" between reviewers.
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
We find that across 119 {checklist question, paper} pairs, the LLM had an 86. 6% accuracy.
Counterfactual Evaluation of Peer-Review Assignment Policies
We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e. g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the "cost of randomization", capturing the difference in expected quality between the perturbed and unperturbed optimal match.
A Gold Standard Dataset for the Reviewer Assignment Problem
We address this challenge by collecting a novel dataset of similarity scores that we release to the research community.
Assisting Human Decisions in Document Matching
Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models.
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
In a top-tier computer science conference (NeurIPS 2021) with more than 23, 000 submitting authors and 9, 000 submitted papers, we survey the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews.
Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?
Specifically, in the often-used holistic allocation, each evaluator is assigned a subset of the applicants, and is asked to assess all relevant information for their assigned applicants.
Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment
Many conferences rely on paper bidding as a key component of their reviewer assignment procedure.
A Dataset on Malicious Paper Bidding in Peer Review
The performance of these detection algorithms can be taken as a baseline for future research on detecting malicious bidding.
Integrating Rankings into Quantized Scores in Peer Review
Our approach addresses the two aforementioned challenges by: (i) ensuring that rankings are incorporated into the updates scores in the same manner for all papers, thereby mitigating arbitrariness, and (ii) allowing to seamlessly use existing interfaces and workflows designed for scores.
Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise
Finally, we evaluate the methods on a dataset from conference peer review.
Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design
Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted.
Debiasing Evaluations That are Biased by Evaluations
For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews.
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others.
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate.
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisions.
Uncovering Latent Biases in Text: Method and Application to Peer Review
In this work, we introduce a novel framework to quantify bias in text caused by the visibility of subgroup membership indicators.
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions.
Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments
We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case.
On the Privacy-Utility Tradeoff in Peer-Review Data Analysis
The crux of the framework lies in recognizing that a part of the data pertaining to the reviews is already available in public, and we use this information to post-process the data released by any privacy mechanism in a manner that improves the accuracy (utility) of the data while retaining the privacy guarantees.
A SUPER* Algorithm to Optimize Paper Bidding in Peer Review
Theoretically, we show a local optimality guarantee of our algorithm and prove that popular baselines are considerably suboptimal.
Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions
We also provide testing algorithms and associated sample complexity bounds for the problem of two-sample testing with partial (or total) ranking data. Furthermore, we empirically evaluate our results via extensive simulations as well as two real-world datasets consisting of pairwise comparisons.
Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons
We show that the MLE incurs a suboptimal rate in terms of bias.
Loss Functions, Axioms, and Peer Review
The key challenge that arises is the specification of a loss function for ERM.
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the commonly used objective of maximizing the total quality over all papers.
On Strategyproof Conference Peer Review
We then empirically show that the requisite property on the authorship graph is indeed satisfied in the submission data from the ICLR conference, and further demonstrate a simple trick to make the partitioning method more practically appealing for conference peer review.
Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings
A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores.
Low Permutation-rank Matrices: Structural Properties and Noisy Completion
We consider the problem of noisy matrix completion, in which the goal is to reconstruct a structured matrix whose entries are partially observed in noise.
Design and Analysis of the NIPS 2016 Review Process
Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning.
A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
The task of aggregating and denoising crowd-labeled data has gained increased significance with the advent of crowdsourcing platforms and massive datasets.
Active Ranking from Pairwise Comparisons and when Parametric Assumptions Don't Help
We first analyze a sequential ranking algorithm that counts the number of comparisons won, and uses these counts to decide whether to stop, or to compare another pair of items, chosen based on confidence intervals specified by the data collected up to that point.
Feeling the Bern: Adaptive Estimators for Bernoulli Probabilities of Pairwise Comparisons
Second, we show that a regularized least squares estimator can achieve a poly-logarithmic adaptivity index, thereby demonstrating a $\sqrt{n}$-gap between optimal and computationally achievable adaptivity.
Parametric Prediction from Parametric Agents
In order to elicit heterogeneous agents' private information and incentivize agents with different capabilities to act in the principal's best interest, we design an optimal joint incentive mechanism and prediction algorithm called COPE (COst and Prediction Elicitation), the analysis of which offers several valuable engineering insights.
Simple, Robust and Optimal Ranking from Pairwise Comparisons
We consider data in the form of pairwise comparisons of n items, with the goal of precisely identifying the top k items for some value of k < n, or alternatively, recovering a ranking of all the items.
Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues
On the other hand, unlike in the BTL and Thurstone models, computing the minimax-optimal estimator in the stochastically transitive model is non-trivial, and we explore various computationally tractable alternatives.
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
Data in the form of pairwise comparisons arises in many domains, including preference elicitation, sporting competitions, and peer grading among others.
Regularized Minimax Conditional Entropy for Crowdsourcing
There is a rapidly increasing interest in crowdsourcing for data labeling.
Approval Voting and Incentives in Crowdsourcing
The growing need for labeled training data has made crowdsourcing an important part of machine learning.
On the Impossibility of Convex Inference in Human Computation
Human computation or crowdsourcing involves joint inference of the ground-truth-answers and the worker-abilities by optimizing an objective function, for instance, by maximizing the data likelihood based on an assumed underlying model.
Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
To address this fundamental challenge in crowdsourcing, we propose a simple payment mechanism to incentivize workers to answer only the questions that they are sure of and skip the rest.
When is it Better to Compare than to Score?
When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements.
