Summary & takeaways
Peer prediction is a branch of mechanism design concerned with eliciting truthful information from individual agents in situations where verifying the truth is difficult or impossible, thus making usage of traditional proper scoring rules impossible. For that, they rely on comparison of agents’ reports with peers (called “references”) and produce a modified scoring rule on this basis.
As a reminder, as usual in elicitation / scoring-rule-based mechanisms, we consider agents who receive signals and make reports to the mechanism. The designer is tasked with producing incentives that elicit truthful reports.
Peer prediction mechanisms introduce agents’ signal posterior, the belief that another agent receives a signal given the current agent’s signal, typically denoted p_i(\cdot|\cdot) for agent i.
All peer prediction mechanisms rely on the following constraints:
- Stochastic relevance: different agent’s signal observations are correlated, ie. p_i(\cdot,s) ≠ p_i(\cdot, s') for s ≠ s'.
- Belief model constraints: each mechanism has a different but necessary belief model constraint, guaranteeing effectiveness of the scoring rule.
For example:
- Output Agreement, p_i(s|s) > p_i(s′|s)
- 1/p Mechanism, p_i(s|s)/y(s) > p_i(s′|s)/y(s′)
- Shadowing Method, p_i(s|s) − y(s) > p_i(s′|s) − y(s′).
Literature
Mechanisms
Eliciting informative feedback: The peer-prediction method.
- Source: https://pubsonline.informs.org/doi/abs/10.1287/mnsc.1050.0379
- Authors: Miller, Nolan, Paul Resnick, and Richard Zeckhauser
- Year: 2005.
- Description: Seminal paper constructing a single-task peer prediction mechanism for product reviews.
A Bayesian truth serum for subjective data
- Source: https://www.science.org/doi/abs/10.1126/science.1102081
- Authors: Prelec, Drazen
- Year: 2004
Learning the prior in minimal peer prediction
- Authors: Witkowski, Jens, and David C. Parkes
- Year: 2013
- Description: Non-minimal mechanism that allows learning the prior.
Mathematical models
A geometric perspective on minimal peer prediction
- Source: A Geometric Perspective on Minimal Peer Prediction | ACM Transactions on Economics and Computation
- Authors: Frongillo, Rafael, and Jens Witkowski. "
- Year: 2017
- Description: Geometric characterization of minimal peer-prediction mechanisms. Produces a method to build a mechanism based on a Scoring Rule and optimize incentives and robustness.
The limits of multi-task peer prediction
- Source: https://dl.acm.org/doi/abs/10.1145/3465456.3467642
- Authors: Zheng, Shuran, Fang-Yi Yu, and Yiling Chen
- Year: 2021
- Description: Characterization of the elicitable multi-task peer prediction problems.