Reader Guidelines
What is this page? This page shows tables extracted from arXiv papers on the left-hand side. It shows extracted results on the right hand side that match the taxonomy on Papers With Code.
What are the colored boxes on the right hand side? These show results extracted from the paper and linked to tables on the left hand side. A result consists of a metric value, model name, dataset name and task name.
What do the colors mean? Green means the result is approved and shown on the website. Yellow is a result that you have added but have not yet saved. Blue is a referenced result that originates from a different paper.
Where do suggested results come from? We have a machine learning model running in the background that makes suggestions on papers.
Where do referenced results come from? If we find referenced results in a table to other papers, we show a parsed reference box that editors can use to annotate to get these extra results from other papers.
Editor Guidelines
I’m editing for the first time and scared of making mistakes. Help! Don’t worry! If you make mistakes we can revert them: everything is versioned! So just tell us on the Slack channel if you’ve accidentally deleted something (and so on) - it’s not a problem at all, so just go for it!
How do I add a new result from a table? Click on a cell in a table on the left hand side where the result comes from. Then select one of the top-5 proposals. You can manually edit the incorrect or missing fields. Then choose a task, dataset and metric name from the Papers With Code taxonomy. You should check if a benchmark already exists to prevent duplication; if it doesn’t exist you can create a new dataset. E.g. ImageNet on Image Classification already exists with metrics Top 1 Accuracy and Top 5 Accuracy.
What are the model naming conventions? Model name should be straightforward, as presented in the paper. Note that you can use parentheses to highlight details, for example: BERT Large (12 layers), FoveaBox (ResNeXt-101), EfficientNet-B7 (NoisyStudent).
Other tips and tricks
- If a benchmark already exists for a dataset/task pair you enter, you’ll see a link appear.
- If the benchmark doesn’t exist, a “new” icon will appear signifying a new leaderboard.
- If you're feeling lucky, Cmd+Click a cell in a table to get the first result automatically.
- When editing multiple results from the same table you can click the "Change all" button to copy the current value to all other records from that table.
How do I add referenced results? If a table has references, you can use the parse references feature to get more results from other papers. First, you’ll need at least one record in the cell that has results (see image below for an example). Then click the "Parse references" button to link references to papers in PapersWithCode and annotate the results. Below you can see an example.
Comparison table extracted from Universal Language Model Fine-tuning for Text Classification paper
(Howard and Ruder, 2018) with parsed references.
How do I save my edits? When you’re happy with your change click save and your suggested changes will turn green!
