Guide to CRP

My notes based on Sensys 2006, John Heidemann

Reviewing Model

Roles: everyone, author, PC members, PC chair, PC chair assistant. PC members have primary and secondary reviews.

Everyone: just account management stuff.

Authors: lets you start a new paper. Each paper only has one primary author who handles the paper; others are not involved.

PC members: can delegate reviews, bug their reviewers, review their papers. When all reviews are in, they can help "grade" papers and examine all papers.

PC chair: run stuff.

PC Chair Assistants: do admin tasks to help out the chair. (Printing out papers, etc.)

Review types: There are both "primary" and "secondary" reviews. Supposedly primary papers are not to be delegated, but secondary are allowed to be delegated. (The UI seems to allow both to be delegated, though.) You can also have external reviewers for additional reviewers by non-PC members.

For sensys I added suggested external reviewers: PC members suggets reviewers for secondary papers; the chair approves or denies them; then the chair actually does the secondary invites. This whole thing is beneficial for double blind review.

Grading: After reviews are back, CRP has PC members "grade" papers based on the reviews. Basically they indicate if they they think the paper should be in or out, and if they're willing to champion or blackball it.

When Setting Things Up

I found it necessary to increase memory_limit to 64M (from default 8M) in /etc/php.ini, otherwise downloading some papers failed. (Possibly uploading as well.)

Also one must change post_max_size to match upload_max_filesize in /etc/php.ini.

Before Anyone Else

Very first, set up conference dates. Things don't become visible until within the date range, so start things early.

Fill out the topics database. The topics can be used to associate papers with PC members.

Set up the reviewing form by creating and editing Code/ClassCONFERENCENAMEReview.inc. (Typically, create by copying Code/ClassHYPO2010Review.inc.)

Before Papers Come In

Get all the PC members to create accounts and specify topics. preferences.

Once All Papers Are Registered

Verify registrations are complete (include title and abstract). Complain about ones that are incomplete (see script johnh_contrib/find_incomplete_paper_registration). Delete all incomplete registrations.

Once All Papers Are In

Deal with exceptions and any out-of-band submitted papers.

Verify all papers present. Ping any missing. Withdraw any that are not submitted? Ping any that are unfinalized. Helpful programs:

find_incomplete_paper_registration
,
find_unfinalized_submissions
,

Conflicts

Walk through each paper. Mark any PC papers as such. Figure out and assign conflicts (first do so automatically by look for conflicts).

PC Interest in Papers

Before reviews are assigned, we let PC members pick what papers they most want. Ideally this results in better reviews since, well, they asked for it :-)

Then invite all PC members to look at the papers and express their preferences. Default preferences are set to a combination of the paper topics and PC member interests, so inattentive PC members should still be OK.

(The PC chair should manually become each PC member and visit the express preferences page to insure the defaults are set.)

Assigning Papers

Before running

johnh_contrib/assign_pc_reviewers
, edit
johnh_contrib/crp.conf
to match your database name and password.

If you did preferences, then an out-of-band program

johnh_contrib/assign_pc_reviewers
does this task automatically based preferences. (The algorithm is simple bidding: each PC member takes turns getting their favorite remaining paper until all papers are covered.) (Bug: this code should be ported to PHP to eliminate the dependency on perl.)

You can then edit the assignments: Assign papers to PC members.

Inform PC members they have their assignments through out-of-band e-mail.

External Reviews

For sensys we treated secondary reviews as external reviews only. Because the conference was double-blind, we had PC members suggest reviewers for their secondary papers, then the PC chairs did the actual inviting.

To do this, each PC member suggests external reviewers. The Chair and add to these suggestions for any paper. The Chair then approves or rejects suggestsions and can invite the reviewers with the same interface.

(Alternatively the chair could kick the approved suggestions back to the PC members, but we didnt' do that.)

Once All Reviews Are In

Check for high or low skew of reviewers.

Look for papers with divergent opinions (use AveragePaperScore with std dev of overall merit). Prompt discussion or get additional viewpoints for these papers.

Look for papers that lack expert reviewers. (use AveragePaperScore with reviewer expertise). Get additional viewpoints for these papers.

Open up the database to the TPC.

Do first round of pruning (cull any clear rejects).

Ask PC members to grade papers.

Week before PC Meeting

Select "leads" for each paper to present that paper at the conference.

Set paper dicsussion order.

Prepare custom lists of papers for each TPC member indicating when they're in conflict. Actually, do a generic list, then a custom list of PC papers.

Just Before the TCP Meeting

Things to bring to TCP meeting: printer (skipped), single hardcopy of all papers, copy of custom review forms for each reviewer. Stapler, tape, projector (maybe).

Things hotel provides at TPC: Food, projector (maybe).

At the TPC Meeting

Walk through papers, managing conflicts.

Have lead reviewer capture any comments in the discussion and add to the review.

PC Chair records outcomes (use the chair grade all papers with "show outcome" checked).

Post-PC meeting dinner.

After TPC Meeting

Open results of papers to TPC.

Inform authors by mail.

Well Before Conference

Divide papers into sessions. Select session chairs.

Just Before Conference

Confirm each paper has a registered author. (Solicit presentation authors?)

CRP Bugs