Roles: everyone, author, PC members, PC chair, PC chair assistant. PC members have primary and secondary reviews.
Everyone: just account management stuff.
Authors: lets you start a new paper. Each paper only has one primary author who handles the paper; others are not involved.
PC members: can delegate reviews, bug their reviewers, review their papers. When all reviews are in, they can help "grade" papers and examine all papers.
PC chair: run stuff.
PC Chair Assistants: do admin tasks to help out the chair. (Printing out papers, etc.)
Review types: There are both "primary" and "secondary" reviews. Supposedly primary papers are not to be delegated, but secondary are allowed to be delegated. (The UI seems to allow both to be delegated, though.) You can also have external reviewers for additional reviewers by non-PC members.
For sensys I added suggested external reviewers: PC members suggets reviewers for secondary papers; the chair approves or denies them; then the chair actually does the secondary invites. This whole thing is beneficial for double blind review.
Grading: After reviews are back, CRP has PC members "grade" papers based on the reviews. Basically they indicate if they they think the paper should be in or out, and if they're willing to champion or blackball it.
I found it necessary to increase memory_limit to 64M (from default 8M) in /etc/php.ini, otherwise downloading some papers failed. (Possibly uploading as well.)
Also one must change post_max_size to match upload_max_filesize in /etc/php.ini.
Very first, set up conference dates. Things don't become visible until within the date range, so start things early.
Fill out the topics database. The topics can be used to associate papers with PC members.
Set up the reviewing form by creating and editing Code/ClassCONFERENCENAMEReview.inc. (Typically, create by copying Code/ClassHYPO2010Review.inc.)
Get all the PC members to create accounts and specify topics. preferences.
Verify registrations are complete (include title and abstract). Complain about ones that are incomplete (see script johnh_contrib/find_incomplete_paper_registration). Delete all incomplete registrations.
Deal with exceptions and any out-of-band submitted papers.
Verify all papers present. Ping any missing. Withdraw any that are not submitted? Ping any that are unfinalized. Helpful programs:
find_incomplete_paper_registration,
find_unfinalized_submissions,
Walk through each paper. Mark any PC papers as such. Figure out and assign conflicts (first do so automatically by look for conflicts).
Before reviews are assigned, we let PC members pick what papers they most want. Ideally this results in better reviews since, well, they asked for it :-)
Then invite all PC members to look at the papers and express their preferences. Default preferences are set to a combination of the paper topics and PC member interests, so inattentive PC members should still be OK.
(The PC chair should manually become each PC member and visit the express preferences page to insure the defaults are set.)
Before running
johnh_contrib/assign_pc_reviewers, edit
johnh_contrib/crp.confto match your database name and password.
If you did preferences, then an out-of-band program
johnh_contrib/assign_pc_reviewersdoes this task automatically based preferences. (The algorithm is simple bidding: each PC member takes turns getting their favorite remaining paper until all papers are covered.) (Bug: this code should be ported to PHP to eliminate the dependency on perl.)
You can then edit the assignments: Assign papers to PC members.
Inform PC members they have their assignments through out-of-band e-mail.
For sensys we treated secondary reviews as external reviews only. Because the conference was double-blind, we had PC members suggest reviewers for their secondary papers, then the PC chairs did the actual inviting.
To do this, each PC member suggests external reviewers. The Chair and add to these suggestions for any paper. The Chair then approves or rejects suggestsions and can invite the reviewers with the same interface.
(Alternatively the chair could kick the approved suggestions back to the PC members, but we didnt' do that.)
Check for high or low skew of reviewers.
Look for papers with divergent opinions (use AveragePaperScore with std dev of overall merit). Prompt discussion or get additional viewpoints for these papers.
Look for papers that lack expert reviewers. (use AveragePaperScore with reviewer expertise). Get additional viewpoints for these papers.
Open up the database to the TPC.
Do first round of pruning (cull any clear rejects).
Ask PC members to grade papers.
Select "leads" for each paper to present that paper at the conference.
Prepare custom lists of papers for each TPC member indicating when they're in conflict. Actually, do a generic list, then a custom list of PC papers.
Things to bring to TCP meeting: printer (skipped), single hardcopy of all papers, copy of custom review forms for each reviewer. Stapler, tape, projector (maybe).
Things hotel provides at TPC: Food, projector (maybe).
Walk through papers, managing conflicts.
Have lead reviewer capture any comments in the discussion and add to the review.
PC Chair records outcomes (use the chair grade all papers with "show outcome" checked).
Post-PC meeting dinner.
Open results of papers to TPC.
Inform authors by mail.
Divide papers into sessions. Select session chairs.
Confirm each paper has a registered author. (Solicit presentation authors?)