Remote procedure call is abstractly like a regular procedure call. In implementation, they are very much like system calls.
A system call is a request to the kernel to do some work for you. Since the kernel lives in a different address space, you can't do this with a procedure call. You need to make a request message, which tells the kernel what work you want done, for instance open a file, and the arguments, in this case the name of the file. You send this request to the kernel with the syscall instruction.
This is very similar to a remote procedure call. In an RPC you need to create a package which can be sent to another machine (or even the same machine) and contains enough information to run the procedure call. This information includes the procedure to run and the arguments.
There is one important difference. The kernel has access to the address space of the calling process in a system call, so it is possible to pass pointers in a system call. For an RPC this won't work, since the address space lives in a separate physical memory from the process running the call. So, when sending arguments through RPC all of the information the procedure needs to run must be in the packet. If a string is to be passed, then the string itself must be present, not just a pointer to the string.
Often procedures share information through global variables (for instance global data structures) or kernel resources (open files, file location). This kind of sharing doesn't work with RPC, since the global space and kernel of the caller are separate from those of the callee.
Side effects also generally don't carry over between machines. A procedure to draw a picture on the screen might not be a good candidate for RPC-hood, since the display is a global resource on the machine that the procedure runs on. You don't want to draw the picture on the wrong machine.
Many window systems, such as X11, allow windowing applications to run on machines other than the one the display lives on, by sending all drawing requests off to the machine with the display. This is essentially using an RPC mechanism to give machines access to a machine local resource, the screen that the user is looking at, which lives on another machine.
Procedures you might want to run remotely would be things like computation intensive routines (which could be run on the local cray) or procedures which access resources not on your local machine (like remote file system access, remote data base access).
An RPC mechanism has a client side and a server side, analogous to the user code and the kernel code in the system call mechanism. The client side code packages the request up and sends it to the server, waits for the reply and unpackages the result. The server side code waits for requests, unpackages the arguments, runs the procedure, and packages and sends the result back to the caller.
RPC is typically implemented on top of unreliable hardware. Network problems include lost packets, corrupted packets, and reordering of packets. The first problem can be solved by giving each packet a unique number (serial number) and having the receiver send back an acknowledgement packet with this serial number, each time it correctly receives a packet. The serial number can also be used to solve the reordering problem, if the packets on each machine are given numbers in increasing order according to when they are sent. This also helps with the lost packet problem, since the receiving machine knows that a packet was lost or reordered as soon as an incoming packet doesn't have the next serial number.
The calling side of an RPC needs to know how to get in touch with the procedure to run on the server side. Two obvious ways of doing this come to mind. The first is to have a universally agreed upon RPC mail box number, which always gets messages for the RPC server. This server then figures out which procedure to run by inspecting the packet, and fires it up. This is like the system call implementation, in which the system call number always lived in the same register and inside the kernel there was a procedure which inspected this register and generated the call.
The other option would be to have each procedure which could be called remotely publish which mail box number it would listen to. To call this procedure the client would send a packet with the arguments to the correct mail box.
The second method is probably better in that it would be easier to debug and saves the demultiplexing step that the first one has to do on packet receipt. On the other hand it could potentially use up a large number of mail box slots, and if these are a limited resource (currently only 16 bits are available for box numbers on Unix), it would not be a good solution.