package rpc_parallel

README

**Mental Model**:

- [Worker.t] identifies a worker rpc server
- [spawn] ([serve]) starts a worker Rpc server in another process (the same process)
- [client] connects to a worker Rpc server
- [run] dispatches on a connection to a worker Rpc server

**Top-level**:

It is highly recommended for [Parallel.start_app] and [Parallel.Make]
calls to be top-level. But the real requirements are:

1) The master's state is initialized before any calls to [spawn]. This will be
achieved either by [Parallel.start_app] or [Parallel.init_master_exn].

2) Spawned workers (runs of your executable with a certain environment variable set)
must start running as a worker. This will be achieved either by [Parallel.start_app]
or [Parallel.run_as_worker_exn]. With [Parallel.run_as_worker_exn], you must have
carefully supplied the necessary command arguments to ensure
[Parallel.run_as_worker_exn] is indeed called.

3) Spawned workers must be able to find their function implementations when they start
running as a worker. These implementations are gathered on the application of the
[Parallel.Make] functor.

4) The worker implementations must be defined completely and in the same order,
regardless of master and worker code paths. This is necessary for the masters and
workers to agree on certain generated ids.

**Cleanup**:

It is *extremely* important to always be thinking about cleanup, as resources can
easily be eaten up on machines if processes don't cleanup after themselves. When a
worker is spawned, there is nothing by default in place to kill this spawned process
if the master process dies. There are a lot of mechanisms exposed to facilitate
cleanup, please think carefully and program defensively. There are 3 main ways to
facilitate cleanup, and they are not necessarily mutually exclusive:

1) Use the [parent_heartbeater] argument to [init_worker_state] in order to set up shutdown
on losing connection to this server. The [connect_and_shutdown_on_disconnect_exn]
does just this, and should commonly be used.

2) Add a function to your worker that shuts it down. There is a built-in [shutdown]
function for you to include to make this easier.

3) Use the [connection] argument to [init_connection_state] to register a callback for
[close_finished]. This should be used if you expect only one connection to the worker
for its entire life, but it is not as safe as [connect_and_shutdown_on_disconnect] as
the worker will be left stranded if no connection is ever made to it.

Some (obvious but worth reinforcing) things to consider:
- A spawned worker is just sitting there in another process doing nothing unless some
  cleanup is setup in [init_worker_state]

- If your worker expects a [shutdown] rpc call, it is still left stranded if it never
  gets this call for whatever reason.

**Monitoring your workers**:

Uncaught exceptions in workers will always result in the worker shutting down.
The master can be notified of these exceptions in multiple ways:

- If the exception occured in a function implementation [f] before [f] is
  determined, the exception will be returned back to the caller.
  E.g. the caller of [spawn] or [run] will get an [Error.t] describing the exception.

- If the exception occured after [f] is determined, [on_failure exn] will be
  called (in [Monitor.current ()] at the time of [spawn]) in the spawning process.

- If [redirect_stderr] specifies a file, the worker will also write its exception to
  that file before shutting down.

**Optional Rpc Settings**:

The master's Rpc server will be started with the [max_message_size],
[heartbeat_config], and [handshake_timeout] settings passed in to
[start_app]/[init_master_exn].

Each worker's Rpc server will be started with the settings passed in upon [spawn] or
[serve] of that worker.

All [client] calls will use the corresponding Rpc settings for the given worker/master
that it is connecting to.