Can Your Agent Leave Modal Without Double-Submitting Jobs?

Replay interrupted jobs against Modal and the candidate runtime using the same synthetic cases. Compare completion time and downstream commit counts before moving traffic.

Make the accepted request recoverable outside the snapshot

Modal documents separate filesystem and directory snapshot operations with configurable retention. Use a snapshot to recreate your test fixture, then test whether a fresh process can recover an accepted job from records your application controls. Give every fixture job a stable ID, an input digest, and a destination for its result in a test ledger. Simulate a lost acknowledgment by letting the ledger accept one write and dropping the response in your fixture. Restart the worker and submit the same job ID again. The observable outcome is one committed result and a successful completion, even if the worker ran twice. Run a second case where the first attempt ends before the ledger accepts the write. That case should still reach one commit. Together the cases expose a system that duplicates completed work and one that suppresses a retry before any work was committed.

Snapshots | Modal Docs

Benchmark a clean restore with the clock running

Docker’s sandbox move preserves a filesystem snapshot while live process state stays behind. Use that distinction to design a fair Modal comparison even if the candidate runtime has a different migration mechanism. Give each environment the same dependency manifest and synthetic job cases, start timing at submission, and stop when the test ledger first records a correct result. Keep measuring ledger commits through a fixed observation window so a late retry can still expose a duplicate. Record deadline misses separately from unfinished cases. A candidate that cuts median time from 40 to 25 seconds still fails a 60-second deadline gate if a cold restore takes 75 seconds. Review each failing case against its counterpart before switching traffic. If the candidate starts slowly only when dependencies are missing, rerun with the build inputs pinned and record both results.

Move a sandbox | Docker Docs

Separate the coordinator from the job

Trace one run from trigger to result. Mark which process waits for work, which step needs a GPU, and where the application records completed jobs. Modal lists GPU task rates by accelerator model, including the Nvidia A100 80 GB. If a step depends on that accelerator, keep it on Modal while you evaluate moving the process that submits work and records the results.

Give the GPU step a small interface with a job ID, an input location, and an output location. Replay the same fixture through the old coordinator and a staging coordinator. Both should produce the expected artifact and record completion under the same job ID. If the GPU step reads local files or relies on credentials available only inside the old coordinator, resolve those dependencies before splitting the workload.

Pricing: Pay per second for GPUs and CPUs | Modal

Prove the checkpoint can travel

Inventory the source agent's configuration, working files, queue cursor, and record of completed actions. Modal documents a sandbox filesystem snapshot operation. Use a snapshot as a source-side recovery point if it covers your files, and export application data separately in a format the destination can read. Reissue credentials through your normal secret provisioning process. Record the cursor and last completed job ID at the same handoff point.

In an illustrative rehearsal, the source has completed jobs 1 through 42, and the staging queue contains a duplicate of job 42 plus new jobs 43 and 44. Restore the checkpoint and replay all three inputs against a staging output destination. Expect zero new effects for 42 and one each for 43 and 44. Restart the destination and replay job 44 once more. Any duplicate effect or missing output blocks the cutover.

Snapshots | Modal Docs

Sources