Using Shared Memory (/dev/shm)

10.1. Using Shared Memory (/dev/shm)#

Concept: A temporary filesystem (tmpfs) residing entirely in RAM.

  • Extremely fast access times.

  • Standard directory interface.

  • Typically capped at 50% of total physical RAM.

10.1.1. HPC Benefits#

  • Apptainer Performance: Pull and unpack container layers at RAM speeds.

  • High-Speed I/O: Near-zero wait times for heavy read/write operations on temporary files.

10.1.2. Critical Pitfalls#

  • ⚠️ “RAM Theft”: Files count towards the Slurm --mem allocation (risk of OOM Kill).

  • ⚠️ Volatility: Data is lost instantly upon node reboot.

  • ⚠️ “Ghost Files”: Files are not automatically cleaned up by Slurm after a job ends.

10.1.3. Proper Management & Apptainer Usage#

Best Practice: Always use a Bash trap to ensure cleanup upon job completion or failure.

export JOB_SHM="/dev/shm/${USER}_${SLURM_JOB_ID}"
mkdir -p "$JOB_SHM"

cleanup_shm() { rm -rf "$JOB_SHM"; }
trap cleanup_shm EXIT SIGTERM SIGINT
Content