Skip to main content

Mosh (Mobile Shell) Beginner Guide for HPC Users

Overview​

Mosh (Mobile Shell) is a replacement for SSH's interactive layer that keeps your remote sessions alive across network changes, laptop sleep, and IP address transitions. If you've ever lost your terminal session because your WiFi dropped or you closed your laptop, Mosh is here to solve that frustration.

Think of Mosh as a UDP-based upgrade to SSH that works on top of SSH for authentication but uses its own connection layer for the interactive terminal. Your commands stay responsive even on laggy networks, and disconnections become recoverable rather than catastrophic.

For HPC/bioinformatics users: Mosh dramatically improves the experience when you're bouncing between your macOS laptop, weak WiFi, and your Linux cluster. Combined with tmux (which you'll learn about here), it's a game-changer for interactive computing workflows.

Prerequisites​

Before diving into Mosh, you should be comfortable with:

  • Basic SSH usage (ssh user@host)
  • Bash shell and command-line navigation
  • Understanding of login nodes vs. compute nodes on HPC clusters
  • (Optional but helpful) Basic tmux knowledge or willingness to learn

System requirements:

  • macOS 10.13+ (laptop) OR Linux distro with apt/dnf (laptop)
  • Linux HPC cluster with standard SSH access
  • Ability to install packages locally (no root needed for user-space Mosh)
  • UDP port access to cluster (typically ports 60000–61000)

Key Concepts​

What Mosh Does (and Doesn't Do)​

FeatureMoshSSHtmux
Survives network disconnection✅❌✅
Survives laptop sleep✅❌✅
Survives IP address change✅❌✅
Local echo (low latency)✅❌N/A
Session persistence❌❌✅
SSH-like authentication✅✅N/A

Critical limitation: Mosh does NOT provide session persistence on its own. When you close your laptop or disconnect, your running processes will be killed unless you're inside a tmux session. This is why we pair Mosh with tmux—tmux keeps your work alive, Mosh keeps your connection alive.

Why Local Echo Matters for HPC Work​

Mosh predicts your keystrokes locally before the server responds. On a cluster with 200ms latency, you'll see your typed characters instantly instead of waiting 200ms between each keystroke. This makes interactive work feel snappy again.

UDP vs. SSH: The Technical Difference​

  • SSH: Maintains a persistent TCP connection; reconnection means starting fresh
  • Mosh: Uses UDP for the terminal layer; survives packet loss and reconnects automatically

You still authenticate via SSH, so your key-based login and security model are unchanged.

Step-by-Step Instructions​

1. Install Mosh on Your Local Machine​

On macOS:

brew install mosh

On Linux (Ubuntu/Debian):

sudo apt update
sudo apt install mosh

On Linux (Fedora/RHEL):

sudo dnf install mosh

Verify the installation:

mosh --version

Expected output:

mosh 1.4.0

2. Install Mosh on Your HPC Cluster​

Most HPC clusters don't have Mosh in the system package manager. Instead, use mamba (or conda) to install it in your user space—no root needed.

# On the cluster login node
mamba install -c conda-forge mosh

Verify it's in your PATH:

which mosh-server

Expected output:

/home/youruser/mambaforge/bin/mosh-server

If this returns nothing, add mamba's bin directory to your $PATH in your ~/.bashrc:

export PATH="/home/youruser/mambaforge/bin:$PATH"

Then source it:

source ~/.bashrc
which mosh-server

3. Test a Basic Mosh Connection​

Replace login1.cluster.edu and youruser with your actual cluster details:

mosh youruser@login1.cluster.edu

Expected behavior:

  • Mosh connects via SSH (you may be prompted for a password or key passphrase)
  • The banner and prompt appear almost instantly
  • You can type commands and see responses with minimal latency

To exit cleanly:

exit

This properly terminates the Mosh server process on the remote end.

4. Configure Non-Standard SSH Port​

If your cluster uses SSH on a non-standard port (e.g., 2222), tell Mosh:

mosh --ssh="ssh -p 2222" youruser@login1.cluster.edu

Better long-term: Add this to your ~/.ssh/config:

Host cluster
    HostName login1.cluster.edu
    User youruser
    Port 2222

Then simply:

mosh cluster

Much cleaner!

5. Handle Firewall and UDP Port Issues​

Mosh uses UDP ports in the range 60000–61000. If you suspect UDP is blocked:

Test connectivity to the cluster:

# On your local machine, test if you can reach the cluster on a specific UDP port
nc -u -z -v login1.cluster.edu 60000

If it hangs or fails, UDP may be blocked by:

  • Your local firewall
  • The cluster's firewall
  • Your ISP or corporate network

Workaround: Use the --port flag to try a port in the allowed range (if you know it):

mosh --port=5000 youruser@login1.cluster.edu

If UDP is completely blocked: You're stuck with SSH. But for most academic HPC centers, UDP is open.

6. Create an Alias for Frequently Used Clusters​

Add this to your ~/.bashrc or ~/.zshrc:

alias mosh-cluster='mosh --ssh="ssh -p 2222" youruser@login1.cluster.edu'

Now you can simply:

mosh-cluster

Practical Examples​

Example 1: Quick One-Off Command Over Laggy Connection​

You're on WiFi and need to check job status. Instead of SSH:

mosh youruser@login1.cluster.edu
# You're immediately connected and responsive, even if WiFi is spotty
squeue -u youruser
# Close laptop without worrying about hanging SSH processes
exit

Example 2: The Mosh + Tmux Combo (Recommended Workflow)​

This is where Mosh truly shines. The golden rule:

  1. Mosh to the login node
  2. Start or attach a tmux session
  3. Do your work
  4. Disconnect (laptop sleep, WiFi drop, etc.)
  5. Next day: Mosh back and tmux attach

Step-by-step:

# Day 1: Connect and start tmux
mosh youruser@login1.cluster.edu
tmux new-session -s work
# ... do analysis, submit jobs, etc.
# Close laptop (Ctrl+D to exit terminal, or just close it)
# Day 2: Reconnect (WiFi, IP, location doesn't matter)
mosh youruser@login1.cluster.edu
tmux attach-session -t work
# Your jobs are still running, your prompt is exactly where you left it!

Example 3: Interactive Slurm Job with Mosh + Tmux​

You need to run an interactive job on a compute node:

# Mosh to the login node
mosh youruser@login1.cluster.edu

# Start tmux (optional but recommended)
tmux new-session -s interactive

# Request compute resources
salloc --nodes=1 --cpus-per-task=8 --mem=16G --time=01:00:00

# Once you get a shell on the compute node, you're inside tmux
# Start your analysis
python my_analysis.py

Important caveat: Mosh cannot SSH directly to a compute node because compute nodes typically don't route UDP to the login node. If your Mosh connection drops after salloc, you can reconnect via the login node and then SSH directly to the compute node:

# From login node (still in tmux)
ssh computenode42
# Rejoin your interactive session

See [[sesh-beginner-guide]] for more session management strategies.

Example 4: Auto-Attach Tmux with an Alias​

Create a single command that connects and attaches tmux automatically:

Add to ~/.bashrc:

alias mosh-work='mosh youruser@login1.cluster.edu -c ~/.tmux.conf -- tmux attach-session -t work || tmux new-session -s work'

Now:

mosh-work
# You're automatically in your tmux session

To make tmux play nicely with Mosh, add these to your ~/.tmux.conf:

# Enable true color support
set -g default-terminal "screen-256color"
set -ga terminal-overrides ",xterm-256color:RGB"

# Reduce escape time (Mosh already handles latency, so we don't need to wait)
set -sg escape-time 0

# Enable mouse support for pane navigation
set -g mouse on

# Friendly keybindings
bind-key -T copy-mode-vi v send-keys -X begin-selection
bind-key -T copy-mode-vi y send-keys -X copy-selection

# Easy pane navigation with arrow keys
bind-key -n M-Up select-pane -U
bind-key -n M-Down select-pane -D
bind-key -n M-Left select-pane -L
bind-key -n M-Right select-pane -R

For more dotfile management, see [[dotfiles-beginner-guide]].

Hands-On Exercises​

Exercise 1: First Mosh Connection​

  1. Open a terminal on your local machine
  2. SSH to your cluster's login node once to verify it works
  3. Now try mosh youruser@cluster instead
  4. Type a few commands (pwd, ls, whoami) and notice the responsiveness
  5. Exit cleanly with exit

Expected outcome: Mosh connects and feels smooth.

Exercise 2: Test Disconnection Recovery​

  1. Mosh into your cluster
  2. Start a long-running command (e.g., sleep 60; echo "Done")
  3. Manually toggle WiFi off on your laptop (or unplug ethernet)
  4. Wait 10 seconds, toggle it back on
  5. Mosh automatically reconnects
  6. The sleep command completes and prints "Done"

Expected outcome: Your session survives the network hiccup.

Exercise 3: Mosh + Tmux Workflow​

  1. Mosh to your cluster
  2. tmux new-session -s exercise
  3. Start a background job: nohup sleep 120 &
  4. Note the job ID
  5. Close your terminal or sleep your laptop
  6. Reconnect: mosh cluster
  7. tmux attach-session -s exercise
  8. Verify the job is still running: ps aux | grep sleep

Expected outcome: Job survives the disconnection because tmux kept it alive.

Exercise 4: UDP Port Testing​

  1. From your local machine, test if a port is reachable: nc -u -z -v login1.cluster.edu 60000
  2. If it fails, try another port in your known-open range
  3. Use mosh --port=XXXX to connect on that port
  4. Verify connection works

Expected outcome: Understanding of how to diagnose firewall issues.

Troubleshooting​

Problem: "mosh-server: command not found"​

Cause: mosh-server isn't in your $PATH on the cluster.

Solution:

# SSH to the cluster
ssh youruser@login1.cluster.edu

# Find where it is
find ~ -name "mosh-server" -type f

# Add it to your PATH in ~/.bashrc
export PATH="/home/youruser/mambaforge/bin:$PATH"

# Source it
source ~/.bashrc

# Verify
which mosh-server

Problem: "mosh-server needs a UTF-8 locale"​

Cause: The cluster's default locale doesn't support UTF-8.

Solution: Override the locale before connecting:

LANG=en_US.UTF-8 mosh youruser@login1.cluster.edu

Or add to your ~/.bashrc on the cluster:

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

Problem: Mosh Connection Hangs​

Cause: UDP is likely blocked by a firewall.

Solutions:

  1. Verify you're on the same network as before (IP address changed?)
  2. Test UDP connectivity: nc -u -z login1.cluster.edu 60000
  3. Ask your cluster admin if UDP 60000–61000 is open from outside
  4. Try SSH instead as a fallback: ssh youruser@login1.cluster.edu

Problem: Stale Mosh Processes​

Cause: Mosh server processes didn't clean up properly after disconnection.

Symptom: Can't reconnect; "connection refused" or hangs.

Solution: Kill stale processes on the cluster:

# SSH to the cluster
ssh youruser@login1.cluster.edu

# List all your mosh-server processes
pgrep -u $USER mosh-server

# Kill them
pkill -u $USER mosh-server

# Now try reconnecting
mosh youruser@login1.cluster.edu

Problem: Mosh and SSH Version Mismatch​

Cause: Your local mosh client is a different version than mosh-server on the cluster.

Symptom: Cryptic errors during connection.

Solution: Upgrade both to the same version:

# Local machine
brew upgrade mosh  # macOS
# or
sudo apt upgrade mosh  # Linux

# On cluster
mamba install -c conda-forge mosh=1.4.0  # Match your local version

Mosh works best alongside these complementary tools and guides:

  • [[dotfiles-beginner-guide]] and [[dotfiles-deep-dive]] — Manage your .tmux.conf, .bashrc, SSH config, and other dotfiles across machines
  • [[sesh-beginner-guide]] and [[sesh-deep-dive]] — Master terminal session management with tmux; understand how sessions, windows, and panes work
  • [[linux-permissions-beginner-guide]] — Understand user-space installations and how to install tools like Mosh without root
  • [[kubernetes-beginner-guide]] — If working in containerized HPC environments, understand how Mosh behaves in Kubernetes clusters
  • [[mosh-deep-dive]] — Advanced Mosh topics, scripting, and optimization for power users

  • [[openmux-beginner-guide|OpenMux Beginner Guide]] and [[openmux-deep-dive|OpenMux Deep Dive]] — modern terminal multiplexer alternative to tmux

  • [[micropython-ttgo-t-display-beginner-guide|MicroPython TTGO T-Display Beginner Guide]] — remote access to headless IoT devices

  • [[micropython-ttgo-t-display-deep-dive|MicroPython TTGO T-Display Deep Dive]] — persistent remote sessions for long-running embedded projects

  • [[hyperqueue-basics|HyperQueue Basics]] — HPC task scheduler you'll manage over remote connections

  • [[hyperqueue-deep-dive|HyperQueue Deep Dive]] — production HQ on Slurm clusters

  • [[honeymux-beginner-guide|Honeymux Beginner Guide]] — TUI wrapper for tmux with SSH pane stitching

  • [[docker-test-container-beginner-guide|Docker Test Container Beginner Guide]] — test mosh/tmux configs in containers

  • [[ssh-tutorial|SSH Tutorial]]

  • [[ssh-config-deep-dive|SSH Config Deep Dive]]

  • [[dtach-beginner-guide|Dtach Beginner Guide]] — lightweight session detach/reattach without a full multiplexer

  • [[dtach-deep-dive|Dtach Deep Dive]] — advanced dtach patterns, scripting, and integration with Mosh

  • [[tmux-claude-code-beginner-guide|Tmux + Claude Code Beginner Guide]] — Run Claude Code in tmux for persistent remote sessions

  • [[tmux-claude-code-deep-dive|Tmux + Claude Code Deep Dive]] — Mosh + tmux + Claude Code for ultimate remote development

  • [[pixi-beginner-guide|Pixi Beginner Guide]] — Modern conda/PyPI package manager (replaces mamba for environment management)

  • [[pixi-deep-dive|Pixi Deep Dive]] — Advanced pixi patterns for HPC and scientific computing

  • [[uv-beginner-guide|uv Beginner Guide]] — Fast Python package manager for pure-Python projects

  • [[uv-deep-dive|uv Deep Dive]] — Advanced uv workflows, CI/CD, and Docker integration

  • [[headscale-beginner-guide|Headscale Beginner Guide]] — self-hosted mesh VPN; Mosh sessions work seamlessly over Headscale tailnets

  • [[headscale-deep-dive|Headscale Deep Dive]] — advanced Headscale networking and DERP relays

Summary​

Mosh is a game-changer for interactive HPC work, especially on laptops with flaky networks. Here's what you now know:

  1. Mosh survives network changes, laptop sleep, and IP transitions — your SSH authentication stays, but Mosh handles the interactive layer with UDP
  2. Mosh is NOT a persistence tool — pair it with tmux for true session survival
  3. Installation is straightforward: brew install mosh locally, mamba install -c conda-forge mosh on the cluster
  4. Basic usage is simple: mosh user@host works just like SSH
  5. UDP firewalls can block Mosh — know your cluster's firewall rules and have SSH as a fallback
  6. The killer combo is Mosh + tmux + a pinned login node: Start a tmux session, disconnect, reconnect days later, and everything is still there
  7. Interactive Slurm jobs work well with Mosh — use salloc inside a tmux session for compute access
  8. Troubleshooting is usually about locale or firewall issues — LANG=en_US.UTF-8 and nc -u -z are your friends

Next steps:

  • Install Mosh locally and on your cluster this week
  • Pair it with tmux and experience the difference
  • Read [[sesh-beginner-guide]] to master tmux if you haven't already
  • See [[mosh-deep-dive]] when you're ready for advanced optimizations

Happy remote computing!