Note: general recommendations only, not professional advice. You are responsible for checking before use. Test on a non-production server first, and keep console access. The full text is at the end.
What you'll have at the end
- Login with an SSH key only, a firewall open only to SSH, automatic blocking of break-in attempts, and security updates that install by themselves.
- Secrets in one protected file, loaded into the service and never in code, git or logs.
- A systemd service for the agent that starts after a reboot and comes back after a crash, and a check that it really responds, not just "runs".
- A watchdog every 5 minutes that checks the agent, the disk, memory and the last job, and sends Telegram one alert per problem and a message when it's fixed.
- Protected memory: swap, and a memory ceiling for heavy commands, so one build can't take the server down.
- Safe scheduled jobs: no overlapping runs, with logs, with a time limit.
- An agent with least privilege, a list of forbidden commands, and an approval queue for every outward action, with a log.
- An encrypted nightly backup off the server, and a real restore test.
- A 10-minute weekly checklist.
What you need
- A small Ubuntu server (VPS), with root or sudo access, and a second server or machine to test on before you touch the real one.
- Console access from your server provider (Console / VNC / Rescue in the control panel). If you lock yourself out of SSH, that's the way back in. Check that you know how to use it before step 1.
- A Telegram bot for alerts (created in BotFather with
/newbot) and your chat_id. - Storage off the server for backups: an S3-compatible bucket, Backblaze B2, Google Drive or any destination rclone supports.
How to read this kit: every step has a why, files and commands to copy, and a check with the output you should see. In the commands, SERVER_IP is your server's address, ops is your user, agent is the agent's user, and my-agent is the service name. You can change the names, just consistently.
The kit, step by step (about two hours)
Step 1 (25 minutes): Basic lockdown
Why: most break-ins on small servers are SSH password guessing and software that wasn't updated. A key instead of a password, a firewall and automatic updates close most of the risk.
1a. An SSH key and a user with sudo. On your computer (if you don't have a key yet):
ssh-keygen -t ed25519 -C "my-laptop"
ssh-copy-id -i ~/.ssh/id_ed25519.pub root@SERVER_IP
On the server, a personal user with sudo that gets the same key:
sudo adduser --gecos "" ops
sudo usermod -aG sudo ops
sudo install -d -m 700 -o ops -g ops /home/ops/.ssh
sudo install -m 600 -o ops -g ops ~/.ssh/authorized_keys /home/ops/.ssh/authorized_keys
The ops password is only for sudo, not for logging in.
What about root? If the agent has to manage the server itself and therefore runs as root, you can keep root login, but with a key only (PermitRootLogin prohibit-password, as in the file below). If you work as ops, change that line to PermitRootLogin no.
1b. Key-only login. Save this file as /etc/ssh/sshd_config.d/00-hardening.conf:
# /etc/ssh/sshd_config.d/00-hardening.conf
# sshd uses the FIRST value it reads, and files here are read in name order,
# so "00-" wins over a cloud image's "50-cloud-init.conf".
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
PermitRootLogin prohibit-password
MaxAuthTries 3
LoginGraceTime 30
X11Forwarding no
sudo sshd -t && sudo systemctl restart ssh
Important: don't close the open window. Open a second window and log in: ssh ops@SERVER_IP, then sudo -v. Only if that works, close the first one. If it doesn't, fix it from the first window, which is still connected.
sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin)'
# permitrootlogin prohibit-password (or no)
# passwordauthentication no
# kbdinteractiveauthentication no
And from your computer, a password login has to fail:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password ops@SERVER_IP
# ops@SERVER_IP: Permission denied (publickey).
1c. Firewall (ufw). Allow SSH first, and only then turn it on:
sudo ufw allow OpenSSH
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw enable
Check: sudo ufw status verbose shows Status: active and a line 22/tcp (OpenSSH) ALLOW IN Anywhere. An agent that talks to Telegram and to APIs needs no other incoming port. If your provider has its own firewall in the control panel, keep only 22 open there too.
1d. fail2ban: blocks an address after a few failed logins.
# /etc/fail2ban/jail.local
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 5
backend = systemd
[sshd]
enabled = true
sudo apt install -y fail2ban
sudo cp jail.local /etc/fail2ban/jail.local
sudo systemctl restart fail2ban
Check: sudo fail2ban-client status sshd starts with Status for the jail: sshd, and shows Currently banned and Total banned.
1e. Automatic security updates:
// /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";
sudo apt install -y unattended-upgrades
sudo cp 20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades
apt-config dump | grep Periodic::Unattended-Upgrade
# APT::Periodic::Unattended-Upgrade "1";
sudo unattended-upgrade --dry-run; echo "exit=$?"
# exit=0
A kernel update needs a reboot: if /var/run/reboot-required exists, reboot when it suits you (it's on the weekly checklist).
Step 2 (10 minutes): Secrets in one place
Why: a token in the code ends up in git, in a screenshot or in a log. One file that only root reads, loaded into the service by systemd, solves that.
sudo install -d -m 755 /etc/my-agent
sudo install -m 600 -o root -g root /dev/null /etc/my-agent/agent.env
sudoedit /etc/my-agent/agent.env
# /etc/my-agent/agent.env owner root, chmod 600. Never in git, never in code, never printed to logs.
TELEGRAM_TOKEN=put-your-bot-token-here
API_KEY=put-your-model-api-key-here
HEARTBEAT_FILE=/var/lib/my-agent/heartbeat
The service in step 3 loads the file with EnvironmentFile=. systemd reads it as root before switching to the agent's user, so the file stays 600 and owned by root.
In the agent's project, a .gitignore with these lines:
.env
.env.*
*.pem
*.key
secrets/
And in the code: don't print environment variables, don't write a token to a log, and hide it in error messages.
sudo stat -c '%a %U %n' /etc/my-agent/agent.env
# 600 root /etc/my-agent/agent.env
git check-ignore -v .env
# .gitignore:1:.env .env
git grep -nIE '(api[_-]?key|token|secret|password)\s*[:=]\s*["'\'']?[A-Za-z0-9_\-:]{16,}'
# (no output)
If a token leaked: revoke it at the provider first (Telegram: /revoke in BotFather; an API: delete the key in its console), create a new one, update the file and restart the service. Deleting it from git isn't enough, because the token stays in the history. Only replacing it closes the leak.
Step 3 (15 minutes): The agent comes back by itself after a reboot
Why: an agent started by hand in a terminal disappears on the first reboot, the first crash or when the connection closes. A systemd service starts it with the server and brings it back after a crash.
A separate user for the agent, with no password and no sudo:
sudo useradd --system --create-home --home-dir /opt/my-agent --shell /usr/sbin/nologin agent
sudo install -d -o agent -g agent /var/lib/my-agent
/opt/my-agent/run.sh starts the agent itself:
#!/usr/bin/env bash
set -euo pipefail
cd /opt/my-agent
exec python3 bot.py # or: exec node index.js
[Unit]
Description=My AI agent
Wants=network-online.target
After=network-online.target
# Stop trying after 5 failed starts in 10 minutes (the watchdog will tell you)
StartLimitIntervalSec=600
StartLimitBurst=5
[Service]
Type=simple
User=agent
Group=agent
WorkingDirectory=/opt/my-agent
EnvironmentFile=/etc/my-agent/agent.env
ExecStart=/opt/my-agent/run.sh
Restart=on-failure
RestartSec=10
# Keep one runaway process from taking the whole server down
MemoryMax=1500M
MemorySwapMax=0
# Basic sandboxing: the agent may write only where it needs to
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=read-only
ReadWritePaths=/opt/my-agent /var/lib/my-agent
[Install]
WantedBy=multi-user.target
sudo chmod 755 /opt/my-agent/run.sh
sudo cp my-agent.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now my-agent
systemctl is-enabled my-agent; systemctl is-active my-agent
# enabled
# active
sudo systemctl kill -s KILL my-agent; sleep 15; systemctl is-active my-agent
# active (systemd brought it back after the kill)
sudo reboot # and after the server is back:
systemctl is-active my-agent
# active
"Running" is not "working". A process can be alive and stuck. So the agent writes a heartbeat: at the end of every successful pass of its loop it touches a file. The watchdog in step 4 checks that the file is fresh.
# bash
touch "$HEARTBEAT_FILE"
# Python
pathlib.Path(os.environ["HEARTBEAT_FILE"]).touch()
# Node.js
fs.writeFileSync(process.env.HEARTBEAT_FILE, String(Date.now()))
Put the touch after something that proves the agent works (a successful Telegram call, an answer from the model), not on a separate timer. If the agent has an HTTP health endpoint, set it as HEALTH_URL in step 4.
Check: sudo find /var/lib/my-agent/heartbeat -mmin -15 prints the path. No output means the agent hasn't written a heartbeat in the last 15 minutes.
An interactive agent (inside tmux): an agent you want to attach to and watch, such as a command-line coding agent, runs inside tmux. This service starts the tmux session with the server:
[Unit]
Description=My AI agent (interactive, inside tmux)
Wants=network-online.target
After=network-online.target
[Service]
Type=forking
User=agent
Group=agent
WorkingDirectory=/opt/my-agent
EnvironmentFile=/etc/my-agent/agent.env
# Attach later with: sudo -u agent tmux attach -t agent
ExecStart=/usr/bin/tmux new-session -d -s agent /opt/my-agent/run.sh
ExecStop=/usr/bin/tmux kill-session -t agent
# tmux exits cleanly when the agent quits, so restart on any exit
Restart=always
RestartSec=15
[Install]
WantedBy=multi-user.target
Attach with sudo -u agent tmux attach -t agent, and detach without stopping it with Ctrl-b then d. Check: sudo -u agent tmux ls shows agent: 1 windows.
Step 4 (15 minutes): A watchdog with Telegram alerts
Why: you won't look at the server every hour. A small script every 5 minutes checks what really breaks agents, and sends a message only when something changes.
What it checks: the service is running; the heartbeat is fresh (and the health URL, if you set one); disk; free RAM and swap; the result of the scheduled job's last run. For each problem: one alert, a reminder only after an hour if it's still open, and a ✅ message when it's fixed. When the agent is down or silent, it restarts it once (not in a loop). The token reaches curl through stdin, so it never shows up in ps or in a log.
#!/usr/bin/env bash
# Agent watchdog: checks the agent and the server, alerts on Telegram once per problem,
# repeats only every REPEAT_MIN minutes, and sends a recovery message when it is fixed.
# Config: /etc/my-agent/watchdog.env (chmod 600). DRY_RUN=1 prints instead of sending.
set -uo pipefail
SERVICE="${SERVICE:-my-agent.service}" # the agent's systemd unit
HEARTBEAT_FILE="${HEARTBEAT_FILE:-/var/lib/my-agent/heartbeat}"
HEARTBEAT_MAX_MIN="${HEARTBEAT_MAX_MIN:-15}" # the agent touches this file at least this often
HEALTH_URL="${HEALTH_URL:-}" # optional: an HTTP health endpoint of the agent
JOB_UNIT="${JOB_UNIT:-}" # optional: the scheduled job whose last result to check
DISK_MAX_PCT="${DISK_MAX_PCT:-85}"
MEM_MIN_AVAIL_MB="${MEM_MIN_AVAIL_MB:-300}"
SWAP_MAX_PCT="${SWAP_MAX_PCT:-80}"
REPEAT_MIN="${REPEAT_MIN:-60}" # repeat an open alert at most this often
AUTO_RESTART="${AUTO_RESTART:-1}" # restart the agent once when it is down or silent
STATE_DIR="${STATE_DIR:-/var/lib/my-agent-watchdog}"
MEMINFO="${MEMINFO:-/proc/meminfo}"
HOST_LABEL="${HOST_LABEL:-$(hostname -s)}"
DRY_RUN="${DRY_RUN:-0}"
mkdir -p "$STATE_DIR"
send() {
local text="$1"
if [[ "$DRY_RUN" == "1" ]]; then
printf 'DRY-RUN message: %s\n' "$text"
return 0
fi
if [[ -z "${TELEGRAM_TOKEN:-}" || -z "${TELEGRAM_CHAT_ID:-}" ]]; then
echo "watchdog: TELEGRAM_TOKEN or TELEGRAM_CHAT_ID missing" >&2
return 1
fi
# The URL (with the token) goes through stdin, so it never shows up in `ps` or in logs
printf 'url = "https://api.telegram.org/bot%s/sendMessage"\n' "$TELEGRAM_TOKEN" |
curl -sS --max-time 15 -o /dev/null -w '%{http_code}' -K - \
--data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
--data-urlencode "text=${text}" | grep -q '^200$' || echo "watchdog: Telegram send failed" >&2
}
# Remember each open problem in a file: alert when it opens, repeat every REPEAT_MIN, recover when it closes
problem() { # problem <key> <message>
local key="$1" msg="$2" f="$STATE_DIR/$1.open"
if [[ ! -f "$f" ]] || [[ -n "$(find "$f" -mmin +"$REPEAT_MIN" 2>/dev/null)" ]]; then
send "⚠️ ${HOST_LABEL}: ${msg}" && printf '%s\n' "$msg" > "$f"
fi
OPEN+=("$key")
}
resolved() { # resolved <key> <message>
local f="$STATE_DIR/$1.open"
if [[ -f "$f" ]]; then
send "✅ ${HOST_LABEL}: ${2}" && rm -f "$f"
fi
}
OPEN=()
restart_once() {
[[ "$AUTO_RESTART" == "1" ]] || return 0
local f="$STATE_DIR/restarted"
# at most one automatic restart per REPEAT_MIN, so a broken agent is not restarted in a loop
if [[ ! -f "$f" ]] || [[ -n "$(find "$f" -mmin +"$REPEAT_MIN" 2>/dev/null)" ]]; then
touch "$f"
if [[ "$DRY_RUN" == "1" ]]; then echo "DRY-RUN restart: $SERVICE"; else systemctl restart "$SERVICE"; fi
fi
}
# 1. The service is running
if systemctl is-active --quiet "$SERVICE"; then
resolved service "$SERVICE is running again"
else
problem service "$SERVICE is not running ($(systemctl is-active "$SERVICE" 2>/dev/null))"
restart_once
fi
# 2. The agent is really alive: a fresh heartbeat file (and the health URL, if set)
if [[ -f "$HEARTBEAT_FILE" ]] && [[ -z "$(find "$HEARTBEAT_FILE" -mmin +"$HEARTBEAT_MAX_MIN" 2>/dev/null)" ]]; then
resolved heartbeat "the agent's heartbeat is fresh again"
else
problem heartbeat "no heartbeat from the agent for over ${HEARTBEAT_MAX_MIN} minutes"
restart_once
fi
if [[ -n "$HEALTH_URL" ]]; then
if curl -fsS --max-time 10 -o /dev/null "$HEALTH_URL"; then
resolved health "the health check answers again"
else
problem health "the health check at $HEALTH_URL does not answer"
fi
fi
# 3. Disk
disk_pct="$(df --output=pcent / | tail -1 | tr -dc '0-9')"
if (( disk_pct >= DISK_MAX_PCT )); then
problem disk "disk is ${disk_pct}% full (limit ${DISK_MAX_PCT}%)"
else
resolved disk "disk is back to ${disk_pct}%"
fi
# 4. Memory and swap
mem_avail_kb="$(awk '/^MemAvailable:/ {print $2}' "$MEMINFO")"
swap_total_kb="$(awk '/^SwapTotal:/ {print $2}' "$MEMINFO")"
swap_free_kb="$(awk '/^SwapFree:/ {print $2}' "$MEMINFO")"
mem_avail_mb=$(( ${mem_avail_kb:-0} / 1024 ))
if (( mem_avail_mb < MEM_MIN_AVAIL_MB )); then
problem memory "only ${mem_avail_mb} MB RAM available (limit ${MEM_MIN_AVAIL_MB} MB)"
else
resolved memory "RAM is back to ${mem_avail_mb} MB available"
fi
if (( ${swap_total_kb:-0} > 0 )); then
swap_pct=$(( (swap_total_kb - swap_free_kb) * 100 / swap_total_kb ))
if (( swap_pct >= SWAP_MAX_PCT )); then
problem swap "swap is ${swap_pct}% used (limit ${SWAP_MAX_PCT}%)"
else
resolved swap "swap is back to ${swap_pct}%"
fi
fi
# 5. The last run of the scheduled job
if [[ -n "$JOB_UNIT" ]]; then
result="$(systemctl show -p Result --value "$JOB_UNIT" 2>/dev/null)"
if [[ -z "$result" || "$result" == "success" ]]; then
resolved job "$JOB_UNIT succeeded again"
else
problem job "the last run of $JOB_UNIT ended with: $result"
fi
fi
if (( ${#OPEN[@]} == 0 )); then echo "watchdog: all ok"; else echo "watchdog: open: ${OPEN[*]}"; fi
exit 0
[Unit]
Description=Watchdog for my AI agent
[Service]
Type=oneshot
EnvironmentFile=/etc/my-agent/watchdog.env
ExecStart=/usr/local/bin/my-agent-watchdog
TimeoutStartSec=120
[Unit]
Description=Run the agent watchdog every 5 minutes
[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
AccuracySec=30s
[Install]
WantedBy=timers.target
sudo install -m 755 watchdog.sh /usr/local/bin/my-agent-watchdog
sudo install -m 600 -o root -g root /dev/null /etc/my-agent/watchdog.env
sudoedit /etc/my-agent/watchdog.env
In watchdog.env:
TELEGRAM_TOKEN=token-of-your-alerts-bot
TELEGRAM_CHAT_ID=your-chat-id
JOB_UNIT=my-job.service
To find the chat_id: send any message to the bot, then open https://api.telegram.org/bot<TOKEN>/getUpdates in your browser. The number is in "chat":{"id":...}. A separate bot for alerts is worth it, so a problem in the agent's bot doesn't silence the alerts too.
sudo cp my-agent-watchdog.service my-agent-watchdog.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now my-agent-watchdog.timer
sudo bash -c 'set -a; . /etc/my-agent/watchdog.env; DRY_RUN=1 /usr/local/bin/my-agent-watchdog'
# watchdog: all ok
systemctl list-timers my-agent-watchdog.timer --no-pager
# a line with NEXT less than 5 minutes away
sudo systemctl stop my-agent && sudo systemctl start my-agent-watchdog.service
# on Telegram: ⚠️ ...: my-agent.service is not running (inactive)
# the watchdog restarts the agent, and on the next run: ✅ ...: my-agent.service is running again
Step 5 (10 minutes): Memory: swap and a ceiling for heavy commands
Why: on a 2 to 4 GB server, one package install or one build can use up the memory. Without swap, the kernel kills a process, sometimes the agent. With swap and no ceiling, the server slows down until it stops answering. The fix: a little swap, and a hard ceiling for anything heavy.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
# /etc/sysctl.d/99-agent-swap.conf
# Use swap only when RAM is really short, instead of pushing the agent out early
vm.swappiness = 10
sudo cp 99-agent-swap.conf /etc/sysctl.d/ && sudo sysctl --system
Check: swapon --show shows /swapfile file 2G, and cat /proc/sys/vm/swappiness returns 10.
A ceiling for a heavy command: instead of running a build directly, run it inside a ceiling. If it goes over, only it is killed:
# as your user
systemd-run --user --scope -p MemoryMax=1500M -p MemorySwapMax=0 npm run build
# as root
sudo systemd-run --scope -p MemoryMax=1500M -p MemorySwapMax=0 apt-get -y upgrade
MemorySwapMax=0 means don't escape into swap: the command stops quickly instead of dragging the whole server down. The agent's service already has MemoryMax in its file. Pick a ceiling that leaves a few hundred MB for the rest of the system.
systemd-run --user --scope -p MemoryMax=200M -p MemorySwapMax=0 python3 -c "x = bytearray(400*1024*1024)"; echo "exit=$?"
# Killed
# exit=137 (and the server keeps working)
journalctl -k --since "7 days ago" | grep -iE 'out of memory|oom-kill|killed process'
journalctl -u my-agent --since "7 days ago" | grep -i oom
Any line here means lowering a ceiling, adding swap or upgrading the server.
Step 6 (10 minutes): Scheduled jobs without surprises
Why: a job that runs every morning and hangs, or two runs working on the same files at once, are problems that are hard to find. A systemd timer prevents most of them up front.
| cron | systemd timer | |
|---|---|---|
| Logs | You redirect them to a file yourself | Automatically in journalctl |
| A missed run (server was off) | Doesn't run | Persistent=true runs it after boot |
| Time and memory limits | None | TimeoutStartSec, MemoryMax |
| Result of the last run | You build it yourself | systemctl show -p Result, the watchdog checks it |
[Unit]
Description=My scheduled agent job
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
User=agent
Group=agent
WorkingDirectory=/opt/my-agent
EnvironmentFile=/etc/my-agent/agent.env
# flock -n: if the previous run still holds the lock, this run exits (code 75) instead of overlapping,
# and 75 counts as success, so a skipped run is not reported as a failure
ExecStart=/usr/bin/flock -n -E 75 /var/lib/my-agent/my-job.lock /opt/my-agent/job.sh
SuccessExitStatus=75
# Kill a run that hangs, so the next one can start
TimeoutStartSec=30min
MemoryMax=1200M
MemorySwapMax=0
[Unit]
Description=Run my agent job every weekday at 07:30
[Timer]
OnCalendar=Mon..Fri 07:30
# Set the server's time zone, or write the zone here, e.g. OnCalendar=Mon..Fri 07:30 Asia/Jerusalem
# Run a missed run after boot (the server was off at 07:30)
Persistent=true
RandomizedDelaySec=2min
[Install]
WantedBy=timers.target
flock -n makes sure two runs never overlap: if the previous one is still running, the new one exits at once. It exits with code 75, which is set as success, so a skipped run doesn't count as a failure.
sudo cp my-job.service my-job.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now my-job.timer
systemctl list-timers my-job.timer --no-pager
# NEXT: the next weekday at 07:30
sudo systemctl start my-job.service; journalctl -u my-job -n 20 --no-pager
# the job's output, ending with: Finished my-job.service
systemctl show -p Result --value my-job.service
# success
Prefer cron? At least with a lock and a log: 30 7 * * 1-5 flock -n /var/lib/my-agent/my-job.lock /opt/my-agent/job.sh >> /var/log/my-job.log 2>&1.
Step 7 (10 minutes): The agent itself: least privilege and approval for every outward action
Why: an AI agent runs commands based on text, and text can be planted (in an email, a web page, a message). The real boundary is what its user can do on the system, not what its instructions say.
- Its own user, no sudo. Done in step 3. The service only allows writes to
/opt/my-agentand/var/lib/my-agent(ProtectSystem,ReadWritePaths). - A list of allowed and forbidden commands. For a coding agent like Claude Code, a
.claude/settings.jsonfile in the project folder:
{
"permissions": {
"allow": [
"Bash(git status)",
"Bash(git diff *)",
"Bash(git log *)",
"Bash(npm test *)",
"Bash(npm run build *)"
],
"deny": [
"Bash(git push *)",
"Bash(gh pr merge *)",
"Bash(rm -rf *)",
"Bash(sudo *)",
"Bash(curl *)",
"Bash(ssh *)",
"Bash(scp *)",
"Read(./.env)",
"Read(./.env.*)",
"Read(//etc/my-agent/**)"
]
}
}
A list like this helps, but it's not a wall: the boundary is the user's permissions. That's why, even without the list, the agent user has no sudo and no access to the watchdog's and the backup's secret files.
- No outward action without approval. Sending email, posting, merging code, paying: the agent doesn't do it. It writes the action to a queue, and you approve it from your own terminal. The script refuses to run as the agent's user or without a terminal, runs the command with the agent's permissions, and logs every approval, rejection and result.
#!/usr/bin/env bash
# Human approval gate for outward actions (send email, post, merge, pay).
# The agent never runs these itself: it writes one file per action into the queue,
# /var/lib/my-agent/outbox/<id>.cmd (one command line)
# /var/lib/my-agent/outbox/<id>.why (one line: why)
# and you approve from your own shell: approve.sh (list)
# approve.sh <id> (show, ask, run, log)
# Install it so the agent cannot run it: sudo install -m 750 -o root -g sudo approve.sh /usr/local/bin/approve
set -euo pipefail
OUTBOX="${OUTBOX:-/var/lib/my-agent/outbox}"
AUDIT="${AUDIT:-/var/log/my-agent/audit.log}"
AGENT_USER="${AGENT_USER:-agent}"
RUN_AS="${RUN_AS:-$AGENT_USER}" # approved commands run with the agent's rights, not yours
mkdir -p "$OUTBOX" "$(dirname "$AUDIT")"
# The agent must never approve its own actions
if [[ "$(id -un)" == "$AGENT_USER" ]]; then echo "refused: the agent user cannot approve"; exit 3; fi
log() { printf '%s\t%s\t%s\t%s\n' "$(date -u +%FT%TZ)" "${SUDO_USER:-$USER}" "$1" "$2" >> "$AUDIT"; }
if [[ $# -eq 0 ]]; then
shopt -s nullglob
for f in "$OUTBOX"/*.cmd; do
id="$(basename "$f" .cmd)"
printf '%s\t%s\n' "$id" "$(cat "$OUTBOX/$id.why" 2>/dev/null || echo '-')"
done
exit 0
fi
id="$1"
[[ "$id" =~ ^[A-Za-z0-9_-]+$ ]] || { echo "bad id"; exit 2; }
cmd_file="$OUTBOX/$id.cmd"
[[ -f "$cmd_file" ]] || { echo "no such action: $id"; exit 2; }
echo "Why: $(cat "$OUTBOX/$id.why" 2>/dev/null || echo '-')"
echo "Command: $(cat "$cmd_file")"
# A person at a terminal types the answer; piped input is refused
[[ -t 0 || "${APPROVE_TEST_NO_TTY:-0}" == "1" ]] || { echo "refused: run this from a terminal"; exit 3; }
read -r -p "Run it? Type yes to approve: " answer
if [[ "$answer" != "yes" ]]; then
log "$id" "rejected: $(cat "$cmd_file")"
mv "$cmd_file" "$cmd_file.rejected"
echo "rejected"
exit 0
fi
log "$id" "approved: $(cat "$cmd_file")"
if [[ "$RUN_AS" == "$(id -un)" ]]; then runner=(bash -c); else runner=(sudo -u "$RUN_AS" bash -c); fi
if "${runner[@]}" "$(cat "$cmd_file")"; then log "$id" "done"; mv "$cmd_file" "$cmd_file.done"; else log "$id" "failed"; exit 1; fi
sudo install -m 750 -o root -g sudo approve.sh /usr/local/bin/approve
sudo install -d -o agent -g agent /var/lib/my-agent/outbox
sudo install -d -m 750 /var/log/my-agent
In the agent's instructions, write: "For any outward action, write it as one command line to /var/lib/my-agent/outbox/<id>.cmd, and the reason in one line to <id>.why. Don't run it yourself."
sudo -u agent sudo -n true
# sudo: a password is required (or: agent is not in the sudoers file)
sudo -u agent touch /etc/should-fail
# touch: cannot touch '/etc/should-fail': Permission denied
sudo approve # the actions waiting
sudo approve <id> # shows, asks, runs, logs
sudo tail -3 /var/log/my-agent/audit.log
Step 8 (15 minutes): An encrypted backup, and a restore you actually tested
Why: a backup you haven't restored from is a hope, not a backup. And a server that gets wiped takes the backup with it, if the backup was on it.
What to back up: config files (/etc/my-agent, the systemd units you wrote), the agent's folder and its data (/opt/my-agent, /var/lib/my-agent). The secret files go into the backup too, which is why the whole archive is encrypted.
sudo apt install -y rclone
sudo rclone config # create a remote called offsite at the storage provider you chose
sudo install -m 600 -o root -g root /dev/null /etc/my-agent/backup.pass
sudoedit /etc/my-agent/backup.pass # a long passphrase
Keep the passphrase off the server too (in a password manager). Without it, the backup can't be opened.
#!/usr/bin/env bash
# Nightly backup: configs, env files and data -> one encrypted archive -> off-server storage (rclone).
# Config: /etc/my-agent/backup.env (chmod 600): BACKUP_PATHS, RCLONE_REMOTE, PASSPHRASE_FILE, KEEP_DAYS
set -euo pipefail
BACKUP_PATHS="${BACKUP_PATHS:-/etc/my-agent /opt/my-agent /var/lib/my-agent /etc/systemd/system}"
RCLONE_REMOTE="${RCLONE_REMOTE:?set RCLONE_REMOTE, e.g. offsite:my-agent-backups}"
PASSPHRASE_FILE="${PASSPHRASE_FILE:-/etc/my-agent/backup.pass}"
KEEP_DAYS="${KEEP_DAYS:-30}"
WORK_DIR="${WORK_DIR:-/var/tmp/my-agent-backup}"
STAMP="$(date -u +%Y-%m-%dT%H%MZ)"
NAME="backup-${HOSTNAME:-host}-${STAMP}.tar.gz.gpg"
umask 077
mkdir -p "$WORK_DIR"
trap 'rm -f "$WORK_DIR/$NAME"' EXIT
# shellcheck disable=SC2086 # BACKUP_PATHS is a space-separated list on purpose
tar --create --gzip --file - --exclude='*.lock' --exclude='node_modules' --exclude='.cache' $BACKUP_PATHS 2>/dev/null |
gpg --batch --yes --quiet --symmetric --cipher-algo AES256 --passphrase-file "$PASSPHRASE_FILE" \
--output "$WORK_DIR/$NAME"
rclone copy --quiet "$WORK_DIR/$NAME" "$RCLONE_REMOTE/"
rclone delete --quiet --min-age "${KEEP_DAYS}d" "$RCLONE_REMOTE/"
echo "backup ok: $RCLONE_REMOTE/$NAME ($(du -h "$WORK_DIR/$NAME" | cut -f1))"
[Unit]
Description=Nightly encrypted backup of my AI agent
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/my-agent/backup.env
ExecStart=/usr/local/bin/my-agent-backup
TimeoutStartSec=1h
Nice=10
IOSchedulingClass=idle
[Unit]
Description=Nightly backup at 03:15
[Timer]
OnCalendar=*-*-* 03:15
Persistent=true
RandomizedDelaySec=10min
[Install]
WantedBy=timers.target
In /etc/my-agent/backup.env (also 600): RCLONE_REMOTE=offsite:my-agent-backups.
sudo install -m 755 backup.sh /usr/local/bin/my-agent-backup
sudo install -m 755 restore-test.sh /usr/local/bin/my-agent-restore-test
sudo cp my-agent-backup.service my-agent-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now my-agent-backup.timer
#!/usr/bin/env bash
# Restore test: download the newest backup, decrypt it, unpack it into a scratch folder and check it.
# Nothing on the live system is touched. Run it by hand once a month.
set -euo pipefail
RCLONE_REMOTE="${RCLONE_REMOTE:?set RCLONE_REMOTE}"
PASSPHRASE_FILE="${PASSPHRASE_FILE:-/etc/my-agent/backup.pass}"
CHECK_FILE="${CHECK_FILE:-etc/my-agent/agent.env}" # a file that must be in every backup (path without the leading /)
DEST="$(mktemp -d /var/tmp/restore-test.XXXXXX)"
trap 'rm -rf "$DEST"' EXIT
latest="$(rclone lsf --files-only "$RCLONE_REMOTE/" | grep '\.tar\.gz\.gpg$' | sort | tail -1)"
[[ -n "$latest" ]] || { echo "restore test FAILED: no backups found"; exit 1; }
rclone copy --quiet "$RCLONE_REMOTE/$latest" "$DEST/"
gpg --batch --quiet --decrypt --passphrase-file "$PASSPHRASE_FILE" "$DEST/$latest" | tar --extract --gzip --file - -C "$DEST"
[[ -s "$DEST/$CHECK_FILE" ]] || { echo "restore test FAILED: $CHECK_FILE missing in $latest"; exit 1; }
echo "restore test ok: $latest, $(find "$DEST" -type f | wc -l) files, $CHECK_FILE present"
sudo systemctl start my-agent-backup.service; journalctl -u my-agent-backup -n 3 --no-pager
# backup ok: offsite:my-agent-backups/backup-...tar.gz.gpg (...)
sudo bash -c 'set -a; . /etc/my-agent/backup.env; /usr/local/bin/my-agent-restore-test'
# restore test ok: backup-...tar.gz.gpg, ... files, etc/my-agent/agent.env present
A full restore to a new server: steps 1 and 5 on the new server, rclone config with the same remote, download the latest backup, then gpg --decrypt --passphrase-file ... backup.tar.gz.gpg | sudo tar -xz -C /, sudo systemctl daemon-reload, and enable the service and the timers. Do it once on a test server, so you know how long it takes.
The weekly checklist (10 minutes)
- This week's watchdog messages: every ⚠️ got a ✅ or was handled
systemctl --failedis emptyls /var/run/reboot-required: no file, or a reboot is scheduleddf -h /andfree -hhave roomjournalctl -k --since "7 days ago" | grep -i oomis emptysudo fail2ban-client status sshdworks- Last night's backup:
journalctl -u my-agent-backup -n 1 - Once a month:
my-agent-restore-testpasses sudo tail /var/log/my-agent/audit.log: every outward action was approved by you- Unused tokens are revoked
Full example: Noa Design Studio (a fictional business)
Everything in this example is fictional: the business, the server, the bot and the numbers.
Noa has a small Ubuntu server with 2 cores and 4 GB, running a Telegram bot that answers clients about availability, and a job every morning at 07:30 that prepares a summary of yesterday's inquiries.
Before the kit: the bot ran in an open SSH window. After a provider update and reboot, the bot didn't come back, and Noa found out only when a client asked why nobody answered.
After:
- Key-only login,
ufwopen only to 22, andfail2banalready blocked a few addresses in the first week. - The bot runs as the
my-agentservice under theagentuser, and touches the heartbeat after every successful Telegram call. - The watchdog runs every 5 minutes. On Tuesday night this arrived:
⚠️ studio-vps: no heartbeat from the agent for over 15 minutes
✅ studio-vps: the agent's heartbeat is fresh again
The bot got stuck after a network drop, the watchdog restarted it once, and within 5 minutes the recovery message came. Noa didn't have to do anything, and in the morning she saw what had happened.
- The 07:30 job runs as a timer with
flock. On the day the server rebooted at 07:20,Persistent=trueran it as soon as it was back. - The bot offers clients to "send a quote", but doesn't send it. It writes the email to the queue, and Noa approves with
sudo approve. The log shows every approval. - The backup runs at 03:15 to a bucket off the server. The first restore test showed that the bot's database folder wasn't on the list. It was added to
BACKUP_PATHS, and the next test passed.
Disclaimer
This content is general recommendations only, not professional, legal, accounting or tax advice. Every business and organization is different. You are responsible for checking that it fits your situation, and for checking the data, prompts and code before you use them. Outside tools, prices and terms of service change, so check them at the source. Test every change on a non-production server first. Keep console access from your server provider, in case you lock yourself out of SSH. For sensitive systems, or when the server holds client data, consult a security professional.
What's next
Here you set up and maintain everything yourself. You can get a server set up this way from the start, with the watchdog, backups, the approval queue and monitoring, and someone who answers when the alert arrives.
Want this running automatically in your business? aviad@gindiinnovation.com · gindiinnovation.com