Building a Self-Hosted Package Mirror for Air-Gapped and Hardened Environments

Air-gapped networks, compliance-mandated environments, and bandwidth-constrained sites all share a common need: internal package mirrors. When your servers cannot reach the internet — or when you need to control exactly which packages are available — a self-hosted mirror is infrastructure you cannot skip.

This guide covers building a complete Debian/Ubuntu package mirror using apt-mirror, serving it with nginx, automating synchronization, distributing GPG keys, and deploying client configuration at scale with Puppet or Ansible.

Why Mirror Internally

The reasons break down into four categories:

Air-gapped compliance. FedRAMP, CMMC, PCI-DSS, and STIG baselines often require that production systems cannot reach the public internet. Packages must come from a controlled internal source.

Supply chain control. When you mirror packages locally, you create a buffer between upstream and your fleet. If a package is compromised upstream, you can freeze your mirror, audit the change, and decide whether to sync. You are not automatically pulling untrusted code onto production systems.

Bandwidth savings. A fleet of 200 servers all pulling the same 500 MB security update from the internet wastes bandwidth and hammers your egress link. An internal mirror downloads once and serves locally at wire speed.

Reliability. External mirrors go down, change URLs, or drop old releases. An internal mirror ensures your package management always works, even if upstream is having a bad day.

Setting Up apt-mirror

Install apt-mirror on a dedicated server (we will call it mirror01):

sudo apt-get install apt-mirror

Configure /etc/apt/mirror.list:

############ apt-mirror configuration ############
set base_path    /srv/apt-mirror
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var
set nthreads     20
set _tilde       0

# Ubuntu 22.04 LTS (Jammy)
deb http://archive.ubuntu.com/ubuntu jammy           main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu jammy-updates    main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu jammy-security   main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu jammy-backports  main restricted universe multiverse

# Debian 12 (Bookworm)
deb http://deb.debian.org/debian bookworm            main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates     main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

# Optional: source packages (doubles disk usage)
# deb-src http://archive.ubuntu.com/ubuntu jammy main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu
clean http://deb.debian.org/debian
clean http://security.debian.org/debian-security

Disk Space Planning

A full mirror of Ubuntu Jammy (main, restricted, universe, multiverse) for amd64 alone consumes approximately 200-250 GB. Add Debian Bookworm and you are looking at 400+ GB. Including multiple architectures or source packages can double or triple this.

Plan for at least 1 TB of storage for a moderate mirror setup, and use a dedicated filesystem so mirror growth never fills your root partition:

sudo mkdir -p /srv/apt-mirror
# Ideally mount a dedicated volume here
sudo chown apt-mirror:apt-mirror /srv/apt-mirror

Initial Sync

The first sync downloads everything and takes hours depending on your bandwidth:

sudo -u apt-mirror apt-mirror

Monitor progress by watching the var directory:

tail -f /srv/apt-mirror/var/cron.log

Serving the Mirror with nginx

Install nginx on mirror01 and configure it to serve the mirror directory:

# /etc/nginx/sites-available/apt-mirror
server {
    listen 80;
    listen [::]:80;
    server_name mirror01.internal.example-corp.com;

    root /srv/apt-mirror/mirror;
    autoindex on;

    # Optimize for large file serving
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # Allow large package downloads
    client_max_body_size 0;

    # Cache control for metadata vs packages
    location ~ /(InRelease|Release|Release.gpg|Packages.gz|Sources.gz) {
        expires 1h;
        add_header Cache-Control "public, must-revalidate";
    }

    location ~ .deb$ {
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    # Access logging for audit
    access_log /var/log/nginx/apt-mirror-access.log;
    error_log /var/log/nginx/apt-mirror-error.log;
}

Enable and start:

sudo ln -s /etc/nginx/sites-available/apt-mirror /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

For air-gapped environments where TLS is required internally, add a self-signed or internal CA certificate:

server {
    listen 443 ssl;
    server_name mirror01.internal.example-corp.com;

    ssl_certificate     /etc/ssl/internal/mirror01.crt;
    ssl_certificate_key /etc/ssl/internal/mirror01.key;

    root /srv/apt-mirror/mirror;
    autoindex on;
    # ... same location blocks as above
}

GPG Key Distribution

Clients need the repository signing keys to verify package authenticity. For standard Ubuntu/Debian mirrors, the keys come from the upstream keyring packages. But you need to ensure they are distributed to all clients.

# On mirror01, export the keys clients need
apt-key exportall > /srv/apt-mirror/mirror/repo-keys.gpg

# Or for modern apt (2.4+), place keys in /etc/apt/keyrings/
# and reference them in sources.list with signed-by=

For Puppet-managed fleets:

# modules/apt_mirror/manifests/keys.pp
class apt_mirror::keys {
  file { '/etc/apt/keyrings/ubuntu-archive.gpg':
    ensure => file,
    source => 'puppet:///modules/apt_mirror/ubuntu-archive.gpg',
    mode   => '0644',
  }

  file { '/etc/apt/keyrings/debian-archive.gpg':
    ensure => file,
    source => 'puppet:///modules/apt_mirror/debian-archive.gpg',
    mode   => '0644',
  }
}

Automating Mirror Syncs

Set up a cron job to sync daily during off-peak hours:

# /etc/cron.d/apt-mirror
0 2    apt-mirror /usr/bin/apt-mirror >> /var/log/apt-mirror/sync.log 2>&1

#!/bin/bash
# /usr/local/bin/mirror-sync.sh
LOG="/var/log/apt-mirror/sync-$(date +%Y%m%d).log"
ALERT_EMAIL="[email protected]"

echo "=== Mirror sync started: $(date) ===" >> "$LOG"

/usr/bin/apt-mirror >> "$LOG" 2>&1
STATUS=$?

if [ $STATUS -ne 0 ]; then
    echo "Mirror sync FAILED with exit code $STATUS" | 
        mail -s "[ALERT] apt-mirror sync failure on mirror01" "$ALERT_EMAIL"
fi

# Log disk usage
echo "Disk usage: $(du -sh /srv/apt-mirror/mirror)" >> "$LOG"
echo "=== Mirror sync completed: $(date) ===" >> "$LOG"

# Check for stale InRelease files (older than 48 hours)
find /srv/apt-mirror/mirror -name "InRelease" -mmin +2880 | while read f; do
    echo "STALE: $f ($(stat -c %y "$f"))" >> "$LOG"
done

# /etc/apt/sources.list
deb [signed-by=/etc/apt/keyrings/ubuntu-archive.gpg] http://mirror01.internal.example-corp.com/archive.ubuntu.com/ubuntu jammy main restricted universe multiverse
deb [signed-by=/etc/apt/keyrings/ubuntu-archive.gpg] http://mirror01.internal.example-corp.com/archive.ubuntu.com/ubuntu jammy-updates main restricted universe multiverse
deb [signed-by=/etc/apt/keyrings/ubuntu-archive.gpg] http://mirror01.internal.example-corp.com/archive.ubuntu.com/ubuntu jammy-security main restricted universe multiverse

# modules/apt_mirror/manifests/client.pp
class apt_mirror::client (
  String $mirror_host = 'mirror01.internal.example-corp.com',
) {
  file { '/etc/apt/sources.list':
    ensure  => file,
    content => template('apt_mirror/sources.list.erb'),
    mode    => '0644',
    notify  => Exec['apt-update'],
  }

  exec { 'apt-update':
    command     => '/usr/bin/apt-get update',
    refreshonly => true,
  }
}

# roles/apt_mirror_client/tasks/main.yml
name: Configure apt sources to use internal mirror
  template:
    src: sources.list.j2
    dest: /etc/apt/sources.list
    mode: '0644'
  notify: Update apt cache

name: Install internal CA certificate
  copy:
    src: internal-ca.crt
    dest: /usr/local/share/ca-certificates/internal-ca.crt
    mode: '0644'
  notify: Update CA certificates

handlers:
  - name: Update apt cache
    apt:
      update_cache: yes

  - name: Update CA certificates
    command: update-ca-certificates

# Verify the mirror is being used
apt-get update 2>&1 | grep mirror01

# Check that package signatures validate
apt-get install --simulate some-package
# Look for "WARNING" or "UNAUTHENTICATED" — there should be none

# Verify a specific package's integrity
apt-get download coreutils
dpkg-sig --verify coreutils_.deb

Monitoring Mirror Freshness

A stale mirror is worse than no mirror — it gives a false sense of security while missing critical updates.

#!/bin/bash
# /usr/local/bin/check-mirror-freshness.sh
MAX_AGE_HOURS=48

find /srv/apt-mirror/mirror -name "InRelease" | while read f; do
    AGE=$(( ($(date +%s) - $(stat -c %Y "$f")) / 3600 ))
    if [ $AGE -gt $MAX_AGE_HOURS ]; then
        echo "CRITICAL: $f is ${AGE}h old (threshold: ${MAX_AGE_HOURS}h)"
    fi
done

Feed this into your monitoring system (Nagios, Zabbix, Prometheus node_exporter textfile collector) to alert when syncs fail.

Troubleshooting Common Issues

404 errors after sync: Incomplete syncs leave partial metadata. Check /var/log/apt-mirror/sync.log for download errors. Re-run apt-mirror and ensure it completes fully.

Permission errors: apt-mirror runs as the apt-mirror user. Ensure the entire /srv/apt-mirror tree is owned by this user. After manual fixes, run chown -R apt-mirror:apt-mirror /srv/apt-mirror.

Stale InRelease files: If a sync fails midway, old InRelease files remain. Clients will reject packages if the InRelease file is older than the Valid-Until date. Fix by completing a full sync.

Disk space exhaustion: The clean directives in mirror.list tell apt-mirror to remove packages no longer referenced by the index. If you forget these, old package versions accumulate indefinitely. Run /var/spool/apt-mirror/var/clean.sh manually if needed.

Hash sum mismatch on clients: Usually means the mirror was being synced while the client ran apt-get update. Schedule syncs during maintenance windows, or use a two-stage sync (sync to staging directory, then atomic swap).