A private Nextcloud on NVMe with object storage behind it

What this builds

Two machines. A Ryzen instance runs Nextcloud, PHP-FPM, Postgres and Redis, all on local Gen4 NVMe, which is where the latency-sensitive work lives. The storage node beside it holds the actual file content in an object bucket, which is where the cheap terabytes live. Between them sits a WireGuard tunnel, so nothing in the arrangement is exposed to anything else.

The split matters because Nextcloud with a primary object store stops writing user data to the local disk entirely. Your NVMe holds the database and the code; the bucket holds every byte anyone uploads. Growing from two hundred gigabytes to twenty terabytes then becomes a storage-node upgrade rather than a migration.

Before you start

• An R-8 for the application and an S-50 for the bucket, ideally in the same city so the tunnel round-trip is under a millisecond.
• A tunnel between them. The WireGuard guide builds exactly this; use 10.9.0.1 for the app and 10.9.0.2 for the storage node.
• A hostname pointing at the app instance, cloud.example.com below.

1. Object storage on the storage node

MinIO is a single binary and needs nothing else. Fetch it, verify it, install it:

cd /usr/local/bin
wget https://dl.min.io/server/minio/release/linux-amd64/minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio.sha256sum
sha256sum -c minio.sha256sum
chmod +x minio
useradd -r -s /sbin/nologin minio
install -d -o minio -g minio /srv/objects

The bulk tier on a storage node is mounted separately from the NVMe. Put the bucket on the bulk tier and leave the NVMe for the write cache, which is what it is there for. Then the unit, at /etc/systemd/system/minio.service:

[Unit]
Description=MinIO
After=network-online.target [email protected]
Wants=network-online.target

[Service]
User=minio
Group=minio
Environment=MINIO_ROOT_USER=nextcloud
Environment=MINIO_ROOT_PASSWORD=<forty random characters>
ExecStart=/usr/local/bin/minio server /srv/objects --address 10.9.0.2:9000
Restart=always
LimitNOFILE=65535
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/srv/objects

[Install]
WantedBy=multi-user.target

Binding to the tunnel address rather than to everything is the important line. The object store has no business answering on the public interface, and this way a firewall mistake cannot expose it.

systemctl daemon-reload && systemctl enable --now minio
ss -ltn | grep 9000

2. Application host: packages

On the Ryzen instance:

apt update && apt install -y nginx postgresql redis-server unzip \
  php8.4-fpm php8.4-pgsql php8.4-gd php8.4-curl php8.4-zip php8.4-xml \
  php8.4-mbstring php8.4-intl php8.4-bcmath php8.4-gmp php8.4-imagick \
  php8.4-redis php8.4-apcu

3. Database

sudo -u postgres psql -c "CREATE USER nextcloud WITH PASSWORD '<a long password>';"
sudo -u postgres psql -c "CREATE DATABASE nextcloud OWNER nextcloud TEMPLATE template0 ENCODING UTF8;"

Default Postgres settings are fine at this scale. If this Nextcloud ends up serving a few hundred people, the Postgres tuning guide applies and the numbers in it are worth an hour.

4. Nextcloud, configured before installation

cd /var/www
wget https://download.nextcloud.com/server/releases/latest.zip
unzip -q latest.zip && rm latest.zip
chown -R www-data:www-data /var/www/nextcloud

Now the part that has to happen before the installer runs, because a primary object store cannot be added to an existing instance without moving every file by hand. Create /var/www/nextcloud/config/config.php:

<?php
$CONFIG = array(
  "objectstore" => array(
    "class" => "\\OC\\Files\\ObjectStore\\S3",
    "arguments" => array(
      "bucket" => "nextcloud",
      "autocreate" => true,
      "key" => "nextcloud",
      "secret" => "<the same forty characters>",
      "hostname" => "10.9.0.2",
      "port" => 9000,
      "use_ssl" => false,
      "use_path_style" => true,
      "region" => "local",
    ),
  ),
);

use_ssl is false on purpose. The traffic crosses a WireGuard tunnel that is already authenticated and encrypted, and terminating TLS a second time on the storage node buys nothing except CPU cycles and a certificate to forget to renew. Should the two machines ever sit in different cities, turn it on.

chown www-data:www-data /var/www/nextcloud/config/config.php
chmod 640 /var/www/nextcloud/config/config.php
cd /var/www/nextcloud
sudo -u www-data php occ maintenance:install --database pgsql \
  --database-name nextcloud --database-user nextcloud \
  --database-pass "<a long password>" \
  --admin-user admin --admin-pass "<another long password>"

5. Cache, cron and the last settings

sudo -u www-data php occ config:system:set memcache.local --value "\\OC\\Memcache\\APCu"
sudo -u www-data php occ config:system:set memcache.distributed --value "\\OC\\Memcache\\Redis"
sudo -u www-data php occ config:system:set memcache.locking --value "\\OC\\Memcache\\Redis"
sudo -u www-data php occ config:system:set redis host --value /run/redis/redis-server.sock
sudo -u www-data php occ config:system:set redis port --value 0
sudo -u www-data php occ config:system:set trusted_domains 0 --value cloud.example.com
sudo -u www-data php occ config:system:set overwrite.cli.url --value https://cloud.example.com
sudo -u www-data php occ background:cron

Redis over a unix socket rather than TCP saves a syscall round trip on every lock operation, and Nextcloud takes a lot of locks. Add www-data to the redis group so it can reach the socket, then wire the background job to a timer:

usermod -aG redis www-data
systemctl restart php8.4-fpm redis-server
echo "*/5 * * * * php -f /var/www/nextcloud/cron.php" | crontab -u www-data -

PHP-FPM defaults assume a shared machine. Yours is not shared, so in /etc/php/8.4/fpm/pool.d/www.conf set pm = dynamic, pm.max_children = 60, pm.start_servers = 12, pm.min_spare_servers = 8 and pm.max_spare_servers = 20, and raise memory_limit to 1G in /etc/php/8.4/fpm/php.ini. Uploads want upload_max_filesize and post_max_size at 16G unless you enjoy explaining why a video failed at ninety percent.

Verify it

The point of this build is that user files are not on the application server. Prove that, in three commands.

sudo -u www-data php occ status
du -sh /var/www/nextcloud/data

Status should report installed and not in maintenance mode. The data directory should be a handful of megabytes and stay that way forever; it holds logs and nothing else once object storage is primary.

Now upload something and watch where it lands:

head -c 50M /dev/urandom > /tmp/test.bin
curl -u admin -T /tmp/test.bin https://cloud.example.com/remote.php/dav/files/admin/test.bin
du -sh /var/www/nextcloud/data

The upload succeeds, and the data directory has not grown by fifty megabytes. On the storage node, the object is visible directly:

ls -R /srv/objects/nextcloud | tail -5
du -sh /srv/objects/nextcloud

Objects are named urn:oid: followed by the file id, which is why the bucket is not a browsable copy of the user’s folders and why you back up the database and the bucket together or not at all.

Afterwards

Take the off-node backup add-on for the Postgres dump and the bucket, or run your own to a third site. A bucket without its database is a pile of numbered blobs, and a database without its bucket is an index of files that no longer exist. Losing either loses both.