Update Podman deployment for rootless support and DNS fixes

- Document rootless Podman port binding limitations and workarounds in README.
- Update `.env.sample` with notes and default high ports for rootless runs.
- Adjust `compose.yml` for network configuration and privileged port handling.
- Introduce fixes for Traefik DNS timeouts using explicit public resolvers and network tweaks.
- Switch MinIO and MinIO setup to use the latest images for better compatibility.

deploy/podman/.env.sample

@@ -32,5 +32,8 @@ MINIO_BUCKET=
 TRAEFIK_DASHBOARD_AUTH=
 
 # Host ports to bind Traefik
-TRAEFIK_HTTP_PORT=80
+# Note: Rootless Podman cannot bind privileged ports (<1024). For rootless runs, use high ports
-TRAEFIK_HTTPS_PORT=443
+# like 8080/4443, or configure the host sysctl net.ipv4.ip_unprivileged_port_start=80 (requires root)
+# to allow binding 80/443.
+TRAEFIK_HTTP_PORT=8080
+TRAEFIK_HTTPS_PORT=4443

deploy/podman/README.md

@@ -5,9 +5,21 @@ This stack deploys Solstice CI services behind Traefik with automatic TLS certif
 Prerequisites
 - Podman 4.9+ with podman-compose compatibility (podman compose)
 - Public DNS records for subdomains pointing to the host running this stack
-- Ports 80 and 443 open to the Internet
+- Ports 80 and 443 open to the Internet (for ACME HTTP-01), see Rootless note below
 - Email address for ACME registration
 
+Rootless Podman note (ports 80/443)
+- Rootless Podman cannot bind privileged ports (<1024). If you run this stack rootless, set high host ports in .env:
+  - TRAEFIK_HTTP_PORT=8080
+  - TRAEFIK_HTTPS_PORT=4443
+- With high ports, public HTTPS will be served on 4443 and the ACME HTTP-01 challenge will not work unless you forward external port 80 to host 8080 (e.g., via a firewall/NAT) or place another reverse proxy in front.
+- To use real public certificates with HTTP-01 directly on this host, either:
+  - Run Podman as root (rootful) for Traefik only, or
+  - Allow unprivileged port binding for your kernel by setting (requires root):
+    sysctl -w net.ipv4.ip_unprivileged_port_start=80
+    and add net.ipv4.ip_unprivileged_port_start=80 to /etc/sysctl.conf to persist.
+- Alternatively, switch Traefik to a DNS-01 challenge (not configured here) if you control DNS.
+
 DNS
 Create A/AAAA records for the following hostnames under your base domain (no environment in hostname; env separation is logical via DB/vhost/buckets):
 - traefik.svc.DOMAIN
@@ -69,3 +81,10 @@ Troubleshooting
 - Certificate issues: check Traefik logs; verify DNS and ports 80/443. For testing, use ACME staging server.
 - No routes: verify labels on services and that traefik sees the podman socket.
 - Healthchecks failing: inspect service logs with podman logs <container>.
+- Arch Linux/Podman DNS timeouts (ACME): If Traefik logs show errors like "dial tcp: lookup acme-v02.api.letsencrypt.org on 10.89.0.1:53: i/o timeout", this is typically a Podman network DNS (netavark/aardvark-dns) issue. Fixes:
+  - We now set explicit public DNS resolvers for the Traefik container in compose.yml (1.1.1.1, 8.8.8.8, 9.9.9.9). Redeploy: podman compose up -d traefik.
+  - Ensure Podman’s network backend and DNS are installed and active (Arch): pacman -S netavark aardvark-dns; systemctl enable --now aardvark-dns.socket; verify `podman info | grep -i network` shows networkBackend: netavark.
+  - Alternatively, mount the host resolv.conf into Traefik: add to the traefik service volumes: - /etc/resolv.conf:/etc/resolv.conf:ro
+  - Check firewall (nftables): allow UDP/TCP 53 from the Podman bridge (e.g., 10.89.0.0/24) to host 10.89.0.1; allow FORWARD for ESTABLISHED,RELATED.
+  - Inspect network: podman network inspect podman; consider creating a custom network with explicit DNS servers: podman network create --dns 1.1.1.1 --dns 8.8.8.8 solstice-net and set networks.core.name to that network in compose.yml.
+  - As a last resort, run Traefik with host networking: network_mode: host (then remove ports and ensure only Traefik is exposed), or switch ACME to DNS-01.

deploy/podman/compose.yml

@@ -16,8 +16,7 @@
 name: solstice-ci
 
 networks:
-  core:
+  core: {}
-    driver: bridge
 
 volumes:
   traefik-acme:
@@ -45,10 +44,12 @@ services:
       # Optional: override ACME CA server via .env (e.g., staging URL)
       - --certificatesresolvers.le.acme.caserver=${TRAEFIK_ACME_CASERVER}
     ports:
+      # Rootless Podman cannot bind privileged ports (<1024). Use high ports via .env (e.g., 8080/4443),
+      # or adjust sysctl on the host: net.ipv4.ip_unprivileged_port_start=80 (requires root).
       - ${TRAEFIK_HTTP_PORT:-80}:80
       - ${TRAEFIK_HTTPS_PORT:-443}:443
     volumes:
-      - /var/run/podman/podman.sock:/var/run/docker.sock:Z
+      - /var/run/docker.sock:/var/run/docker.sock:Z
       - traefik-acme:/acme
     networks:
       - core
@@ -125,7 +126,7 @@ services:
       - traefik.http.services.mq.loadbalancer.server.port=15672
 
   minio:
-    image: quay.io/minio/minio:RELEASE.2025-02-07T22-39-53Z
+    image: quay.io/minio/minio:latest
     container_name: solstice-minio
     restart: unless-stopped
     command: server /data --console-address ":9001"
@@ -157,7 +158,7 @@ services:
       - traefik.tcp.services.s3.loadbalancer.server.port=9000
 
   minio-setup:
-    image: quay.io/minio/mc:RELEASE.2025-02-07T22-47-51Z
+    image: quay.io/minio/mc:latest
     container_name: solstice-minio-setup
     depends_on:
       minio: