The product is named Cloudfield. Its commands, variables, paths and hostnames,
and what the apps and the CLI print, still say aether or Aether until later
releases.
This guide installs, upgrades and removes Cloudfield on every platform. For the shortest path to a first session, follow quickstart.md, then come back here for other platforms, options and errors. Cloudfield's terms are defined in the glossary.
What gets installed where
Cloudfield has two parts. The server, aether-server, runs on one Linux
machine: it keeps your repositories and runs each agent in its own Docker
container or virtual machine. A
client is any machine you work from: the aether CLI, the desktop app and the
Android app all connect to that server.
| Component | Sessions on | What it is | Install it with |
|---|---|---|---|
aether-server |
Linux, x64 and arm64 | The server. Needs Docker. | The install script, answering server |
aether |
Linux, macOS and Windows, x64 and arm64 | The CLI. aether gui also serves the dashboard on your machine. |
The install script or the Windows install script |
| Desktop app | Linux, macOS, Windows | A window around the dashboard. Needs the CLI. | aether gui build |
| Android app | Android | The dashboard on a phone. | aether-android.apk |
Clients reach the server over SSH, port 2222 by default. Without a direct or
Tailscale route they go through the edge, a relay at https://edge.onaether.dev
that both sides dial out to after a GitHub sign-in (edge.md).
flowchart LR
laptop["Your machine<br/>CLI, desktop app"] -- "SSH, direct or over Tailscale" --> server["Linux server<br/>aether-server"]
phone["Phone<br/>Android app"] -- "HTTPS over Tailscale" --> server
laptop & phone -- "GitHub sign-in" --> edge["Edge relay"]
edge -- "relays SSH" --> server
server -- "one container per session" --> docker["Docker"]Server prerequisites
- Linux; a headless server is fine. Windows and macOS run only clients.
- Docker Engine, running
(install Docker Engine). Every session
and member environment is a container, unless they run in
VMs. The snap package does not work: the
aether-serverunit requiresdocker.service. Sharing an agent account between members (teams.md) needs Docker Engine 26.0 or newer;docker version --format '{{.Server.Version}}'prints yours. - git, and
ssh-keygen(packageopenssh-clienton Debian and Ubuntu), which git uses to sign the commits Cloudfield makes for members who connected GitHub. - At least 2.5 GiB of available memory and 6 GiB of free disk, plus disk for
the images the server pulls on first use. A session starts only while 2 GiB of
memory (a tenth of RAM above 20 GiB) plus 512 MiB per session starting stays
available, and while the data and Docker filesystems keep their reserve (5%
of the filesystem, at least 5 GiB, at most 20 GiB;
min-free-diskoverrides it) plus 1 GiB per starting session free (environments.md).aether-server installwarns below this. - Optional: Sysbox, so agents can run Docker inside their sessions. Setup offers it on Ubuntu and Debian (environments.md).
- Optional: sessions in virtual machines instead of containers need an x86-64
host with
/dev/kvm(Hosts for VM sessions). - Optional: Tailscale. Without it the server needs only outbound HTTPS to the edge (networking.md).
- Optional: dictation, which transcribes speech on the server in a Docker
container, needs 690 MB of disk plus a 100 MB image, and 2 GiB of available
memory while it runs; install it with
sudo aether-server dictation install(dictation.md).
Hosts for VM sessions
With session-runtime vm, each session runs in its own Cloud Hypervisor
virtual machine (environments.md).
The host must be x86-64 and expose /dev/kvm: bare metal, or a cloud VM with
nested virtualization. Install a server for it with:
sudo aether-server install --session-runtime vm # a new server
sudo aether-server vm install # a server already installedBoth first open /dev/kvm and stop with the error when they cannot, such as
session VMs need KVM: open /dev/kvm: no such file or directory; install
then writes nothing. They install qemu-img, mkfs.ext4,
/usr/libexec/virtiofsd, iptables, nft and ip with apt-get when
missing (Debian 12 or Ubuntu 24.04 and newer ship virtiofsd at that path), Cloud Hypervisor
v54.1 at /usr/local/bin/cloud-hypervisor after checking its SHA-256, the
oh-my-pi release whose auth broker holds session VMs' omp logins at
/usr/local/libexec/aether/omp after checking its SHA-256
(harnesses.md), and the
release's guest kernel aether-vmlinux-x86_64 at
/usr/local/lib/aether/vmlinux after checking it against the signed release
manifest, noting the release in vmlinux.release beside it. Nothing is
replaced until every download has checked out. A vm-kernel set to another
path, or a development build, keeps the kernel already there. With
session-runtime vm the server manages /usr/local/bin/cloud-hypervisor and
/usr/local/libexec/aether/omp: replace either, and the next start puts the
pinned one back, the omp in the background after the server is up. The server
also adds the system user aether-omp that omp's auth brokers run as.
install leaves
Sysbox out: sessions and environment terminals run their own Docker daemon in
their VM.
On AWS:
Use an Intel instance from the M7i, C7i, R7i, M8i, C8i or R8i families, which support nested virtualization, or any
.metalinstance (EC2 nested virtualization). The M8id, C8id and R8id variants add local NVMe, where a sleeping session's saved memory wakes fastest.Turn nested virtualization on at launch with
--cpu-options NestedVirtualization=enabled, or on a stopped instance withaws ec2 modify-instance-cpu-options --nested-virtualization enabled.Keep the instance metadata hop limit at 1 (
--metadata-options HttpTokens=required,HttpPutResponseHopLimit=1), so the instance's credentials never answer a VM, whose traffic takes one more hop.Format the local NVMe as XFS and mount the data directory on it before installing (
lsblknames the device,Amazon EC2 NVMe Instance Storage):CODEsudo mkfs.xfs /dev/nvme1n1 sudo mkdir -p /var/lib/aether sudo mount /dev/nvme1n1 /var/lib/aetherXFS clones package-cache disks without copying them. VM disks, checkouts and the image store must share one filesystem, because they hard-link each other. Instance storage is lost when the instance stops or fails, so keep the backups current, or keep the data directory on EBS and accept slower wakes.
Nitro Sandboxes do not fit: they cannot snapshot, pause or resume, which saving an idle session needs, and a sandbox's memory is fixed for its lifetime, so every session would hold the memory of its busiest moment.
The install script
On Linux or macOS:
curl -fsSL https://raw.githubusercontent.com/3xDevOps/cloudfield/main/scripts/install.sh | shThe script needs curl, and sha256sum or shasum. It downloads the
binaries for your OS and CPU, checks each one before replacing anything, then
asks What is this machine?:
| Answer | Installs | Into | Then runs |
|---|---|---|---|
server |
aether-server and aether |
/usr/local/bin, using sudo; ~/.local/bin on a machine without sudo |
sudo aether-server setup (First boot) |
client |
aether |
~/.local/bin |
aether gui build (Desktop app) |
none |
The same as server |
/usr/local/bin |
Nothing |
macOS gets only aether. Enter takes server on Linux without a desktop
session (no DISPLAY or WAYLAND_DISPLAY), and client on a Linux desktop
and on macOS; with no terminal, as in CI, the script acts like --role none. A
client goes in ~/.local/bin so the dashboard's Update now can replace it.
If the directory is not on your PATH, the script prints the line to add (it
never edits a profile), and it prints the sudo rm -f for an older aether
in /usr/local/bin that would run first.
--role serverwith no terminal installs the binaries, then exits 1 with the command that sets the server up from flags (First boot):sudo aether-server install --edge-url https://edge.onaether.dev --edge-access <account|approved-devices>.When
sudofails, for example because it needs a password and there is no terminal, the script exits 1. If it was installingaetheralone, as on macOS or with--client, it names the install that needs nosudo:CODEcurl -fsSL https://raw.githubusercontent.com/3xDevOps/cloudfield/main/scripts/install.sh | sh -s -- --client --role none --bin-dir "$HOME/.local/bin"Otherwise it says that installing
aether-serverneeds root, passwordlesssudo, or a terminal wheresudocan ask for the password, such asssh -t <user>@<host> '<install command>'.When setup does not run on a host whose
aether-serverunit already exists,next:sayssudo aether-server update, which restarts the running server onto the new binary.When the desktop build fails, the script still exits 0 and its last line says the CLI is installed and the app is not, with the exit status of
aether gui build.
| Flag | Variable | Effect |
|---|---|---|
--version <tag> |
AETHER_VERSION |
Install exactly this release, or stop. |
--bin-dir <dir> |
AETHER_BIN_DIR |
Install into this directory, whatever the role. |
--client or --server |
AETHER_COMPONENTS=client or server |
Install only aether, or only aether-server, and answer to match. |
--role <role> |
AETHER_ROLE |
Answer the question in advance: server, client or none. |
AETHER_REPO, AETHER_BASE_URL |
Download from a fork, or release assets from an HTTPS mirror. |
Flags passed through a pipe need sh -s --, as in
curl -fsSL <install.sh URL> | sh -s -- --client --bin-dir ~/bin.
Without --version, the script installs the newest release that has
published checksums.txt, the signed manifest (aether-release.json and
aether-release.json.sig) and every binary you need. While the latest one is
still uploading or has failed, it takes the one before and says why.
--version <tag> installs exactly that tag, prereleases included, or stops.
With OpenSSL 3 on PATH the script verifies the signature; without it (macOS
ships LibreSSL) it warns and checks SHA-256 and size only. Any failure stops
the install before an installed binary is replaced, so rerunning is safe.
Windows install script
In Windows PowerShell 5.1 or PowerShell 7, download the installer, review it, then run it:
Invoke-WebRequest -UseBasicParsing -Uri https://raw.githubusercontent.com/3xDevOps/cloudfield/main/scripts/install.ps1 -OutFile "$env:TEMP\aether-install.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\aether-install.ps1"Windows runs only the client, so the script asks nothing. It downloads the
x64 or ARM64 CLI from the newest complete release, checks its SHA-256 and size
against checksums.txt and aether-release.json, installs it as
%LOCALAPPDATA%\Programs\Aether\aether.exe, and adds that directory to your
user PATH. Then it runs aether gui build, which installs the
desktop app and adds Aether to the Start Menu. It does not
ask for administrator rights or change the execution policy or Defender
settings: -ExecutionPolicy Bypass applies to that one process only. Open a
new terminal afterwards: open ones keep their old PATH.
It takes -Version <tag>, -BinDir <dir> and -Role client|none (none
installs only the CLI), or AETHER_VERSION, AETHER_BIN_DIR, AETHER_ROLE,
AETHER_REPO and an https AETHER_BASE_URL. The desktop app needs
v0.5.1-alpha.4 or newer, so an older -Version needs -Role none.
PowerShell cannot check the manifest's signature, so the script checks SHA-256
and size only.
To reinstall or upgrade, close Cloudfield and rerun the script with the same
-BinDir. If Cloudfield still runs, the script prints the Stop-Process -Id <pids> (and schtasks /End /TN aether-daemon) to run, and changes nothing.
If only the desktop step fails, it prints the gui build command to rerun.
The Windows client
Windows runs only the client. It keeps its config in %APPDATA%\aether\ and
uses the keys in %USERPROFILE%\.ssh\. Differences from Linux and macOS:
aether initrefuses to run. It prepares a server data directory, so run it on the Linux server.scripts/install.shdoes not run on Windows. Useinstall.ps1.aether updateworks, but Windows cannot overwrite a runningaether.exe: the update renames it toaether.exe.old, and the nextaethercommand deletes that copy. If Smart App Control, AppLocker or an antivirus blocks the new build, the update restores the old one and fails withreplace <path>: the new build does not start, so the previous build was restored: <Windows error>.
For a passphrase-protected key, start the Windows OpenSSH agent, which is
stopped by default (as admin: Start-Service ssh-agent and
Set-Service ssh-agent -StartupType Automatic), then ssh-add your key.
Windows Defender and SmartScreen
Cloudfield's Windows binaries and locally built desktop apps are not code-signed, so SmartScreen, Smart App Control or an antivirus may warn about or block them.
| What you see | What it is | What to do |
|---|---|---|
| The browser blocks the download, or "Windows protected your PC" when it starts | Download or SmartScreen reputation | Check the file's SHA-256 against checksums.txt (Manual install), then follow your organization's policy. |
| The file disappears, or "virus detected" | Antivirus detection | Read the detection name in Windows Security → Virus & threat protection → Protection history, then report it. |
"Smart App Control blocked an app that may be unsafe", or An Application Control policy has blocked this file |
Smart App Control, which runs only signed or already trusted apps | See Windows Security → App & browser control → Smart App Control. The installer does not bypass it. |
Do not turn protection off, add an exclusion or restore a quarantined file: a matching checksum proves the download is intact, not that it is safe. To report a detection on a file whose checksum matches, submit it at microsoft.com/wdsi/filesubmission as a software developer, and open an issue with the detection name.
Manual install
Every release publishes
checksums.txt and these assets, where <arch> is amd64 or arm64: the
CLI as aether-linux-<arch>, aether-darwin-<arch> and
aether-windows-<arch>.exe; aether-server-linux-<arch>;
aether-edge-linux-<arch>; aether-desktop-windows-<arch>.zip; and
aether-android.apk and .aab. aether-edge also ships as the image
ghcr.io/3xdevops/aether-edge:<tag> (edge.md).
On Linux or macOS, download the binary and checksums.txt, check it, and put
it on your PATH. Install a server binary as /usr/local/bin/aether-server,
the path the systemd unit runs.
grep ' aether-linux-amd64$' checksums.txt | sha256sum -c - # macOS: shasum -a 256 -c -
mkdir -p ~/.local/bin && install -m 0755 aether-linux-amd64 ~/.local/bin/aetherOn Windows, in PowerShell, from the directory you downloaded into:
# 1. Verify. Compare this against the matching line in checksums.txt.
Get-FileHash -Algorithm SHA256 .\aether-windows-amd64.exe
# 2. Put it somewhere on PATH under the name aether.exe.
$dir = "$env:LOCALAPPDATA\Programs\Aether"
New-Item -ItemType Directory -Force -Path $dir
Copy-Item -Force .\aether-windows-amd64.exe "$dir\aether.exe"
# 3. Add that directory to your user PATH (once), then open a new terminal.
[Environment]::SetEnvironmentVariable(
"Path", "$([Environment]::GetEnvironmentVariable('Path','User'));$dir", "User")Use aether-windows-arm64.exe on an Arm device. & "$dir\aether.exe" gui build
then adds the desktop app; from a zip you downloaded, verify the zip the same
way and add --zip <zip>, because --zip checks only its layout. To check the
release signature too, see Verify a release by hand.
To build from source, see CONTRIBUTING.md.
Desktop app
The desktop app is a window around the dashboard. It starts aether gui, the
dashboard server built into the CLI, with the same access the CLI has. While
open, it notifies you when a session needs attention and opens aether://run/<id>
and edge join and claim links (edge.md). Both install
scripts can install it; to install or rebuild it yourself:
aether gui buildOn Windows this downloads the release's prebuilt
aether-desktop-windows-<arch>.zip (about 100 MB), checks it against the
signed aether-release.json, and unpacks it. On Linux and macOS it builds the
app with Electron, using node 22 or newer from PATH or a pinned Node.js 22
it downloads into its cache. The first build also downloads Electron (about
100 MB). --source builds on Windows too, --zip <path> installs a Windows
app zip you already have, --build-dir <dir> replaces the cache directory,
and --json prints one JSON line per build phase.
| OS | App | Launcher |
|---|---|---|
| Linux | ~/.local/share/aether/desktop/ |
~/.local/share/applications/aether-desktop.desktop |
| macOS | /Applications/Aether.app, or ~/Applications/Aether.app without administrator rights |
Applications folder and Spotlight |
| Windows | %LOCALAPPDATA%\Programs\Aether Desktop\ |
Start Menu → Aether |
The app does not contain the CLI: gui build records the path of the CLI
that ran it, and the app starts that one (AETHER_BIN overrides it). If that
CLI is moved or deleted, the app's error names the path; restore it, set
AETHER_BIN, or rerun gui build. The dashboard ships inside the CLI, and
aether update rebuilds the app (Upgrading). On Windows, close
Cloudfield before rebuilding by hand.
On Windows the app runs its dashboard server without opening a console window.
To diagnose startup failures, run aether gui in a terminal.
Android app
The Android app shows the dashboard full screen on a phone. It is not on
Google Play yet; every release carries aether-android.apk. What it stores and
sends is in privacy.md, its licences in notices.md.
- On the phone, download
aether-android.apkfrom the latest release. - When Android asks, allow your browser to install apps, then open the file.
- Choose how the app reaches your server: Cloudfield Edge or Tailscale.
| Mode | You need | How it works |
|---|---|---|
| Cloudfield Edge | A GitHub account the server lets in | The app runs its own copy of the aether CLI. Sign in with GitHub in the browser, pick your server, check its host key and tap Link and pin. A server that admits only approved devices makes the phone wait for approval (teams.md). |
| Tailscale | The phone on the server's tailnet, and the server's dashboard on (web-port, see networking.md) |
Type the server's MagicDNS name, such as my-server.tailnet-name.ts.net. The app opens the server's HTTPS dashboard; your tailnet login identifies you. |
In Cloudfield Edge mode SSH runs end to end; the edge relays encrypted bytes. The app keeps a device key and edge tokens in private storage, excluded from backups. Uninstalling the app or clearing its storage deletes the key, so the phone signs in as a new device and may need approval again. To switch mode or check for updates, long-press the app icon and choose Connection.
Share text or a link to Cloudfield's New session from any app and the dashboard opens a new session with it as the prompt, cut to 8000 characters; nothing starts until you send it. Back walks the dashboard's history, returns to its home from a page a share or notification opened, and leaves the app only from home.
To get notifications on the phone, install a UnifiedPush distributor such as ntfy, then turn on Notify this device when a session needs me in the dashboard's Settings (networking.md).
About once an hour, when you bring it to the front, the app checks for a
release and shows a banner. In Cloudfield Edge mode it offers the release
the server runs, from the gateway's update.check, and nothing newer until the
server updates. In Tailscale mode it runs the bundled CLI's check, which
follows the config.json that CLI has, and stable (beta until the first
stable exists) when it has none. Download update fetches the new APK; open
it and Android installs it over the app, keeping your data. A Google Play
install gets Open Google Play once a stable release is 7 days old; Play
carries no beta or nightly. In Cloudfield Edge mode the server refuses an
older app: a screen such as This app (v0.7.0) is too old for the server (v0.8.0 needs v0.8.0 or newer). stays until you install the update it links.
Android installs an APK over the app only when both carry the same signing
key; uninstall an APK you built from a checkout first. The release key's
certificate SHA-256, as apksigner verify --print-certs prints it, is
2d7e0777aa8a24ac0090dcdd4c620910c9ef13c7775b1576ed69230f1dd37c3a.
First boot
aether-server setup sets the server up as a systemd service. The install
script runs it when you answer server; to run it yourself:
sudo aether-server setupIt asks these questions in order, and shows /etc/aether/server.conf for you
to confirm before the last one. Enter takes the default, or on a rerun the
current value. Input that ends at a prompt, such as Ctrl-D, stops setup with
nothing written.
| Question | Default | What it decides |
|---|---|---|
Continue without Docker |
no | Asked only when Docker does not answer. No stops with nothing written; install or start Docker, then run setup again. |
SSH listen address for Cloudfield, not the system sshd |
:2222 |
The address and port clients connect to. |
Data directory |
/var/lib/aether |
Where the server keeps everything (data directory). |
Use Tailscale to reach this server |
yes, when tailscaled names its tailnet | Asked only when Tailscale runs on the host. Yes signs people in by their tailnet login. No turns tailnet sign-in off and the edge on. |
Let anyone on the tailnet join without an admin approving them |
no | Asked after a tailnet yes (networking.md). |
Also require a registered SSH key, for tailnet devices more than one person uses |
no | Asked after a tailnet yes. |
Dashboard HTTPS port (0 = off) |
443 | Asked after a tailnet yes without a key requirement (networking.md). |
Turn on the edge too |
no | Asked after a tailnet yes. Without a tailnet, setup turns the edge on without asking. |
Who may reach this server through the edge? |
no default | 1, account access: a GitHub sign-in is enough. 2, approved devices: each new device waits for approval (edge.md). |
Let agents use Docker inside their sessions? This installs the Sysbox runtime (Ubuntu/Debian) |
yes | Skipped when Docker already has Sysbox or does not answer. |
With the edge on and no owner yet, setup prints a claim link holding a code
valid for 30 minutes (sudo aether-server edge claim-code prints a new one).
Setup starts nothing; it ends on next:: start the service, then on your
machine link it. On a tailnet that is aether link <MagicDNS name>, or
Tailscale in the app's Connect step; through the edge, aether login and
aether link --claim <code>, or Cloudfield relay with the claim link; with
neither, aether link <this-host>:2222. When the edge is not
https://edge.onaether.dev, both client commands carry --edge <url>, because
a claim code names no edge.
sudo systemctl daemon-reload && sudo systemctl enable --now aether-server
sudo journalctl -u aether-server -fOn first start the server creates its database, repositories and SSH host
key. Do not delete the key: clients pin it, and the edge knows the server by
it. The first identity to link becomes the super admin: the SSH key or tailnet login
of whoever runs aether link first, or the account that uses the claim code.
With the edge on, a first SSH key must come from the server itself or an SSH
tunnel (networking.md).
For automation, aether-server install writes the same files from flags,
such as --edge-url https://edge.onaether.dev --edge-access approved-devices.
It installs Sysbox unless you pass --sysbox=false or
--session-runtime vm, turns the edge on only with --edge-url, and issues
no claim code. With Tailscale on the host, the first tailnet login becomes the
super admin unless you pass --tailnet-require-key.
With the edge on, install also prints the edge, policy and server id. It
prints one warning: line naming what the host lacks (git, ssh-keygen, a
running systemd, or the memory and disk above), without failing, and ends on
next: with sudo aether-server edge claim-code, then aether login and
aether link --claim <code> on your computer.
A rerun that names an option the kept config file sets differently exits 1,
writes nothing, and prints aether-server config set <key> <value> for each
such option (with --config <file> when not the default):
aether-server: /etc/aether/server.conf exists and sets other values than you named; install keeps an existing config file, so nothing was written. Apply them with:
sudo aether-server config set edge-access approved-devices
or rewrite the file from these flags alone by adding --forceServer options
Options live in /etc/aether/server.conf. Updates never rewrite it, and a
rerun of setup keeps it unless you pass --force, and a rerun of install
that names a different value fails as shown above.
aether-server config show prints each option's value and source, and
aether-server serve -h describes them. Change one with
sudo aether-server config set <key> <value> (or config edit), then
sudo systemctl restart aether-server. --config <file> may go before or
after the key and value of config set.
| Option | Default | What it does |
|---|---|---|
web-port |
0 (off) |
Serve the dashboard over HTTPS on the tailnet at this port (networking.md). |
edge-url, edge-access |
off, approved-devices |
The edge the server enrolls with, and who may reach it through the edge: account or approved-devices. |
edge-reviewer |
off | Under edge-access account, a connector authorized by the edge's reviewer account acts as the member an admin binds it to with aether member link --reviewer <login> <member-id>. |
tailnet-auto-join |
off | Tailnet logins join as approved members instead of pending. |
tailnet-require-key |
off | Tailnet connections must also present a registered SSH key. Cannot be combined with web-port. |
run-cpus, run-memory, run-pids |
8 CPUs (at most the host's), 8 GiB, 4096 | Ceilings per session and member environment, not reservations. |
min-free-disk |
0 (5% of the filesystem, 5 to 20 GiB) |
Free-space reserve below which new sessions are refused. Negative turns the check off. |
checkout-ttl |
72h |
How long a finished session's checkout is kept after the session ends. It also stays while the session's container is kept. Negative keeps it forever. |
run-container-ttl |
168h |
How long a closed session's container is kept so the session can be relaunched. A relaunched session's clock restarts at its next close. |
run-sleep-after |
30m; 5m for session VMs |
How long an idle Enhanced Claude Code session keeps its container running before Cloudfield stops it until its next use, and how long an idle session VM runs before Cloudfield saves it to disk (Idle sleep). A value you set applies to both. 0 keeps every session awake. |
browser-image |
ghcr.io/3xdevops/aether-browser:<release> |
The image a session's Browser tab starts from. Set it and updates stop moving it. |
conflict-coordination |
on | Let overlapping sessions message each other (coordination.md). |
API keys for key-based agents go in /etc/aether/aether-server.env, which the
unit loads; subscription logins live in each member's home
(harnesses.md). The unit runs the server as root, since Docker
access is root-equivalent anyway. Its header comment explains how to run
unprivileged; then every image needs a root user and self-update is off.
Keep the server reachable under load
The server creates every container it starts, for sessions, terminals, the
assistant, harness updates, browser companions, the dictation engine and
cleanup helpers, under one cgroup parent: aether.slice with Docker's systemd
cgroup driver, the default on current Ubuntu and Debian, and /aether with the
cgroupfs driver. docker info --format '{{.CgroupDriver}}' prints the
driver. This parent overrides cgroup-parent in /etc/docker/daemon.json.
The server reads the driver once, so after changing Docker's cgroup driver,
restart it: sudo systemctl restart aether-server. All sessions together
then compete as one group with system.slice, which holds aether-server,
docker, containerd, ssh and tailscaled, and with user.slice, which
holds login shells.
aether-server install and setup write /etc/systemd/system/aether.slice
beside the unit, with CPUWeight=50 and IOWeight=50. system.slice and
user.slice keep the default 100, so while all three want more than the host
has, sessions get a fifth of CPU and disk and system.slice two fifths;
against system.slice alone, sessions get a third. Weights only divide CPU
and disk while they are contended; they cap nothing, so sessions use every
core of an otherwise idle host. Image builds, pulls and other services on the
host, such as a CI runner, stay in system.slice at twice the session group's
weight. The slice sets no memory limit and never throttles, stops or kills a
session. Its name has no dash because systemd nests a-b.slice inside
a.slice, whose default weight would decide instead. install rewrites the
file, so change the weights in a drop-in: sudo systemctl edit aether.slice,
under [Slice].
A container created before the slice existed stays a docker-<id>.scope in
system.slice at weight 100 until the session's container is recreated. With
N of those busy, a service at the default weight gets about 1/(N + 1) of
contended CPU. The unit's CPUWeight=10000 and IOWeight=10000, the systemd
maximum, put the server ahead of them and of the other services in
system.slice: 10000/(10000 + 100N) of contended CPU. The server is idle most
of the time, and a weight only counts while the unit has work, so the maximum
costs others only the CPU and disk the server actually uses. The git commands
the server runs on the host, such as diff snapshots and evidence, run in its
unit and get the same weight.
To add the slice to an existing install, without restarting the server or any session:
sudo aether-server install --sysbox=false
sudo systemctl daemon-reloadinstall rewrites the unit and the slice and keeps /etc/aether/server.conf.
Running containers stay where they are; containers the server creates from now
on land in the slice. Check it:
systemctl status aether.slice
systemd-cgls --unit aether.slice
cat /sys/fs/cgroup/aether.slice/cpu.weight
sudo docker ps --filter label=aether.managed=true --format '{{.Names}}'
sudo docker inspect --format '{{.HostConfig.CgroupParent}} {{.HostConfig.Runtime}}' <container>
cat /proc/"$(sudo docker inspect --format '{{.State.Pid}}' <container>)"/cgroupsystemctl status shows the slice inactive until its first container starts.
cpu.weight prints 50; daemon-reload applies the file to a slice that is
already running. While the slice is inactive the file does not exist, and 100
without a drop-in of your own means the file is not loaded. With the systemd
driver, a container created after the update prints aether.slice and its
runtime: with the default container-runtime auto, sysbox-runc for terminals
and sessions that are not isolated when Sysbox is installed and usable,
otherwise Docker's default runtime, usually runc. A container created before
the slice prints an empty parent and sits under Docker's default parent:
system.slice, unless cgroup-parent in /etc/docker/daemon.json names
another. On cgroup v2, a new container's /proc/<pid>/cgroup reads
0::/aether.slice/docker-<id>.scope, with the full container ID and, under
sysbox-runc, /init.scope appended. Read it on the host: inside a Sysbox
container the cgroup namespace shows only 0::/init.scope. With the cgroupfs
driver the parent is /aether and the path 0::/aether/<id>. On cgroup v1 the
file has one line per controller, each ending in the same path.
If new sessions fail to start with cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice", Docker's cgroup driver changed after the
server started: sudo systemctl restart aether-server. To give the session
group the default weight again, remove any systemctl edit drop-in and run
sudo systemctl set-property aether.slice CPUWeight=100 IOWeight=100.
MemoryLow=1G makes the kernel reclaim the server's first gigabyte only after
unprotected memory. Memory protection passes down the cgroup tree, so it
protects the server only once system.slice holds at least as much; the
commands below set that, and the unit cannot.
An existing install keeps its old unit until you run the command the journal's unit warning prints (What an update does not change). To apply the settings now, without a restart, to the server and to the services it and you depend on:
sudo systemctl set-property system.slice MemoryLow=1G
sudo systemctl set-property aether-server CPUWeight=10000 IOWeight=10000 MemoryLow=1G
for unit in docker containerd sysbox-fs sysbox-mgr ssh tailscaled; do
sudo systemctl set-property "$unit" CPUWeight=10000 IOWeight=10000
doneset-property changes the running unit and keeps the change in
/etc/systemd/system.control/<unit>.d/. sysbox-fs serves /proc inside
every Sysbox container, so the server's commands in sessions wait on it; skip
both sysbox-* units without Sysbox. Work docker, containerd and sysbox-fs
do for a container, such as the /proc reads sysbox-fs serves, runs at their
raised weight, not the container's. The SSH unit is sshd on Fedora and
Arch; skip tailscaled without Tailscale. To keep the setting with your other
overrides instead, put the same lines in a drop-in, such as
sudo systemctl edit docker, which reloads systemd when you save: under
[Service] for a service, under [Slice] for system.slice.
With cgroupfs, systemd does not manage /aether, so the slice file does not
apply and the group keeps weight 100 beside system.slice; the service weights
order only the services inside system.slice. IOWeight takes effect only on
a disk whose IO scheduler honors weights, such as bfq; NVMe disks often use
none (cat /sys/block/<disk>/queue/scheduler). MemoryLow needs cgroup v2.
systemd 232 and newer know all three directives; an older one logs
Unknown key name and ignores them.
Client files and variables
The aether CLI keeps these files on your computer and reads these
variables. <config dir> is the directory that holds config.json.
aether status prints its path.
| File or variable | What it is |
|---|---|
config.json |
The server link aether link writes: $AETHER_CONFIG_DIR/aether/config.json when that variable is set, else ~/.config/aether/config.json on Linux (honours $XDG_CONFIG_HOME), ~/Library/Application Support/aether/config.json on macOS, %AppData%\aether\config.json on Windows. It also lists the linked clones, one per workspace (repos). |
<config dir>/edge-device-key |
This computer's private key for edge links, created by the first aether login. |
<config dir>/edge-tokens.json |
The device token for each edge you signed in to with aether login. Do not share or commit it. aether logout revokes the token at the edge and deletes it here. |
~/.ssh/id_ed25519 |
Your SSH key, which aether link <host:port> creates only when you have none. |
~/.ssh/known_hosts |
Where aether link pins the server's host key. |
~/.ssh/aether_config, ~/.ssh/aether_known_hosts |
Written by aether ssh-config (terminal.md): the <session>.aether host block and its host key. |
AETHER_CONFIG_DIR |
Replaces the platform config directory above. |
AETHER_NO_UPDATE_CHECK |
Any non-empty value turns off release checks. |
AETHER_BIN |
Path of the aether binary the desktop app starts and aether gui build requires. |
The client-side sync daemon
The sync daemon keeps a local clone current without git fetch: it fetches
session branches as agents commit to them, and in a workspace without a source
mirror it pushes your local base branch to the server. It is optional and sessions
once per clone.
aether daemon install --repo ~/code/myproject
systemctl --user daemon-reload && systemctl --user enable --now aether-daemondaemon install writes a user service (a systemd unit, a launchd agent
com.aether.daemon, or the Scheduled Task aether-daemon) and prints the
command that starts it; the second line is for Linux. --server and
--server-id default to the linked server, with its key. Pass
--server-id <id> --edge-url <edge> (through the edge) or --server only for a
different server. aether daemon run runs in the foreground.
A workspace's base branch is whatever members push, unless an admin gives it
a source mirror: a read-only repository the server fetches from, with a deploy
key under mirrors/. The daemon's base push is then refused
(teams.md).
Upgrading
Update the server first, then its clients. A server refuses clients older than itself, and a client newer than its server gains nothing.
Update the server
A super admin can update from any client or from the dashboard's Updates dialog (dashboard-frontend.md), and root can update on the server host:
aether server update [--version <tag>] # from any client, as a super admin
sudo aether-server update [--version <tag>] # on the server hostBoth check the signature, replace aether-server and the aether beside
it, and restart the service. The old binaries stay as aether-server.prev and
aether.prev, and the database is copied first when its schema changes. If
the new server does not finish starting, it restores both and records why.
Sessions keep going; aether attach and aether sync --live reconnect.
sudo aether-server update prints each step as it runs, with how long it
took. The database copy appears only when the release changes the schema, and
the wait at the end lasts until the new server has reconnected every open
session:
aether-server v0.8.7-beta.12 -> v0.8.7-beta.13
https://github.com/3xDevOps/cloudfield/releases/tag/v0.8.7-beta.13
checking and downloading the release ... done in 4.12s
stopping aether-server; sessions keep running ... done in 2.31s
copying the 9.1 GB database to /var/lib/aether/server-update/aether.db.pre-update so a failed start can roll back ... done in 21.43s
starting aether-server ... done in 120ms
waiting for aether-server v0.8.7-beta.13 to reconnect its sessions and finish starting; sudo journalctl -u aether-server -f shows each session
aether-server v0.8.7-beta.12 -> v0.8.7-beta.13 restarted in 1m12s (pid 1234 -> 5678)
aether /usr/local/bin/aether v0.8.7-beta.13With session-runtime vm, sudo aether-server update has the new binary run
aether-server vm install before the restart, and prints its steps, so new VMs
boot the release's Cloud Hypervisor and guest kernel. Running VMs keep theirs,
and their helper disk, until they stop; Ctrl-C during that step undoes the
update. Before it starts any VM, a server started by its systemd unit also puts
its release's Cloud Hypervisor and kernel in place, downloading the kernel only
when the one there came from another release. That covers an update from a
client or the dashboard, and a rollback. If it cannot during an update's first
start, the update rolls back; on any other start the server logs the error and
keeps serving. AETHER_NO_UPDATE_CHECK=1 turns that check off.
aether server update --when idle waits until no session is working, no terminal
is attached and no SSH connection into a session is open; --cancel clears that
pending update. --allow-downgrade allows an older release, --yes skips the
confirmation, and --status, which any member can run, shows the versions,
any pending update and the last update's outcome and error.
The server refuses a release missing any client binary or the Android app,
and names the asset. sudo aether-server update needs an aether-server
systemd unit that runs as root, and aether server update refuses when the
server cannot replace its own binary. On such hosts, run sudo aether update
and restart the server yourself.
After a rollback, sudo journalctl -u aether-server -n 50 shows why. If the
new binary cannot start far enough to roll back, restore the old one by hand,
running the two database lines only if
/var/lib/aether/server-update/aether.db.pre-update exists:
sudo systemctl stop aether-server
sudo mv /usr/local/bin/aether-server.prev /usr/local/bin/aether-server
sudo mv /usr/local/bin/aether.prev /usr/local/bin/aether
sudo rm -f /var/lib/aether/aether.db-wal /var/lib/aether/aether.db-shm
sudo mv /var/lib/aether/server-update/aether.db.pre-update /var/lib/aether/aether.db
sudo systemctl start aether-serverRelease channel
Cloudfield ships three release channels. A channel takes its own releases and those of every channel above it in stability:
| Channel | Takes | Where its releases come from |
|---|---|---|
stable |
stable releases | an admin promotes a beta |
beta |
beta and stable | an admin promotes a nightly |
nightly |
all three | the newest main commit whose CI is green, built at most every 4 hours |
The tag shows the channel (v0.9.0-nightly.12, v0.9.0-beta.2, v0.9.0),
and the signed manifest carries it too (Verified
releases). Nightly builds come straight from main:
breaking changes arrive when they merge, and a nightly can contain a
security issue that a later build fixes. Run nightly only to develop
Cloudfield. A promotion rebuilds the same commit under the new tag and keeps its
build number, so a nightly and the beta promoted from it are the same code.
A server follows one channel, the release_channel setting. Its release
check, aether server update and sudo aether-server update with no
--version take the newest release on that channel, read from the
channel's signed pointer (Verified releases). A server
that existed before channels follows beta. A new server chooses at its
release checks: stable when stable.json is published and verifies, beta
once stable.json itself (not its signature) has answered 404 at two checks
in a row, so one 404 while the asset is replaced does not decide it. Until
then, and on any other answer, such as on a first boot without network, the
server follows beta and server.update_status reports channel_pending.
Any member sees the channel: server.update_status carries it. Only a super
admin changes it, in the dashboard's Settings → Server → Release channel
(Stable, Beta and Nightly, with the confirmations below) or as root on the
host:
sudo aether-server update --channel beta # prints what changes
sudo aether-server update --channel beta --yes # saves itWithout --yes it prints the confirmation and changes nothing. A plain
sudo aether-server update only reads the channel, following beta while
the server has not chosen. Saving installs nothing: the server checks the
new channel at its next release check, and sudo aether-server update
installs what it finds.
A server never installs a lower build because its channel changed. Moving
to nightly asks first (aether-server update --channel nightly prints the
same text):
Switch to nightly builds?
Not recommended unless you are developing Cloudfield.
- Nightly builds come from Cloudfield's main branch. Breaking changes
arrive as soon as they merge.
- A nightly build may contain security issues that a later build fixes.
- You cannot go back right away. This server stays on its nightly build
until the channel you return to catches up with it.Leaving nightly for a channel below its build, the server stays on its
build until that channel reaches it, then updates as usual:
Switch to beta?
This server runs v0.9.0-nightly.12 (build 412). Beta is at v0.9.0-beta.1 (build 380). The server stays on its current build until beta reaches build 412, then updates as usual. It never installs an older build.Meanwhile server.update_status reports channel_wait, and sudo aether-server update prints
aether-server v0.9.0-nightly.12 (build 412) stays on its build until beta reaches it: beta is at v0.9.0-beta.1 (build 380).
The same build under the lower channel's tag installs. Each channel keeps
its own pointer mark in server-update/check.json, so switching away and
back never lets an older pointer in.
A pending update waiting for an idle moment applies only a release the
server's channel takes. --when idle refuses any other with the message
below; the dashboard's switch retires a pending one, recording it as failed
with the reason, and one left pending after a switch on the host fails at
the idle moment. When the switch cannot fetch a pending update's manifest to
check it, the update stays pending (kept, with kept_error giving the
fetch error) and is refused at its idle moment if the new channel does not
take it. --when now installs any release.
release preflight failed: the server no longer takes this update: v0.9.0-nightly.13 is a nightly release, which an install on beta does not take; update with --when now to install it anywayClients follow their server. aether update and the desktop app take the
release of the channel their default linked server follows, and by default
never one above that server; a member who cannot update the server, or
--force, can (Update a client). The Android app in
Cloudfield Edge mode offers the server's release; in Tailscale mode its bundled
CLI follows its own config.json, and stable when it has none
(Android app).
Automatic updates
On every channel, a server applies updates by itself when idle. Its check
runs every 6 hours. When it finds a release on the server's channel above
the running build, the server schedules it as aether server update --when idle would, and applies it once no session is working, no terminal is
attached and no SSH connection into a session is open. server.update_status
marks that pending update by_server, and the timeline names server as
its actor, not a member.
The server schedules nothing when:
- It has not chosen its channel yet, it runs a build from source (a version
past a tag, or dirty), or it runs a release from before channels, which has
no signed build.
server.update_statusstill offers the release; a server onv0.8.7-beta.Nshows the update and stays until a super admin runsaether server updateorsudo aether-server update. Self-applying starts once it runsv0.8.7, build 1. - The release is from before channels, has no signed build, or fails the
update preflight.
schedule_errorinserver.update_statussays why the server last failed to schedule the newest release; it retries on a backoff from 5 minutes, and declines the release after 3 failures in a row. - A member's update is already pending. The server never replaces it.
- The release's build is not above a declined one (below).
See it with any member's aether server update --status. It prints the
server's version, the latest release on its channel, update available,
when the check ran and why it failed, and the phase of the current update.
A server-scheduled update shows as pending update: <tag>, requested by server at <time>, and waiting: 1 session is still working says what holds
it. last attempt: gives the outcome and error of the previous update. The
channel, channel_wait, declined, decline_note and schedule_error are
in server.update_status (local-gateway.md), which the
dashboard's Settings reads; --status does not print them.
Turn it off with AETHER_NO_UPDATE_CHECK=1 (any non-empty value) in
/etc/aether/aether-server.env, then restart the server. That stops the
release check and so these updates, including one the server scheduled
earlier. A member's scheduled update still applies.
A server never schedules again a build a super admin turned down. Each of
these records that build as declined in the database: --cancel, an update
with --allow-downgrade, an update that failed (after 3 tries for a network
error), and a server-scheduled update whose new binary failed its start and
rolled back. The server then schedules only a build above the declined one,
so a rollback holds until a newer build ships. A super admin's own
aether server update ignores the mark, and so does --version. Recording
comes first: a cancel or rollback whose mark cannot be saved is refused with
the error. A cancel whose release's build cannot be read from its signed
manifest still cancels, and says in not_declined (decline_note in
server.update_status) that the server may schedule it again.
A cancel is refused once an update is past the point of no return, from the
start of --when now and once a scheduled update has downloaded, and a new
update is refused meanwhile:
a server update is already being applied: v0.9.0-nightly.13 is applying now and can no longer be cancelledAn update whose download could not be undone refuses the next one until
aether-server restarts and puts the previous binaries back. --when idle
cannot go with --allow-downgrade; the CLI refuses with aether server update: --allow-downgrade applies now only; drop --when idle. A release that is not above the running build is refused
without --allow-downgrade:
refusing to downgrade: v0.9.0-nightly.12 (build 412) is older than the running v0.9.0-nightly.13 (build 413); pass --allow-downgrade (allow_downgrade) to install it anywayThe same build on the same channel reads is the same build as the running
instead. --allow-downgrade never crosses a database schema: a release whose
schema is below the database's fails with this database is at schema <n> and <tag> only knows schema <m>; it could not open it.
Update a client
aether updateIt installs the newest release on the channel its default linked server
follows (the top-level fields of config.json; gui_server does not decide)
when that is newer, after checking its signature, then rebuilds the desktop app if one is installed (--no-app skips
that). --version <tag> picks a release, and --check, with --json for a
script, only reports whether a newer one exists. When this CLI is behind its
server, it installs the server's release instead. When the server is behind
and you are a super admin, it refuses unless you pass --force:
aether: update the server first: aether server update (the server runs v0.8.0 and this installs v0.9.0; --force skips this check)It never asks for privileges. A CLI in a directory you cannot write, such as
/usr/local/bin, fails before anything downloads with
/usr/local/bin is not writable by this user; re-run as `sudo aether update` .
sudo resets HOME, so sudo aether update cannot see your link; pass the
server's version (from aether server update --status) with --version. On a
server host it also replaces aether-server and prints the restart to run;
with a root unit, sudo aether update runs sudo aether-server update.
The dashboard's Update now runs the same update and relaunches the app.
For a CLI in a directory you cannot write, macOS asks once for an administrator
password (a dialog titled osascript); Linux shows sudo aether update.
A running sync daemon restarts onto the new binary within about a minute.
One from v0.7.0-alpha.12 or older needs one restart by hand:
systemctl --user restart aether-daemon (macOS:
launchctl kickstart -k gui/$(id -u)/com.aether.daemon; Windows:
schtasks /End /TN aether-daemon, then schtasks /Run /TN aether-daemon).
With no linked server, or one that does not answer and was never read, the
CLI and the desktop app follow stable, and beta until the first stable.json
exists. A running desktop gateway keeps the last channel it read while the server is down; a one-shot aether update does not, and checks stable then. Each channel's
pointer mark is in update.json beside config.json.
AETHER_NO_UPDATE_CHECK=1 turns off release checks in the CLI and dashboard.
What an update does not change
/etc/aether/, and a browser image pinned withbrowser-image. Without a pin, the server removesaether-browsertags from earlier releases that no container uses.- The systemd unit. When the release ships a different one, the journal warns
with the command that applies it:
sudo aether-server install --sysbox=false && sudo systemctl daemon-reload && sudo systemctl restart aether-server. When onlyaether.sliceis missing or differs, and a local Docker uses the systemd cgroup driver, the warning leaves out the restart (Keep the server reachable under load). Keep your own changes in a drop-in (sudo systemctl edit aether-server). - Sysbox's settings: rerun
sudo aether-server sysbox install. With containers running, it printssudo systemctl restart sysboxfor later. - An open member environment, which keeps its image until it is stopped and reopened (environments.md).
- A session's open Browser tab, which keeps its image until you stop and reopen the session. A browser action it lacks fails with an error that says so.
When the server refuses an older client
A server refuses any client older than itself. The CLI prints this and exits 1:
aether: rpc error -32005: this aether is v0.7.0-alpha.11 and the server runs v0.8.0; update this client first: run `aether update` (desktop: Update in the app; Android: install the latest aether-android.apk), then retryRun aether update, or Update now on the dashboard's update screen, then
retry; both install the server's release. Updating and git fetch and push keep
working meanwhile. Rerun a refused aether link --repo afterwards to set the
git remote. Development builds (dev or a bare commit) are never refused.
Verified releases
Each release publishes aether-release.json, listing every asset's SHA-256
and size, and aether-release.json.sig, an ed25519 signature over it. The
updaters install a release only when the signature verifies against a key
built into the running binary, the manifest names the tag, and each binary
matches; otherwise they print the error and change nothing. Releases from
before signing have no manifest, so install those with the install script.
The manifest also signs the release's channel (nightly, beta or
stable), which must match the tag's form (-nightly.N, -beta.N, none),
and its build and hotfix numbers. Releases from before channels carry
none of the three and count as build 0. Release checks, aether server update and sudo aether-server update order a release by its (build, hotfix), whatever the tags say: v0.9.0-beta.2 at build 412 is newer than
v0.9.0-nightly.10 at build 400. Between two build-0 releases, the version
decides. The server refuses a lower build, or the same build on the same
channel, without --allow-downgrade; the same build under another
channel's tag installs. aether update and the dashboard's Update now
never pick a lower build on their own; aether update --version <tag>
installs the release you name. server.info reports the server's build
and min_client_build, and a client sends its build in its SSH banner as
v0.9.0-beta.2+b.412 (+b.412.h.1 for a hotfix). When both carry a
build, the server refuses a client below its build and the client matches
its server by build; when either is build 0, the version decides, as
before. A server from before builds ignores the build in the banner. A
match to a lower build is refused, and aether update --version <tag>
installs it anyway.
The manifest also signs image digests (edge.md).
Verify a release by hand
You need OpenSSL 3 and jq. In a directory holding the downloaded asset and
the same release's aether-release.json and aether-release.json.sig:
cat > aether-release.pub <<'EOF'
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAJ+AIq6GUVbRtEQ1ORqKJe3uh9JhvZkEQJQtYpxKbxD8=
-----END PUBLIC KEY-----
EOF
base64 -d aether-release.json.sig > aether-release.sig.bin
openssl pkeyutl -verify -pubin -inkey aether-release.pub -rawin \
-in aether-release.json -sigfile aether-release.sig.bin
jq -r '.assets["aether-linux-amd64"].sha256' aether-release.json
sha256sum aether-linux-amd64openssl must print Signature Verified Successfully, and the two digests
must match. On macOS use shasum -a 256 in place of sha256sum.
What lives in the data directory
The server keeps all its state in /var/lib/aether, or the data-dir you set.
| Path | Contents |
|---|---|
aether.db |
The SQLite database: members, workspaces, sessions and their history. |
ssh/, edge/, push/ |
The SSH host key, which clients pin and the edge knows the server by; edge enrollment; and the push notification key, without which every device must turn notifications off and on again. |
repos/ |
One bare git repository per workspace. |
checkouts/ |
One git worktree per session, and each session's recorded diff history in <run-id>.diffsnap/. With session-runtime vm, <run-id>/ holds the session's checkout disk instead of a worktree. |
transcripts/ |
Terminal recordings and Enhanced conversation logs, kept until you delete the session. |
homes/<member>/ |
Each member's persistent home: installed agents, agent logins, uploaded configuration, and their GitHub token. |
signing-keys/ |
Each member's private commit signing key, which no container mounts. Without a backup, each member runs aether github connect again for a new key. |
workspace-secrets/ |
Secret workspace variables in plain text, readable only by the server's user. |
mirrors/, profiles/ |
Source mirror settings and deploy keys, and agent profile snapshots. |
server-update/ |
Update state, and the database copy taken before an update that changes its schema. |
home-caches/, scratch/ |
Package caches and each container's /tmp. Rebuildable; no backup needed. |
vm/ |
With session-runtime vm: each VM's disks, logs and saved memory in <key>/, and in images/ the disks VMs boot from, package caches, and the environments members saved (images/tags.json and images/bases/local-*/). |
dictation/ |
The speech-to-text engine and model aether-server dictation install downloaded. Reinstallable; no backup needed. |
The other directories hold internal server state. To recover a server, back
up aether.db, repos/, checkouts/*.diffsnap/, transcripts/, homes/,
signing-keys/, profiles/, mirrors/ and workspace-secrets/. With
session-runtime vm, also back up vm/images/tags.json and
vm/images/bases/local-*/, the environments members saved; they are sparse
disks, so copy them with tar --sparse or cp --sparse=always. Like
containers, each VM's own disks and saved memory are left out: a recovered
server keeps those sessions' branches and history, and cannot reopen them.
[!WARNING] These backups hold every member's agent logins, GitHub tokens and signing keys, the mirror deploy keys and the workspace secrets. Encrypt them, restrict who can read them, and never attach them to an issue (security.md).
Recordings and diff history stay until you delete the session, so watch the disk.
Keep the path short: one too deep for Unix sockets under coord/ (about 100
characters) logs coordination unavailable for this session. For agents you do
not trust, mount the data directory nosuid,nodev (security.md).
Uninstalling
There is no uninstall script, because removing a server deletes agent logins and git history. Follow the steps in order.
Server
# 0. With the edge on: unenroll, so the edge stops listing the server.
sudo aether-server edge leave
# 1. Stop the service.
sudo systemctl disable --now aether-server
sudo rm -f /etc/systemd/system/aether-server.service /etc/systemd/system/aether.slice
sudo systemctl daemon-reload
# 2. Containers and VMs. Stopping the server does NOT remove them.
sudo docker rm -f $(sudo docker ps -aq --filter label=aether.managed=true)
sudo systemctl stop 'aether-vm-*' # with session-runtime vm
# Volumes for shared agent accounts. Removing one does not delete a home.
sudo docker volume rm $(sudo docker volume ls -q --filter label=aether.managed=true)
# Remove the standard image and saved member images according to your Docker
# image retention policy.
# 3. State, config, binary.
sudo rm -rf /var/lib/aether /etc/aether
sudo rm -f /usr/local/bin/aether-server
sudo rm -rf /usr/local/bin/cloud-hypervisor /usr/local/lib/aether # with session-runtime vm
# The VM firewall rules and IP forwarding stay until the next reboot.
sudo rm -rf /tmp/aether-patch-*Step 2 is the one people miss: containers outlive the service so a restarted
server can reattach to them. Every container and volume the server creates
carries aether.managed=true. The server writes no log files.
Client
Sign out of every edge first: deleting the config does not revoke this
device's tokens. aether logout signs out of one edge; with several, run
aether logout --edge <url> for each origin under edges in
edge-tokens.json. If an edge is unreachable, logout keeps the token and says
so. Retry, or revoke it later on that edge's Devices page.
# The sync daemon, if you installed it. On macOS instead:
# launchctl unload -w ~/Library/LaunchAgents/com.aether.daemon.plist, then delete it.
systemctl --user disable --now aether-daemon
rm -f ~/.config/systemd/user/aether-daemon.service
systemctl --user daemon-reload
# The client.
rm -rf ~/.config/aether ~/.config/aether-desktop
ssh-keygen -R '[<server-host>]:2222'
rm -f ~/.local/bin/aether # the client default
# sudo rm -f /usr/local/bin/aether if you installed there instead
# only if you ran `aether gui build`:
rm -rf ~/.local/share/aether/desktop ~/.local/share/applications/aether-desktop.desktop
rm -rf ~/.cache/aether ~/.cache/electron ~/.cache/electron-builderOn macOS the client's files are in ~/Library/Application Support/aether
and ~/Library/Application Support/aether-desktop instead. Remove those, the
app (/Applications/Aether.app or ~/Applications/Aether.app) and
~/Library/Caches/{aether,electron,electron-builder}. Run ssh-keygen -R for
every address you linked through: a reinstalled server has a new host key, and
a stale entry makes aether link fail with a host key mismatch. Keep your SSH
keys: ~/.ssh/id_ed25519 is yours, even when aether link created it because
you had none.
Windows client
Sign out of every edge as under Client (edge-tokens.json is in
%APPDATA%\aether), close Cloudfield, and run this in PowerShell. A line for
something you never installed prints an error; the rest still run:
schtasks /End /TN aether-daemon
schtasks /Delete /TN aether-daemon /F
Remove-Item "$env:USERPROFILE\aether-daemon.xml"
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\Programs\Aether Desktop", "$env:LOCALAPPDATA\Programs\.aether-*"
Remove-Item (Join-Path ([Environment]::GetFolderPath('Programs')) 'Aether.lnk')
Remove-Item -Recurse HKCU:\Software\Classes\aether
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\Programs\Aether"
Remove-Item -Recurse -Force "$env:APPDATA\aether", "$env:APPDATA\aether-desktop"
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\aether"
# only if you ran `aether gui build --source`:
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\electron", "$env:LOCALAPPDATA\electron-builder"
ssh-keygen -R '[<server-host>]:2222'
rundll32 'sysdm.cpl,EditEnvironmentVariables'- The Scheduled Task goes first because it runs
aether.exe. .aether-*are app copies an update set aside while the app was running.Aether.lnkis the Start Menu entry. Folder redirection can move the Programs folder, soGetFolderPath('Programs')finds it.HKCU:\Software\Classes\aetheris theaether://link handler.%LOCALAPPDATA%\Programs\Aetheris the default CLI directory, with theaether.exe.old*copies updates leave. With a custom-BinDir, deleteaether.exeandaether.exe.old*there;Get-Command aether -Alllists every copy onPATH.%APPDATA%\aetherholdsconfig.json, the device key and the edge tokens;%APPDATA%\aether-desktopthe app's saved settings;%LOCALAPPDATA%\aetherits build directory.ssh-keygen -Redits%USERPROFILE%\.ssh\known_hosts. Keep%USERPROFILE%\.ssh\id_ed25519(Client says why for both).- In the Environment Variables dialog
rundll32opens, editPathunder User variables and delete the CLI directory. Open terminals keep the oldPATH.
Linked repositories
Each clone that aether link --repo or aether workspace import linked
(aether status lists them) has an aether remote and a branch per session you
pulled. Repoint the base branch, which aether link set to the aether
remote, before removing it, or git push on it fails. This runs in PowerShell
too:
cd ~/code/myproject
git branch --set-upstream-to=origin/main main
git branch -D $(git branch --format='%(refname:short)' --list 'aether/run-*')
git remote remove aetherAn interrupted aether sync --live can leave *.aether-conflict files next
to your originals, holding local edits from when it paused.
Limits
- The server runs only on Linux, and setup installs Sysbox only on Ubuntu and
Debian. The Windows client has no
aether init. - Release binaries and locally built desktop apps are not code-signed (Windows Defender and SmartScreen).
- The Windows install script, and the install script without OpenSSL 3, do not check the release signature.
- The Android app is not on Google Play, needs a UnifiedPush distributor for notifications (networking.md) and has no port forwarding. Join and claim links open only in Cloudfield Edge mode.
- On Linux the desktop app runs Electron with
--no-sandbox; the dashboard page still has no Node.js access.
Troubleshooting
| Error or symptom | Fix |
|---|---|
runtime: docker engine API "1.44" cannot mount a path beneath a member home; that needs API 1.45 (Docker Engine 26.0) or newer |
Upgrade Docker Engine to 26.0 or newer. If DOCKER_API_VERSION is set below 1.45 in the server's environment, unset or raise it. |
aether link fails with knownhosts: key mismatch after the server was reinstalled |
The server has a new host key. Run ssh-keygen -R '[<server-host>]:2222' for every address you linked through, then link again. |
| The sync daemon fails with exit status 2 after an upgrade | Its service passes a flag that no longer exists, such as --sync-origin. Rerun both commands in The client-side sync daemon. |
Aether is missing from the Windows Start Menu
-Role none and a manual install add no shortcut. Check what Windows has:
$link = Join-Path ([Environment]::GetFolderPath('Programs')) 'Aether.lnk'
Test-Path "$env:LOCALAPPDATA\Programs\Aether Desktop\aether-desktop.exe"
if (Test-Path $link) { (New-Object -ComObject WScript.Shell).CreateShortcut($link).TargetPath }It must point at aether-desktop.exe, not the CLI. If not, close Cloudfield and
run & "$env:LOCALAPPDATA\Programs\Aether\aether.exe" gui build, which prints
launcher <path> for the shortcut it writes. A missing entry alone is not a
Defender quarantine (check Protection history). If it still fails, open an
issue with the CLI version and path, the gui build output, the shortcut's
target and your Windows version.