Sleep and wake

A sandbox’s status is starting, running, sleeping or failed. awake_because says why a running one is awake.

When it sleeps

A sandbox goes to sleep when, for sleep_after_seconds (30 unless you say; null for never):

• no call is waiting on it (a call waiting for a command’s result counts; an open output stream or terminal does not);
• it is not computing: the CPU its own programs use, as the guest counts it, averages under 2% of one core;
• no packet crosses its network link;
• it holds no TCP connection to the outside (connections at its own IPv6 address do not count: while it sleeps its machine answers their keepalives for it);
• every timer that was pending when the quiet began has fired;
• no program in it holds it awake (below), or the holds have lasted sleep_hold_max_ms.

Timers do not keep it awake for long: the sandbox sleeps, and the platform wakes it 0.4 seconds before its next timer is due, with its clock moved on by the time it slept. A timer due within 5 seconds is not worth a sleep.

Awake while a client is connected. A sandbox given awake_ports (up to 16 ports, when it is made or by a PATCH) stays awake for as long as a connection upgraded through its link on one of them is open: a WebSocket, such as a browser’s DevTools client on 9222 or a screen’s viewer on 6080. It is billed as awake meanwhile, and its awake_connections says how many are open. Such a sandbox is idle by those connections, its requests and its calls alone: its own CPU, its timers and its connections out do not keep it awake, and no timer wakes it. It sleeps sleep_after_seconds after the last of them ends. The platform turns on TCP keepalive on them, so a client that went away without closing is found out within about a minute, and connection_idle_seconds (1 to 86,400) closes one that has carried nothing either way for that long. A one-shot run takes neither.

POST /v1/sandboxes/{id}/sleep (glac sandboxes sleep web) puts it to sleep now. A sleeping sandbox keeps its memory, its processes, its disk and its network place: when it wakes, open connections carry on.

glac sandboxes update web --sleep-after-seconds 300     # five minutes idle
glac sandboxes update web --sleep-after-seconds never   # only when asked

What wakes it

Anything that needs it: running a command, reading or writing a file, a request on a link to one of its ports, a new connection to a port open at its own IPv6 address (networking), new data on a connection it already has, a key pressed in its terminal, a snapshot, a fork, a timer. There is no wake call. An API call that had to wake it says how long that took, in milliseconds, in its Server-Timing: wake;dur=… header. Reading the sandbox, or listing sandboxes, wakes nothing.

A sleeping sandbox’s sleep_tier says how it sleeps:

Keeping it awake from inside

A program in the sandbox that still has work after it has answered (a queue to drain, a write to finish, a log to send) can keep the sandbox awake by itself. It connects to the Unix socket /run/sbx/awake, which every process may use, and sends + to raise a hold and - to lower it. Each byte is answered with a line held <this connection's holds> <all holds>; ? only asks. A connection’s holds end with it: when the program closes it, or dies however it dies, its holds are gone, so a crash never keeps a sandbox awake. Several programs, and several holds on one connection, add up.

import socket

def hold_awake():
    """Raises a hold; close the socket it returns to lower it."""
    s = socket.socket(socket.AF_UNIX)
    s.connect("/run/sbx/awake")
    s.sendall(b"+")
    s.recv(64)  # "held 1 1": the hold is in place
    return s

const net = require('node:net');
// Resolves once the hold is in place, with the function that lowers it.
const holdAwake = () => new Promise((resolve, reject) => {
  const s = net.connect('/run/sbx/awake', () => s.write('+'));
  s.once('data', () => resolve(() => s.end()));
  s.once('error', reject);
});

c, err := net.Dial("unix", "/run/sbx/awake")
if err == nil {
	c.Write([]byte("+"))
	bufio.NewReader(c).ReadString('\n') // "held 1 1"
	defer c.Close()                       // lowers it
}

• While held the sandbox is awake: it runs, and its CPU and memory are metered as awake. awake_because says its guest holds it awake (…). When the last hold is lowered, the sandbox is idle from that moment and sleeps sleep_after_seconds later.
• For a while at the most. sleep_hold_max_ms (0 to 900,000; 30,000 unless you say) is how long holds keep an otherwise idle sandbox awake. After that it sleeps whatever holds it; the holds are still there when it wakes, and the time counts again from its next quiet. So a program that forgets to lower a hold costs that much at the most each time. 0 ignores holds.
• Being told. With sleep_notice_ms (1 to 60,000; none unless you say), every connection on the socket is sent the line sleep <ms> that long before the sandbox goes to sleep for being idle: flush what you hold, close connections, or raise a hold to stay awake. The sleep comes after that time whether or not anything answers; with nobody connected, nobody is waited for. When the cap has ended the holds, the line is sleep <ms> cap, and a hold does not stop that sleep. A sleep you ask for (POST …/sleep) is not announced.
• Raise it before you need it. A hold raised in the last moment before the sandbox stands still (under a millisecond) holds it from its next wake. Connect again if the connection ends without your doing (the agent behind the socket was restarted).

Restarts and failures

POST /v1/sandboxes/{id}/restart (glac sandboxes restart web) boots it again from its disk: what was running is gone, what was written stays; its id, name, links and Drives do not change. The same happens when the machine stops itself (reboot, poweroff, a kernel panic): the platform notices within a quarter of a second, stops counting it as awake, and the next call that needs it boots it from disk. A call that was waiting for one of its commands then gets 409 command_lost. A sandbox that failed to start says why in failure and failure_code; restarting it is the way out.

Checkpoints

A sandbox that is awake when its machine is lost starts again on another from its disk, and what was running is gone (when a machine is lost). With checkpoints it goes on there with its processes, from its newest checkpoint. They are off unless you ask:

glac sandboxes create --name agent --checkpoint 10     # every 10 seconds while it is awake (on: 10)
glac sandboxes update agent --checkpoint off           # none from now on

• With the API: checkpoint: {"every_seconds": 10} (1 to 600; 10 unless you say) when it is made, or in a PATCH; "checkpoint": null takes them away. The SDKs take checkpoint on create and update, and the MCP server has sandbox_checkpoint.
• What is taken, and what a takeover loses, are as for a keyed instance: what its memory and disk changed since the last one goes to the object store while it is awake, its Drives pinned with it; a sandbox with a volume that is not a Drive takes none. See checkpoints. Its checkpoint says how they go (last_at, behind_seconds, stored_bytes, error), and sandbox.moved says with_processes: true when it came back with them.
• Where. Only a machine that copies its disks to an object store and keeps memory in layers takes checkpoints, and a sandbox with them is made on such a machine: where the platform has none, it is refused (409 checkpoints_unavailable), and while each one is away or full, 503 no_capacity, retryable.
• What it costs. What the store keeps of its newest checkpoint is billed while it runs as a sleeping sandbox’s state is (state, $0.02 a GB-month); sending it costs nothing.

Deleted when it sleeps too long

A sandbox can say how long it may sleep before it is deleted (delete_after_idle_ms). A project can say it for all of its sandboxes, in days (PATCH /v1/project with delete_after_idle_days, 1 to 3650; glac project --delete-after-idle-days 30, and the SDKs’ update_project). Without either, nothing is deleted for sleeping, and expires_at deletes it at a time. A sandbox put to sleep by a spend limit or by the plan is never deleted for that.