upstream/upstream-yaak
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

snapshot of https://github.com/mountain-loop/yaak.git (main, 2026-09-23; blobs > 131072B omitted)
Some checks failed
Lint and Test / Lint/Test (push) Has been cancelled
Details
Generate Sponsors README / deploy (push) Has been skipped
Details

Browse Source
This commit is contained in:
upstream-mirror
2026-09-23 17:39:41 +05:00
commit 906fb231cc
1544 changed files with 164131 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
crates-server/yaak-web/Cargo.toml Normal file
View File

@@ -0,0 +1,40 @@
[package]
name = "yaak-web"
version = "0.1.0"
edition = "2024"
publish = false
description = "The server behind Yaak in the browser: executes sends, and can serve the app"
# The send engine (yaak-http) and the model types it speaks (yaak-models, for
# HttpRequest / Cookie / HttpResponseEventData). Deliberately NOT yaak (the
# render + storage orchestration), yaak-plugins, or the RPC router: this crate
# opens no database, runs no plugins, and renders nothing. yaak-models comes
# along only because yaak-http's types are its types; nothing here calls into
# its query layer.
[[bin]]
name = "yaak-web"
path = "src/main.rs"
[dependencies]
async-trait = "0.1"
axum = "0.7"
base64 = "0.22.1"
bytes = "1.11.1"
clap = { version = "4.5", features = ["derive", "env"] }
env_logger = "0.11"
futures-util = "0.3"
log = { workspace = true }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "signal", "sync", "io-util", "time", "net"] }
tower-http = { version = "0.6", features = ["compression-gzip", "compression-zstd", "cors", "fs"] }
ts-rs = { workspace = true }
url = "2"
uuid = { version = "1", features = ["v4"] }
yaak-http = { workspace = true }
yaak-models = { workspace = true }
[dev-dependencies]
tempfile = "3"
tower = { version = "0.5", features = ["util"] }

237
crates-server/yaak-web/README.md Normal file
View File

@@ -0,0 +1,237 @@
# yaak-web
The network half of Yaak in a browser — and, with `--serve`, the half that
hands the browser the app in the first place.
A tab can't see an HTTP response the way a desktop app can: CORS hides most
headers (2 of 8 in a typical response), redirects are followed silently, and
there is no timeline. So the tab renders the request and posts it here, and this
process puts it on the network with the desktop's own engine (`yaak-http`) and
streams back everything that happened — every header, every redirect hop, DNS
timing, the body — for the tab to store.
It is a **stateless executor**. It keeps nothing: no database, no files, no
sessions, no cookies between calls. Every byte it sees comes from the tab in the
request, and every byte it returns is stored by the tab. Restart it any time.
## Self-hosting it
One container, no configuration, nothing behind it:
```shell
docker run -p 8080:8080 ghcr.io/mountain-loop/yaak-web
```
Open <http://localhost:8080>. The image carries the built web client and this
binary, which serves it — so the app and its sends are on one origin, and the
tab's send URL is a path (`/v1/http/send`) rather than an address anyone has to
configure. The image is `linux/amd64` and `linux/arm64`, built from
`Dockerfile.web` at the repo root.
Your data lives in your browser (SQLite compiled to wasm, in IndexedDB), not in
the container. The container is stateless: nothing is written to disk, so
upgrading is `docker pull` and nothing else.
Two settings are worth knowing about:
```shell
docker run -p 8080:8080 \
-e YAAK_WEB_ALLOW_PRIVATE_NETWORKS=true \
-e YAAK_WEB_RATE_LIMIT_PER_MINUTE=0 \
ghcr.io/mountain-loop/yaak-web
```
- **`YAAK_WEB_ALLOW_PRIVATE_NETWORKS=true`** lets sends reach loopback,
private and link-local addresses. Off by default, and it should stay off on
anything strangers can reach — see [What it refuses](#what-it-refuses-and-why).
Turn it on for an instance on your own network, where calling the API on the
next machine is the whole point. Note that "private" is relative to the
_container_: `127.0.0.1` is the container itself, and reaching the Docker
host means `host.docker.internal` (or `--network host`).
- **`YAAK_WEB_RATE_LIMIT_PER_MINUTE`** defaults to 120 sends per client IP,
which suits a public instance and not a team of your own; `0` disables it.
Behind a reverse proxy, add `YAAK_WEB_TRUST_FORWARDED_FOR=true` so the rate
limit sees real client addresses instead of its own — and only then, since
otherwise anyone can spoof the header. If the reverse proxy buffers responses,
tell it not to: sends are streamed, and the `X-Accel-Buffering: no` header this
binary sets is honoured by nginx-shaped ones.
## Running it from source
```shell
cargo run -p yaak-web -- --serve dist/apps/yaak-client
```
after a `YAAK_TARGET=web SKIP_WASM_BUILD=1 npx vp -C apps/yaak-client build`.
Without `--serve` it is the send executor alone, which is what the frontend
dev server wants:
```shell
cargo run -p yaak-web
YAAK_TARGET=web npm run dev --workspace @yaakapp/yaak-client
```
The dev server passes `/v1` through to this binary, so a dev build sends to its
own origin exactly like a production build does — one address to open, and no
CORS in the loop. `PORT` moves this server and the dev server follows it. A
production build also sends to its own origin, unless `VITE_YAAK_WEB_URL` was
set when it was built.
Reaching the dev server from another machine is `HOST=0.0.0.0`. Vite allows
addresses but not names, so opening it as a hostname also needs
`ALLOWED_HOSTS=that-name`.
## Configuration
Every flag has an environment variable, so a container needs no arguments;
`--help` lists them all. The listen address uses the platform conventions —
`HOST` and `PORT` — so the published image runs unchanged on a
platform-as-a-service that assigns a port. Everything else is `YAAK_WEB_*`.
| Flag | Default | What |
| -------------------------- | ----------- | --------------------------------------------------------------------------------------------- |
| `--serve` | off | Also serve a built web client from this directory, on the same origin. |
| `--host` | `127.0.0.1` | Interface to listen on. The image sets `0.0.0.0`. May be a name; `localhost` resolves. |
| `--port` | `9227` | Port to listen on. The image sets `8080`. |
| `--allow-private-networks` | off | Allow sends to loopback, private and link-local addresses. |
| `--allowed-origins` | `*` | CORS origins, comma-separated. Unused when the app is served from here: same origin, no CORS. |
| `--max-request-bytes` | 16 MiB | Largest rendered request accepted from the tab. |
| `--max-response-bytes` | 64 MiB | Largest upstream body relayed before the send is cut off. |
| `--max-timeout-secs` | 60 | Ceiling on a send's timeout; a request asking for more (or none) gets this. |
| `--rate-limit-per-minute` | 120 | Sends per client IP per minute; 0 disables. |
| `--max-concurrent` | 256 | Sends in flight at once. |
| `--trust-forwarded-for` | off | Take the client IP from `X-Forwarded-For`. Only behind a load balancer that sets it. |
## Logging
Default logs contain a startup message and operational warnings and errors, not request URLs, client IPs,
DNS lookups, or per-request timing. To diagnose a self-hosted instance, opt in with
`RUST_LOG=warn,yaak_http=error,yaak_web=debug`. This includes request URLs and IPs,
so only enable it while debugging. Hosting infrastructure may maintain its own
access logs independently.
## Serving the app
`--serve DIR` puts a file server behind the API routes: `/v1/*` is matched
first, everything else comes from `DIR`, and a path with no file behind it gets
`index.html` so the app's own routes survive a refresh. Responses are compressed
(gzip or zstd) on the fly. `/assets/*` is cached forever — Vite content-hashes
those names — and everything else is `no-cache`, so a new deploy arrives on the
next reload.
Serving files changes nothing about sending: the same rendered request, the same
destination policy, the same stateless executor. It exists so that a
self-hosted Yaak is one thing to run rather than two.
## Split deployments
The app and the sender can still be separate services — one CDN-hosted bundle and
one server elsewhere, or one server shared by several fronts. Then the bundle has
to be told where to send, at build time:
```shell
docker build -f Dockerfile.web \
--build-arg VITE_YAAK_WEB_URL=https://send.example.com .
```
and the server needs the CORS origins its callers use, since the requests are no
longer same-origin:
```shell
docker run -p 8080:8080 \
-e YAAK_WEB_ALLOWED_ORIGINS=https://yaak.example.com \
ghcr.io/mountain-loop/yaak-web \
yaak-web
```
The trailing `yaak-web` is a command override: the same image run without
`--serve`, so it executes sends and serves no app.
## What it refuses, and why
A hosted sender is, by construction, a machine that makes HTTP requests on
behalf of strangers. Left alone that is an open relay into whatever network it
sits on. So by default it refuses to connect to:
- loopback (`127/8`, `::1`), private (`10/8`, `172.16/12`, `192.168/16`,
`fc00::/7`), link-local (`169.254/16` — where cloud metadata lives — and
`fe80::/10`), carrier-grade NAT, multicast, reserved and unspecified ranges,
IPv4 addresses carried inside IPv6 forms (`::ffff:a.b.c.d`, the well-known
NAT64 prefix, 6to4), and the whole NAT64 local-use range;
- anything not `http://` or `https://`.
The check runs **on the resolved addresses, after DNS**, for every hop of a
redirect chain, so a public hostname that points at an internal address is
caught, and so is a `Location:` header that points at one. It also refuses body
types that would read files on its own disk (`binary`, multipart file
fields), since no browser tab could legitimately mean those.
Refusal reasons are included in debug logs when enabled. On a public instance
(`web.yaak.app`, or anything else strangers can reach), private-network protection
must stay on: the machine's private
network is the host's, not the user's, so a `localhost` or LAN API is not the
user's to reach through it — the desktop app is what reaches those. On an
instance you run for yourself, that reasoning is inverted, and
`--allow-private-networks` inverts the policy with it. It allows every range
above, including `169.254.169.254`, so use it only where the network on the
other side is one the users are entitled to.
There is no authentication either way: an instance is anonymous, protected by
the per-client rate limit and the destination policy. Anything more (a shared
token, per-user quotas) is a later slice and would sit in front of `send_http`
in `lib.rs`, or in middleware on the exported router. Put TLS in front of a public instance.
## The wire
`POST /v1/http/send` with a JSON body:
```json
{
"request": { "url": "https://…", "method": "GET", "headers": […], "body": {…}, "bodyType": null, "urlParameters": […] },
"settings": { "validateCertificates": true, "followRedirects": true, "timeoutMs": 0, "sendCookies": true, "storeCookies": true },
"cookies": [ … ]
}
```
`request` is a Yaak `HttpRequest` in the desktop's own model shape with every
template already rendered by the tab; the server builds the URL, headers and
body from it exactly the way the desktop does after rendering. `cookies` is the
jar's contents (or `null` for no jar).
The reply is `application/x-ndjson`, one JSON frame per line, in the order things
happened:
| `type` | When | Carries |
| ---------- | ----------------------------------------- | ---------------------------------------------------------------------------------- |
| `event` | as the engine produces them | one timeline event, in the desktop's `http_response_event.event` shape |
| `response` | once, when the final hop's headers arrive | status, all headers, request headers as sent, remote address, HTTP version, timing |
| `body` | as the body is read | a decompressed chunk, base64 |
| `done` | last, on success | elapsed, byte counts, and the cookie jar as the send left it |
| `error` | last, on failure | the reason, and any cookies collected before the failure |
Refusals that happen before anything is sent (a blocked destination, a bad body,
rate limit, capacity) are plain HTTP errors (`403`, `400`, `429`, `503`) with
`{"error": "…"}`, not streams.
Why a streamed HTTP response and not a WebSocket: one `POST` is stateless by
construction, cancellable by closing the connection, readable with `curl`, and
needs no upgrade handling on either side. A WebSocket only earns its keep when
traffic is bidirectional, which a single send is not.
The TypeScript side of this contract is generated from `src/wire.rs` by ts-rs
into `bindings/` (run `cargo test -p yaak-web` after changing a frame)
and published to the tab as `@yaakapp-internal/web`, so a change to the
wire on one side is a type error on the other.
`GET /v1/health` reports the version and the effective limits.
## What comes later
Not built, by design, but the router is shaped for it: a WebSocket relay
(`/v1/ws/relay`) and a gRPC relay (`/v1/grpc/relay`) would be long-lived,
bidirectional endpoints on the same binary, behind the same destination policy
and limits. They differ from this endpoint in holding per-connection
in-memory state while a connection is open (never persisted), which brings
connection limits and a larger abuse surface — the reason they are separate
work.

135
crates-server/yaak-web/bindings/gen_models.ts generated Normal file
View File

@@ -0,0 +1,135 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export type Cookie = {
name: string;
value: string;
domain: CookieDomain;
expires: CookieExpires;
path: string;
secure: boolean;
httpOnly: boolean;
sameSite: CookieSameSite | null;
};
export type CookieDomain = { HostOnly: string } | { Suffix: string } | "NotPresent" | "Empty";
export type CookieExpires = { AtUtc: string } | "SessionEnd";
export type CookieSameSite = "Strict" | "Lax" | "None";
export type HttpRequest = {
model: "http_request";
id: string;
createdAt: string;
updatedAt: string;
workspaceId: string;
folderId: string | null;
authentication: Record<string, any>;
authenticationType: string | null;
body: Record<string, any>;
bodyType: string | null;
description: string;
headers: Array<HttpRequestHeader>;
method: string;
name: string;
sortPriority: number;
url: string;
/**
* URL parameters used for both path placeholders (`:id`) and query string entries.
*/
urlParameters: Array<HttpUrlParameter>;
settingSendCookies: InheritedBoolSetting;
settingStoreCookies: InheritedBoolSetting;
settingValidateCertificates: InheritedBoolSetting;
settingFollowRedirects: InheritedBoolSetting;
settingRequestTimeout: InheritedIntSetting;
settingHttpVersion: InheritedHttpVersionSetting;
};
export type HttpRequestHeader = { enabled?: boolean; name: string; value: string; id?: string };
/**
* Serializable representation of HTTP response events for DB storage.
* This mirrors `yaak_http::sender::HttpResponseEvent` but with serde support.
* The `From` impl is in yaak-http to avoid circular dependencies.
*/
export type HttpResponseEventData =
| {
type: "setting";
name: string;
value: string;
source_model?: string;
source_id?: string;
source_name?: string;
}
| { type: "info"; message: string }
| {
type: "redirect";
url: string;
status: number;
behavior: string;
dropped_body: boolean;
dropped_headers: Array<string>;
}
| {
type: "send_url";
method: string;
scheme: string;
username: string;
password: string;
host: string;
port: number;
path: string;
query: string;
fragment: string;
}
| { type: "receive_url"; version: string; status: string }
| { type: "header_up"; name: string; value: string }
| { type: "header_down"; name: string; value: string }
| { type: "chunk_sent"; bytes: number }
| { type: "chunk_received"; bytes: number }
| {
type: "dns_resolved";
hostname: string;
addresses: Array<string>;
duration: bigint;
overridden: boolean;
};
export type HttpResponseHeader = { name: string; value: string };
/**
* The resolved send settings, values only: what an executor has to obey, with the sources
* (which model each came from) left behind in [`ResolvedHttpRequestSettings`]. This is what
* crosses from a tab to the Yaak server, and what the server reads.
*/
export type HttpSendSettings = {
validateCertificates: boolean;
followRedirects: boolean;
/**
* Milliseconds. Zero or negative means no timeout.
*/
timeoutMs: number;
sendCookies: boolean;
storeCookies: boolean;
httpVersion: HttpVersion;
};
export type HttpUrlParameter = {
enabled?: boolean;
/**
* Colon-prefixed parameters are treated as path parameters if they match, like `/users/:id`
* Other entries are appended as query parameters
*/
name: string;
value: string;
id?: string;
};
export type HttpVersion = "auto" | "http1" | "http2";
export type InheritedBoolSetting = { enabled?: boolean; value: boolean };
export type InheritedHttpVersionSetting = { enabled?: boolean; value: HttpVersion };
export type InheritedIntSetting = { enabled?: boolean; value: number };

64
crates-server/yaak-web/bindings/gen_web.ts generated Normal file
View File

@@ -0,0 +1,64 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { Cookie, HttpRequest, HttpResponseEventData, HttpResponseHeader, HttpSendSettings } from "./gen_models";
/**
* One line of the reply stream. Tags are snake_case like the timeline event tags; fields are
* camelCase like every model the tab stores.
*/
export type Frame = { "type": "event", event: HttpResponseEventData, } | { "type": "response", status: number, statusReason: string | null,
/**
* The URL that answered, after redirects.
*/
url: string, remoteAddr: string | null, version: string | null, headers: Array<HttpResponseHeader>,
/**
* The headers that were actually sent on the final hop, cookies and all.
*/
requestHeaders: Array<HttpResponseHeader>,
/**
* `Content-Length` as declared by the server, if it declared one.
*/
contentLength: number | null,
/**
* Milliseconds from the start of the send to the response head.
*/
elapsedHeaders: number,
/**
* Milliseconds spent in DNS on the last lookup, or zero.
*/
elapsedDns: number, } | { "type": "body", data: string, } | { "type": "done",
/**
* Milliseconds from the start of the send to the end of the body.
*/
elapsed: number,
/**
* Bytes of body relayed, after decompression.
*/
contentLength: number,
/**
* Bytes on the wire as declared by the server, or the relayed size when unknown.
*/
contentLengthCompressed: number,
/**
* The jar as the send left it, for the tab to persist. `None` when the tab sent none.
*/
cookies: Array<Cookie> | null, } | { "type": "error", message: string, cookies: Array<Cookie> | null, };
/**
* The body of `POST /v1/http/send`.
*/
export type SendRequest = {
/**
* The request to send, in the desktop's own model shape but with every template already
* rendered by the tab. The server builds the URL, headers and body from it exactly the way
* the desktop does after rendering.
*/
request: HttpRequest,
/**
* The resolved settings, values only. Where they came from is the tab's to record in
* its timeline; the server only needs to obey them.
*/
settings: HttpSendSettings,
/**
* The cookies to start with. `None` means no jar at all: nothing sent, nothing kept.
*/
cookies: Array<Cookie> | null, };

4
crates-server/yaak-web/index.ts Normal file
View File

@@ -0,0 +1,4 @@
// The server's wire contract, generated by ts-rs from src/wire.rs
// (`cargo test -p yaak-web`). The tab imports these so a change to a
// frame on the Rust side is a type error in packages/platform/src/web.
export type { Frame, SendRequest } from "./bindings/gen_web";

6
crates-server/yaak-web/package.json Normal file
View File

@@ -0,0 +1,6 @@
{
"name": "@yaakapp-internal/web",
"version": "1.0.0",
"private": true,
"main": "index.ts"
}

98
crates-server/yaak-web/src/config.rs Normal file
View File

@@ -0,0 +1,98 @@
use clap::Parser;
use std::net::{SocketAddr, ToSocketAddrs};
use std::path::PathBuf;
/// The server behind Yaak running in a browser.
///
/// The tab renders the request and owns the data; this binary puts the bytes on the network
/// and streams back what came back, and with `--serve` hands the browser the app as well.
/// Nothing is written to disk or a database.
#[derive(Parser, Debug, Clone)]
#[command(name = "yaak-web", version, about, long_about = None)]
pub struct Config {
/// Interface to listen on. 127.0.0.1 for a local instance; 0.0.0.0 inside a container,
/// which is what makes the port reachable from outside it.
#[arg(long, env = "HOST", default_value = "127.0.0.1")]
pub host: String,
/// Port to listen on. Named `PORT` because that is the variable a platform-as-a-service
/// assigns and expects to be obeyed, so the same image runs there and under `docker run`
/// with nothing changed.
#[arg(long, env = "PORT", default_value_t = 9227)]
pub port: u16,
/// Port the app is served on elsewhere. A browser that opens this server directly is
/// redirected there, keeping whatever hostname it used — so `home:9227` becomes
/// `home:1424` without this server knowing what "home" is.
///
/// Set by `vp run web:dev`, where the app is on the dev server and this is only the
/// executor behind it. Irrelevant with `--serve`, which puts both on one port.
#[arg(long, env = "YAAK_WEB_APP_PORT")]
pub app_port: Option<u16>,
/// Also serve a built web client from this directory, on the same origin as the API.
/// Unknown paths fall back to `index.html` so the app's own routes work on a refresh.
/// Without this the binary is only the send executor.
#[arg(long, env = "YAAK_WEB_SERVE", value_name = "DIR")]
pub serve: Option<PathBuf>,
/// Allow sends to loopback, private and link-local addresses. Off by default, because a
/// server reachable by strangers is an open relay into the network it sits on. Turn it on
/// only for an instance whose users are meant to reach that network — a self-hosted one
/// on a LAN, where the point is to call the API on the next machine.
#[arg(long, env = "YAAK_WEB_ALLOW_PRIVATE_NETWORKS", default_value_t = false)]
pub allow_private_networks: bool,
/// Browser origins allowed to call this server (CORS), comma-separated. `*` allows any.
/// A local dev instance wants the Vite origin; a hosted instance wants its own web origin.
#[arg(
long,
env = "YAAK_WEB_ALLOWED_ORIGINS",
default_value = "*",
value_delimiter = ','
)]
pub allowed_origins: Vec<String>,
/// Largest request the server accepts from the tab (the rendered request JSON, body included).
#[arg(long, env = "YAAK_WEB_MAX_REQUEST_BYTES", default_value_t = 16 * 1024 * 1024)]
pub max_request_bytes: usize,
/// Largest upstream response body the server will relay before cutting the send off.
#[arg(long, env = "YAAK_WEB_MAX_RESPONSE_BYTES", default_value_t = 64 * 1024 * 1024)]
pub max_response_bytes: usize,
/// Ceiling on a send's timeout, in seconds. A request asking for longer (or for no timeout)
/// gets this instead.
#[arg(long, env = "YAAK_WEB_MAX_TIMEOUT_SECS", default_value_t = 60)]
pub max_timeout_secs: u64,
/// Sends allowed per client IP per minute. 0 disables the limit. This and the concurrency
/// cap are the whole of what protects an instance: there is no authentication.
#[arg(long, env = "YAAK_WEB_RATE_LIMIT_PER_MINUTE", default_value_t = 120)]
pub rate_limit_per_minute: u32,
/// Sends in flight at once across all clients.
#[arg(long, env = "YAAK_WEB_MAX_CONCURRENT", default_value_t = 256)]
pub max_concurrent: usize,
/// Take the client IP from `X-Forwarded-For` (first hop) instead of the socket. Only turn
/// this on behind a load balancer that sets the header; otherwise anyone can spoof their way
/// past the rate limit.
#[arg(long, env = "YAAK_WEB_TRUST_FORWARDED_FOR", default_value_t = false)]
pub trust_forwarded_for: bool,
}
impl Config {
/// The address to listen on.
///
/// Resolved rather than parsed, so `HOST` may be a name — `localhost` is the one people
/// actually type, and it is not an `IpAddr`. The first address wins; a host resolving to
/// several is a machine with several interfaces, and any of them is a listen address.
pub fn listen_addr(&self) -> Result<SocketAddr, String> {
(self.host.as_str(), self.port)
.to_socket_addrs()
.map_err(|e| format!("Invalid HOST {:?}: {e}", self.host))?
.next()
.ok_or_else(|| format!("HOST {:?} resolved to no address", self.host))
}
}

271
crates-server/yaak-web/src/guard.rs Normal file
View File

@@ -0,0 +1,271 @@
//! Where a send may go.
//!
//! A hosted sender is, by construction, a machine that makes HTTP requests on
//! behalf of strangers. Left alone that is an open relay into whatever network
//! it sits on: cloud metadata endpoints, internal admin panels, the database
//! next door. So every destination is checked twice — once on the URL before a
//! hop is attempted (literal IPs, host allow/deny lists) and once on the
//! addresses a hostname actually resolves to, right before the connection is
//! made. The second check is the one that matters for a hostname pointing at
//! an internal address, and it runs on every redirect hop because the engine
//! resolves every hop.
use async_trait::async_trait;
use log::debug;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::sync::Arc;
use tokio::sync::mpsc;
use url::Url;
use yaak_http::dns::AddressFilter;
use yaak_http::sender::{HttpResponse, HttpResponseEvent, HttpSender};
use yaak_http::types::SendableHttpRequest;
/// The destination policy, shared by every send: public addresses only, unless the operator
/// has said otherwise. A hosted server's "private network" is the cloud's, not the user's, so
/// the default is public-only; a self-hosted instance on a LAN can be told that its private
/// network *is* the user's, which is what `--allow-private-networks` means.
#[derive(Clone, Default)]
pub struct DestinationPolicy {
allow_private: bool,
}
impl DestinationPolicy {
pub fn new(allow_private: bool) -> Self {
Self { allow_private }
}
/// Check a URL before a hop is attempted: scheme and literal IPs. A hostname that passes
/// here still has its resolved addresses checked by [`Self::address_filter`].
pub fn check_url(&self, raw: &str) -> Result<(), String> {
let url = Url::parse(raw).map_err(|e| format!("Invalid URL {raw:?}: {e}"))?;
match url.scheme() {
"http" | "https" => {}
other => return Err(format!("Refusing to send over {other:?}; only http and https")),
}
let host = url.host_str().ok_or_else(|| format!("URL {raw:?} has no host"))?;
let host = host.trim_matches(|c| c == '[' || c == ']');
// A literal IP never reaches the resolver, so it is checked here. Hostnames are checked
// where their addresses become known.
if let Ok(ip) = host.parse::<IpAddr>() {
self.check_ip(ip)?;
}
Ok(())
}
/// The veto the engine's resolver applies to every address a hostname resolves to.
pub fn address_filter(&self) -> AddressFilter {
let policy = self.clone();
Arc::new(move |ip| policy.check_ip(ip))
}
pub fn check_ip(&self, ip: IpAddr) -> Result<(), String> {
if self.allow_private {
return Ok(());
}
match non_public_reason(ip) {
Some(reason) => Err(format!(
"Refusing to connect to {ip}: {reason}. This server only sends to public addresses"
)),
None => Ok(()),
}
}
}
/// Why an address is not a public internet address, or `None` if it is one.
///
/// Every range here is one a hosted relay must never be talked into reaching: the machine
/// itself, the network it sits on, and the link-local range where cloud metadata services
/// (169.254.169.254) live. IPv4 addresses carried inside fixed-layout IPv6 forms — IPv4-mapped,
/// the well-known NAT64 prefix, 6to4 — are unwrapped and judged as IPv4, since that is where
/// the packets end up; the NAT64 local-use range is refused outright. This is the stable-Rust
/// stand-in for `IpAddr::is_global`, which is still behind `#![feature(ip)]`; a network-specific
/// NAT64 prefix is not knowable here.
pub fn non_public_reason(ip: IpAddr) -> Option<&'static str> {
match ip {
IpAddr::V4(v4) => non_public_v4(v4),
IpAddr::V6(v6) => {
if let Some(v4) = v6.to_ipv4_mapped() {
return non_public_v4(v4);
}
if let Some(v4) = embedded_v4(&v6) {
return non_public_v4(v4);
}
if v6.is_loopback() {
Some("loopback")
} else if v6.is_unspecified() {
Some("unspecified")
} else if v6.is_unique_local() {
Some("unique local (fc00::/7)")
} else if v6.is_unicast_link_local() {
Some("link-local (fe80::/10)")
} else if v6.is_multicast() {
Some("multicast")
} else if v6.segments()[..3] == [0x64, 0xff9b, 1] {
Some("NAT64 local-use (64:ff9b:1::/48)")
} else if v6.segments()[..4] == [0x100, 0, 0, 0] {
Some("discard-only (100::/64)")
} else if (v6.segments()[0] & 0xffc0) == 0xfec0 {
Some("site-local (fec0::/10)")
} else if v6.segments()[0] == 0x2001 && v6.segments()[1] == 0x0db8 {
Some("documentation (2001:db8::/32)")
} else {
None
}
}
}
}
fn non_public_v4(v4: Ipv4Addr) -> Option<&'static str> {
let o = v4.octets();
if v4.is_loopback() {
Some("loopback (127.0.0.0/8)")
} else if v4.is_private() {
Some("private (10/8, 172.16/12, 192.168/16)")
} else if v4.is_link_local() {
Some("link-local (169.254.0.0/16, where cloud metadata lives)")
} else if v4.is_unspecified() || o[0] == 0 {
Some("this network (0.0.0.0/8)")
} else if o[0] == 100 && (o[1] & 0xc0) == 64 {
Some("carrier-grade NAT (100.64.0.0/10)")
} else if v4.is_broadcast() {
Some("broadcast")
} else if v4.is_multicast() {
Some("multicast (224.0.0.0/4)")
} else if o[0] >= 240 {
Some("reserved (240.0.0.0/4)")
} else if v4.is_documentation() {
Some("documentation")
} else if o[0] == 192 && o[1] == 0 && o[2] == 0 {
Some("IETF protocol assignments (192.0.0.0/24)")
} else if o[0] == 198 && (o[1] & 0xfe) == 18 {
Some("benchmarking (198.18.0.0/15)")
} else {
None
}
}
/// The IPv4 address an IPv6 address stands for, when it is one of the fixed-layout translation
/// forms: the NAT64 well-known prefix (64:ff9b::/96) or 6to4 (2002::/16, IPv4 in the next 32
/// bits). The NAT64 local-use range (64:ff9b:1::/48) is a pool operators carve their own
/// prefix from, at a length only they know, so it is refused wholesale in [`non_public_reason`]
/// rather than decoded — the same call `std`'s (still unstable) `Ipv6Addr::is_global` makes.
fn embedded_v4(v6: &Ipv6Addr) -> Option<Ipv4Addr> {
let s = v6.segments();
let o = v6.octets();
if s[0] == 0x64 && s[1] == 0xff9b && s[2..6].iter().all(|x| *x == 0) {
return Some(Ipv4Addr::new(o[12], o[13], o[14], o[15]));
}
if s[0] == 0x2002 {
return Some(Ipv4Addr::new(o[2], o[3], o[4], o[5]));
}
None
}
/// An [`HttpSender`] that checks each hop's URL against the policy before delegating.
///
/// The engine's redirect loop calls the sender once per hop with the hop's URL, so wrapping
/// the sender is what makes `Location:` headers subject to the same rules as the first URL —
/// including a redirect to a literal internal IP, which the resolver would never see.
pub struct GuardedSender<S> {
inner: S,
policy: DestinationPolicy,
}
impl<S: HttpSender> GuardedSender<S> {
pub fn new(inner: S, policy: DestinationPolicy) -> Self {
Self { inner, policy }
}
}
#[async_trait]
impl<S: HttpSender> HttpSender for GuardedSender<S> {
async fn send(
&self,
request: SendableHttpRequest,
event_tx: mpsc::Sender<HttpResponseEvent>,
) -> yaak_http::error::Result<HttpResponse> {
if let Err(reason) = self.policy.check_url(&request.url) {
debug!("Refused {} {}: {reason}", request.method, request.url);
return Err(yaak_http::error::Error::RequestError(reason));
}
self.inner.send(request, event_tx).await
}
}
#[cfg(test)]
mod tests {
use super::*;
fn ip(s: &str) -> IpAddr {
s.parse().unwrap()
}
#[test]
fn refuses_the_ranges_a_relay_must_never_reach() {
for addr in [
"127.0.0.1",
"127.9.9.9",
"10.0.0.1",
"172.16.0.1",
"172.31.255.255",
"192.168.1.1",
"169.254.169.254",
"169.254.0.1",
"0.0.0.0",
"100.64.0.1",
"255.255.255.255",
"224.0.0.1",
"240.0.0.1",
"::1",
"::",
"fc00::1",
"fd12::1",
"fe80::1",
"::ffff:127.0.0.1",
"::ffff:169.254.169.254",
"64:ff9b::7f00:1",
"ff02::1",
] {
assert!(non_public_reason(ip(addr)).is_some(), "{addr} should be refused");
}
}
#[test]
fn allows_public_addresses() {
for addr in [
"1.1.1.1",
"8.8.8.8",
"93.184.216.34",
"172.32.0.1",
"2606:4700:4700::1111",
] {
assert!(non_public_reason(ip(addr)).is_none(), "{addr} should be allowed");
}
}
#[test]
fn literal_private_addresses_in_urls_are_refused() {
let policy = DestinationPolicy::new(false);
assert!(policy.check_url("http://127.0.0.1/").is_err());
assert!(policy.check_url("http://[::1]/").is_err());
assert!(policy.check_url("http://169.254.169.254/latest/meta-data").is_err());
}
#[test]
fn allow_private_networks_opens_the_local_ranges_but_not_other_schemes() {
let policy = DestinationPolicy::new(true);
assert!(policy.check_url("http://127.0.0.1/").is_ok());
assert!(policy.check_ip(ip("10.0.0.1")).is_ok());
assert!(policy.check_ip(ip("169.254.169.254")).is_ok());
assert!(policy.check_url("file:///etc/passwd").is_err());
}
#[test]
fn only_http_schemes() {
let policy = DestinationPolicy::new(false);
assert!(policy.check_url("ftp://example.com/").is_err());
assert!(policy.check_url("file:///etc/passwd").is_err());
assert!(policy.check_url("https://example.com/").is_ok());
}
}

295
crates-server/yaak-web/src/lib.rs Normal file
View File

@@ -0,0 +1,295 @@
//! yaak-web: the network half of Yaak in a browser.
//!
//! A tab can't see a response the way a desktop app can — CORS hides most
//! headers, redirects are followed silently, there is no timeline. So the tab
//! renders the request and hands it here; this process puts it on the network
//! with the desktop's own engine and streams back everything that happened,
//! for the tab to store. It keeps nothing: no database, no files, no session.
//!
//! Embed [`router`] in your own Axum server, or run the `yaak-web` binary
//! configured by flags or `YAAK_WEB_*` environment variables.
//! See README.md for running and deploying it, and `guard.rs` for what it
//! refuses to talk to.
mod config;
mod guard;
mod limits;
mod send;
mod wire;
use axum::Router;
use axum::body::Body;
use axum::extract::{ConnectInfo, DefaultBodyLimit, Request, State};
use axum::http::{HeaderMap, HeaderValue, Method, StatusCode, header};
use axum::middleware::{self, Next};
use axum::response::{IntoResponse, Json, Redirect, Response};
use axum::routing::{get, post};
pub use config::Config;
use guard::DestinationPolicy;
use limits::RateLimiter;
use log::{debug, info, warn};
use send::{Refusal, SendLimits};
use serde_json::json;
use std::net::{IpAddr, SocketAddr};
use std::path::Path;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::Semaphore;
use tower_http::compression::CompressionLayer;
use tower_http::cors::{AllowOrigin, CorsLayer};
use tower_http::services::{ServeDir, ServeFile};
use wire::SendRequest;
/// Minimal operational logging; request diagnostics require an explicit `RUST_LOG` setting.
pub const DEFAULT_LOG_FILTER: &str = "warn,yaak_http=error,yaak_web::startup=info";
#[derive(Clone)]
struct AppState {
config: Arc<Config>,
limits: Arc<SendLimits>,
rate_limiter: Arc<RateLimiter>,
in_flight: Arc<Semaphore>,
}
/// Build the API router and, when configured, the web client's static-file fallback.
///
/// The returned router owns its state and accepts normal Axum middleware via `.layer()`.
/// Construction does not parse arguments, initialize logging, bind a socket, or install
/// shutdown handlers. `config.host` and `config.port` are used only by the standalone binary.
///
/// Serve with `into_make_service_with_connect_info::<SocketAddr>()` so the send endpoint
/// can extract the peer address for rate limiting.
pub fn router(config: Config) -> Router {
let policy = DestinationPolicy::new(config.allow_private_networks);
if config.allow_private_networks {
warn!(
"Sends to loopback, private and link-local addresses are ALLOWED. Only run this way \
on an instance strangers cannot reach"
);
}
let state = AppState {
limits: Arc::new(SendLimits {
policy,
max_response_bytes: config.max_response_bytes,
max_timeout: Duration::from_secs(config.max_timeout_secs),
}),
rate_limiter: Arc::new(RateLimiter::new(config.rate_limit_per_minute)),
in_flight: Arc::new(Semaphore::new(config.max_concurrent)),
config: Arc::new(config),
};
let cors = CorsLayer::new()
.allow_methods([Method::GET, Method::POST, Method::OPTIONS])
.allow_headers([header::CONTENT_TYPE])
.allow_origin(allowed_origins(&state.config.allowed_origins));
let api = Router::new()
.route("/v1/health", get(health))
// A WebSocket or gRPC relay would sit beside this as `/v1/ws/relay` and `/v1/grpc/relay`
// on the same router, behind the same policy, limits and auth. Not built; see README.
.route("/v1/http/send", post(send_http))
.layer(DefaultBodyLimit::max(state.config.max_request_bytes))
.layer(cors)
.with_state(state.clone());
match &state.config.serve {
Some(dir) => {
info!("Serving the web client from {}", dir.display());
api.merge(web_router(dir))
}
// Without `--serve` there is no app here, only `/v1`. Someone who opens this
// port in a browser guessed wrong about which of the two dev servers hosts the
// app, so send them to the right one when we have been told where it is.
None => {
let app_port = state.config.app_port;
api.fallback(move |headers: HeaderMap| async move { no_app_here(app_port, headers) })
}
}
}
/// The built web client, served on the same origin as the API.
///
/// This is what makes a single container zero-configuration: the tab's send URL is a path on
/// the page's own origin, so there is no CORS, no second service and no URL to bake in. It is
/// only a file server — a send behaves exactly as it does without this flag.
///
/// Merged as a fallback, so the `/v1` routes are matched first and a request that matches no
/// file at all gets `index.html` (the app routes client-side; a deep link must survive a
/// refresh).
fn web_router(dir: &Path) -> Router {
let index = ServeFile::new(dir.join("index.html"));
Router::new()
// `fallback`, not `not_found_service`: the app's own routes are real pages, so
// index.html is served with the 200 the browser expects, not a 404 carrying HTML.
.fallback_service(ServeDir::new(dir).fallback(index))
.layer(middleware::from_fn(cache_control))
.layer(CompressionLayer::new())
}
/// Vite gives everything in `/assets` a content-hashed name, so those can be cached forever.
/// Everything else — `index.html` above all, including the copy served for an unknown path —
/// must be revalidated, or a browser keeps serving the deploy before last.
async fn cache_control(req: Request, next: Next) -> Response {
let hashed_name = req.uri().path().starts_with("/assets/");
let mut res = next.run(req).await;
if !res.status().is_success() {
return res;
}
let is_html = res
.headers()
.get(header::CONTENT_TYPE)
.and_then(|v| v.to_str().ok())
.is_some_and(|v| v.starts_with("text/html"));
let value =
if hashed_name && !is_html { "public, max-age=31536000, immutable" } else { "no-cache" };
res.headers_mut().insert(header::CACHE_CONTROL, HeaderValue::from_static(value));
res
}
/// The name in a `Host` header, without the port this server is reached on.
///
/// An IPv6 literal keeps its brackets and its own colons: the last colon only separates a
/// port when what follows it isn't part of the address, which is what the `]` test decides.
fn host_without_port(host: &str) -> &str {
match host.rfind(':') {
Some(i) if !host[i..].contains(']') => &host[..i],
_ => host,
}
}
/// What the send executor does with a browser that came looking for the app.
///
/// With an app port configured this is a redirect, built from the `Host` header so the
/// hostname the browser already used is the one it keeps — this server has no idea which of
/// its addresses someone typed, and does not need to.
fn no_app_here(app_port: Option<u16>, headers: HeaderMap) -> Response {
if let Some(port) = app_port
&& let Some(host) = headers.get(header::HOST).and_then(|v| v.to_str().ok())
{
return Redirect::temporary(&format!("http://{}:{port}/", host_without_port(host)))
.into_response();
}
(
StatusCode::NOT_FOUND,
[(header::CONTENT_TYPE, "text/plain; charset=utf-8")],
"This is the Yaak send executor. It serves the API under /v1 and no app.\n\n\
To serve the app from here too, restart with --serve <DIR> pointing at a built\n\
web client.\n",
)
.into_response()
}
fn allowed_origins(origins: &[String]) -> AllowOrigin {
if origins.iter().any(|o| o.trim() == "*") {
return AllowOrigin::any();
}
let parsed: Vec<HeaderValue> =
origins.iter().filter_map(|o| HeaderValue::from_str(o.trim()).ok()).collect();
AllowOrigin::list(parsed)
}
async fn health(State(state): State<AppState>) -> impl IntoResponse {
Json(json!({
"ok": true,
"version": env!("CARGO_PKG_VERSION"),
"maxResponseBytes": state.config.max_response_bytes,
"maxTimeoutSecs": state.config.max_timeout_secs,
}))
}
fn error_response(status: StatusCode, message: impl Into<String>) -> Response {
let message = message.into();
(status, Json(json!({ "error": message }))).into_response()
}
/// The client's address for rate limiting: the socket peer, or the first `X-Forwarded-For`
/// hop when the operator has said the header can be trusted.
fn client_ip(config: &Config, headers: &HeaderMap, peer: SocketAddr) -> IpAddr {
if config.trust_forwarded_for
&& let Some(forwarded) = headers.get("x-forwarded-for").and_then(|v| v.to_str().ok())
&& let Some(first) = forwarded.split(',').next()
&& let Ok(ip) = first.trim().parse::<IpAddr>()
{
return ip;
}
peer.ip()
}
async fn send_http(
State(state): State<AppState>,
ConnectInfo(peer): ConnectInfo<SocketAddr>,
headers: HeaderMap,
Json(body): Json<SendRequest>,
) -> Response {
let ip = client_ip(&state.config, &headers, peer);
if let Err(wait) = state.rate_limiter.check(ip) {
debug!("Rate limited {ip}");
let mut res = error_response(
StatusCode::TOO_MANY_REQUESTS,
format!("Rate limit reached; try again in {}s", wait.as_secs().max(1)),
);
res.headers_mut().insert(header::RETRY_AFTER, HeaderValue::from(wait.as_secs().max(1)));
return res;
}
let Ok(permit) = state.in_flight.clone().try_acquire_owned() else {
debug!("At capacity; refusing {ip}");
return error_response(StatusCode::SERVICE_UNAVAILABLE, "This server is at capacity");
};
let prepared = match send::prepare(state.limits.clone(), body).await {
Ok(p) => p,
Err(Refusal::Unsupported(m)) => return error_response(StatusCode::BAD_REQUEST, m),
Err(Refusal::Invalid(m)) => return error_response(StatusCode::BAD_REQUEST, m),
Err(Refusal::Destination(m)) => {
debug!("Refused send from {ip}: {m}");
return error_response(StatusCode::FORBIDDEN, m);
}
};
let description = prepared.describe();
debug!("{ip} -> {description}");
let started = Instant::now();
let (tx, rx) = tokio::sync::mpsc::channel(send::FRAME_CHANNEL_CAPACITY);
tokio::spawn(async move {
prepared.run(tx).await;
send::log_outcome(&description, started, "finished");
drop(permit);
});
let stream = tokio_stream_from(rx);
Response::builder()
.status(StatusCode::OK)
.header(header::CONTENT_TYPE, "application/x-ndjson")
.header(header::CACHE_CONTROL, "no-store")
// Some reverse proxies buffer streamed responses unless told not to
.header("x-accel-buffering", "no")
.body(Body::from_stream(stream))
.expect("valid response")
}
fn tokio_stream_from<T: Send + 'static>(
mut rx: tokio::sync::mpsc::Receiver<T>,
) -> impl futures_util::Stream<Item = T> + Send + 'static {
futures_util::stream::poll_fn(move |cx| rx.poll_recv(cx))
}
#[cfg(test)]
mod tests {
use super::host_without_port;
#[test]
fn strips_the_port_and_keeps_the_name() {
assert_eq!(host_without_port("home:9227"), "home");
assert_eq!(host_without_port("home"), "home");
assert_eq!(host_without_port("192.168.1.5:9227"), "192.168.1.5");
assert_eq!(host_without_port("192.168.1.5"), "192.168.1.5");
// An IPv6 literal is full of colons, and only the one outside the brackets is a port.
assert_eq!(host_without_port("[::1]:9227"), "[::1]");
assert_eq!(host_without_port("[::1]"), "[::1]");
assert_eq!(host_without_port("[2606:4700::1111]:8080"), "[2606:4700::1111]");
assert_eq!(host_without_port("[2606:4700::1111]"), "[2606:4700::1111]");
}
}

90
crates-server/yaak-web/src/limits.rs Normal file
View File

@@ -0,0 +1,90 @@
//! Per-client rate limiting, kept deliberately small.
//!
//! One token bucket per client IP, refilled continuously, in a mutex-guarded
//! map that is swept of idle entries as it goes. Good enough to keep one
//! caller from monopolising a hosted instance; not a substitute for whatever
//! sits in front of it in production.
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::Mutex;
use std::time::{Duration, Instant};
pub struct RateLimiter {
per_minute: u32,
buckets: Mutex<HashMap<IpAddr, Bucket>>,
}
struct Bucket {
tokens: f64,
last: Instant,
}
impl RateLimiter {
/// `per_minute == 0` disables limiting.
pub fn new(per_minute: u32) -> Self {
Self { per_minute, buckets: Mutex::new(HashMap::new()) }
}
/// Take one token for `client`, or say how long until one is available.
pub fn check(&self, client: IpAddr) -> Result<(), Duration> {
if self.per_minute == 0 {
return Ok(());
}
let capacity = self.per_minute as f64;
let per_second = capacity / 60.0;
let now = Instant::now();
let mut buckets = self.buckets.lock().unwrap_or_else(|e| e.into_inner());
// Sweep buckets that have been idle long enough to be full again; there is nothing
// to remember about them.
if buckets.len() > 1024 {
buckets.retain(|_, b| now.duration_since(b.last).as_secs_f64() * per_second < capacity);
}
let bucket = buckets.entry(client).or_insert(Bucket { tokens: capacity, last: now });
let elapsed = now.duration_since(bucket.last).as_secs_f64();
bucket.tokens = (bucket.tokens + elapsed * per_second).min(capacity);
bucket.last = now;
if bucket.tokens >= 1.0 {
bucket.tokens -= 1.0;
Ok(())
} else {
let wait = (1.0 - bucket.tokens) / per_second;
Err(Duration::from_secs_f64(wait.max(0.001)))
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_full_bucket_then_a_wait() {
let limiter = RateLimiter::new(3);
let ip: IpAddr = "203.0.113.5".parse().unwrap();
assert!(limiter.check(ip).is_ok());
assert!(limiter.check(ip).is_ok());
assert!(limiter.check(ip).is_ok());
let wait = limiter.check(ip).expect_err("fourth call in a burst should wait");
assert!(wait > Duration::ZERO && wait <= Duration::from_secs(20));
}
#[test]
fn clients_are_independent_and_zero_disables() {
let limiter = RateLimiter::new(1);
let a: IpAddr = "203.0.113.5".parse().unwrap();
let b: IpAddr = "203.0.113.6".parse().unwrap();
assert!(limiter.check(a).is_ok());
assert!(limiter.check(a).is_err());
assert!(limiter.check(b).is_ok());
let unlimited = RateLimiter::new(0);
for _ in 0..1000 {
assert!(unlimited.check(a).is_ok());
}
}
}

35
crates-server/yaak-web/src/main.rs Normal file
View File

@@ -0,0 +1,35 @@
//! Standalone server for Yaak in a browser.
use clap::Parser;
use log::info;
use std::net::SocketAddr;
use yaak_web::{Config, router};
#[tokio::main]
async fn main() {
env_logger::Builder::from_env(
env_logger::Env::default().default_filter_or(yaak_web::DEFAULT_LOG_FILTER),
)
.init();
let config = Config::parse();
let bind = config.listen_addr().unwrap_or_else(|e| {
eprintln!("{e}");
std::process::exit(1);
});
let rate_limit_per_minute = config.rate_limit_per_minute;
let app = router(config);
let listener = tokio::net::TcpListener::bind(bind).await.unwrap_or_else(|e| {
eprintln!("Failed to bind {bind}: {e}");
std::process::exit(1);
});
info!(target: "yaak_web::startup", "yaak-web listening on http://{bind} (rate limit: {}/min)", rate_limit_per_minute,);
axum::serve(listener, app.into_make_service_with_connect_info::<SocketAddr>())
.with_graceful_shutdown(async {
let _ = tokio::signal::ctrl_c().await;
info!("Shutting down");
})
.await
.expect("server error");
}

359
crates-server/yaak-web/src/send.rs Normal file
View File

@@ -0,0 +1,359 @@
//! The one thing this binary does: execute a rendered request and stream back what happened.
//!
//! This is the "execute" half of the desktop's `send_http_request` — the part after rendering
//! and before storage — driven through the same `HttpTransaction` the desktop drives, with the
//! same redirect loop, cookie jar, decompression and timeline events. Everything the desktop
//! would write to its database is written to the reply stream instead, and the tab stores it.
use crate::guard::{DestinationPolicy, GuardedSender};
use crate::wire::{Frame, SendRequest};
use base64::Engine;
use bytes::Bytes;
use log::{debug, warn};
use std::convert::Infallible;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use tokio::io::AsyncReadExt;
use tokio::sync::{mpsc, watch};
use yaak_http::client::{HttpConnectionOptions, HttpConnectionProxySetting};
use yaak_http::cookies::CookieStore;
use yaak_http::sender::{HttpResponseEvent, ReqwestSender};
use yaak_http::transaction::HttpTransaction;
use yaak_http::types::{SendableHttpRequest, SendableHttpRequestOptions};
use yaak_models::models::HttpResponseHeader;
/// How many frames may sit unread by the client before body reading pauses. Backpressure, so a
/// slow tab slows the upstream read rather than filling memory.
pub const FRAME_CHANNEL_CAPACITY: usize = 64;
const EVENT_CHANNEL_CAPACITY: usize = 256;
const BODY_READ_CHUNK: usize = 64 * 1024;
/// What a send needs from the process, beyond the request itself.
pub struct SendLimits {
pub policy: DestinationPolicy,
pub max_response_bytes: usize,
pub max_timeout: Duration,
}
/// Why a send was refused before anything was put on the network. Distinct from a failure
/// mid-stream: these become a plain HTTP error, not a stream with an error frame.
#[derive(Debug)]
pub enum Refusal {
/// The request asks for something a browser-originated send cannot mean.
Unsupported(String),
/// The destination is not one this server will talk to.
Destination(String),
/// The request could not be turned into something sendable.
Invalid(String),
}
pub type FrameSender = mpsc::Sender<Result<Bytes, Infallible>>;
/// Check and prepare a send, then hand back the task that runs it. Refusals happen here, before
/// the caller has committed to a streaming response.
pub async fn prepare(limits: Arc<SendLimits>, send: SendRequest) -> Result<PreparedSend, Refusal> {
let request = send.request;
// The engine reads files for these body types. There are no files here that a browser tab
// could legitimately mean, and letting a request name a path on this machine would be a
// local file read for anyone who can reach the server.
if request.body_type.as_deref() == Some("binary") {
return Err(Refusal::Unsupported(
"Binary file bodies can't be sent from the browser: the server has no access to your files"
.to_string(),
));
}
if request.body_type.as_deref() == Some("multipart/form-data") {
let names_a_file =
request.body.get("form").and_then(|f| f.as_array()).is_some_and(|entries| {
entries.iter().any(|e| {
e.get("enabled").and_then(|v| v.as_bool()).unwrap_or(true)
&& e.get("file").and_then(|v| v.as_str()).is_some_and(|f| !f.is_empty())
})
});
if names_a_file {
return Err(Refusal::Unsupported(
"Multipart file fields can't be sent from the browser: the server has no access to your files"
.to_string(),
));
}
}
// The tab's requested timeout, capped. Zero means "none", which here means the cap.
let requested = if send.settings.timeout_ms > 0 {
Some(Duration::from_millis(send.settings.timeout_ms as u64))
} else {
None
};
let timeout = requested.map_or(limits.max_timeout, |t| t.min(limits.max_timeout));
let timeout_capped = requested.is_none_or(|t| t > limits.max_timeout);
let sendable = SendableHttpRequest::from_http_request(
&request,
SendableHttpRequestOptions {
timeout: Some(timeout),
follow_redirects: send.settings.follow_redirects,
},
)
.await
.map_err(|e| Refusal::Invalid(e.to_string()))?;
// The first hop, checked up front so a bad destination is a clean refusal rather than a
// stream that opens and immediately errors. Every later hop is checked by GuardedSender.
limits.policy.check_url(&sendable.url).map_err(Refusal::Destination)?;
Ok(PreparedSend {
limits,
sendable,
settings: send.settings,
cookies: send.cookies,
timeout,
timeout_capped,
})
}
pub struct PreparedSend {
limits: Arc<SendLimits>,
sendable: SendableHttpRequest,
settings: yaak_models::models::HttpSendSettings,
cookies: Option<Vec<yaak_models::models::Cookie>>,
timeout: Duration,
timeout_capped: bool,
}
impl PreparedSend {
pub fn describe(&self) -> String {
format!("{} {}", self.sendable.method, self.sendable.url)
}
/// Run the send, writing frames to `frames` until the terminal frame. Returns when the
/// stream is complete or the client has gone away.
pub async fn run(mut self, frames: FrameSender) {
let cookie_store = self.cookies.take().map(CookieStore::from_cookies);
let store_for_result = cookie_store.clone();
let outcome = self.execute(frames.clone(), cookie_store).await;
let cookies = store_for_result.as_ref().map(|s| s.get_all_cookies());
let terminal = match outcome {
Ok(done) => Frame::Done {
elapsed: done.elapsed,
content_length: done.content_length,
content_length_compressed: done.content_length_compressed,
cookies,
},
Err(message) => Frame::Error { message, cookies },
};
let _ = write_frame(&frames, &terminal).await;
}
async fn execute(
self,
frames: FrameSender,
cookie_store: Option<CookieStore>,
) -> Result<DoneStats, String> {
let limits = self.limits;
let (client, resolver) = HttpConnectionOptions {
id: uuid::Uuid::new_v4().to_string(),
validate_certificates: self.settings.validate_certificates,
http_version: self.settings.http_version,
// The proxy connects directly. Going through a system proxy would move DNS, and
// therefore the address check, somewhere this process can't see.
proxy: HttpConnectionProxySetting::Disabled,
client_certificate: None,
dns_overrides: Vec::new(),
address_filter: Some(limits.policy.address_filter()),
}
.build_client()
.map_err(|e| format!("Failed to build HTTP client: {e}"))?;
// Timeline events go into the same frame stream as everything else, as they happen.
// The desktop persists them from a task like this one; here the task serialises them.
let (event_tx, mut event_rx) = mpsc::channel::<HttpResponseEvent>(EVENT_CHANNEL_CAPACITY);
resolver.set_event_sender(Some(event_tx.clone())).await;
let dns_elapsed = Arc::new(AtomicU64::new(0));
let event_frames = frames.clone();
let event_dns = dns_elapsed.clone();
let event_task = tokio::spawn(async move {
while let Some(event) = event_rx.recv().await {
if let HttpResponseEvent::DnsResolved { duration, .. } = &event {
event_dns.store(*duration, Ordering::Relaxed);
}
let frame = Frame::Event { event: event.into() };
if write_frame(&event_frames, &frame).await.is_err() {
break;
}
}
});
// Cancellation: the client hanging up, or the overall deadline. The deadline exists
// because a per-hop timeout times each hop separately; ten slow redirects must not add
// up to ten timeouts.
let (cancel_tx, cancel_rx) = watch::channel(false);
let deadline = self.timeout * 2 + Duration::from_secs(5);
let deadline_cancel = cancel_tx.clone();
let deadline_task = tokio::spawn(async move {
tokio::time::sleep(deadline).await;
let _ = deadline_cancel.send(true);
});
let hangup_frames = frames.clone();
let hangup_task = tokio::spawn(async move {
hangup_frames.closed().await;
let _ = cancel_tx.send(true);
});
if self.timeout_capped {
let _ = event_tx.try_send(HttpResponseEvent::Info(format!(
"Timeout set to {:?} (this server's ceiling)",
self.timeout
)));
}
let sender = GuardedSender::new(ReqwestSender::with_client(client), limits.policy.clone());
let transaction = match cookie_store {
Some(store) => HttpTransaction::with_cookie_behavior(
sender,
store,
self.settings.send_cookies,
self.settings.store_cookies,
),
None => HttpTransaction::new(sender),
};
let started_at = Instant::now();
let result = transaction
.execute_with_cancellation(self.sendable, cancel_rx.clone(), event_tx.clone())
.await;
resolver.set_event_sender(None).await;
let mut response = match result {
Ok(response) => response,
Err(err) => {
drop(event_tx);
let _ = event_task.await;
deadline_task.abort();
hangup_task.abort();
return Err(describe_error(&err));
}
};
let elapsed_headers = started_at.elapsed().as_millis() as u64;
let head = Frame::Response {
status: response.status,
status_reason: response.status_reason.clone(),
url: response.url.clone(),
remote_addr: response.remote_addr.clone(),
version: response.version.clone(),
headers: to_wire_headers(&response.headers),
request_headers: to_wire_headers(&response.request_headers),
content_length: response.content_length,
elapsed_headers,
elapsed_dns: dns_elapsed.load(Ordering::Relaxed),
};
write_frame(&frames, &head).await.map_err(|_| "Client went away".to_string())?;
let declared_length = response.content_length;
let mut body = response
.into_body_stream()
.map_err(|e| format!("Failed to read response body: {e}"))?;
let mut buf = vec![0u8; BODY_READ_CHUNK];
let mut total: usize = 0;
let mut cancel_rx = cancel_rx;
let base64 = base64::engine::general_purpose::STANDARD;
let read_result: Result<(), String> = loop {
if *cancel_rx.borrow() {
break Err("Request canceled".to_string());
}
let read = tokio::select! {
biased;
_ = cancel_rx.changed() => break Err("Request canceled".to_string()),
r = body.read(&mut buf) => r,
};
match read {
Ok(0) => break Ok(()),
Ok(n) => {
total += n;
if total > limits.max_response_bytes {
break Err(format!(
"Response body exceeds this server's limit of {} bytes",
limits.max_response_bytes
));
}
let frame = Frame::Body { data: base64.encode(&buf[..n]) };
if write_frame(&frames, &frame).await.is_err() {
break Err("Client went away".to_string());
}
}
Err(e) => break Err(format!("Failed to read response body: {e}")),
}
};
drop(body);
// Let the timeline drain before the terminal frame, so nothing arrives after "done".
drop(event_tx);
let _ = event_task.await;
deadline_task.abort();
hangup_task.abort();
read_result?;
Ok(DoneStats {
elapsed: started_at.elapsed().as_millis() as u64,
content_length: total as u64,
content_length_compressed: declared_length.unwrap_or(total as u64),
})
}
}
/// A send error as a sentence, not a debug dump.
///
/// A connection error from reqwest arrives wrapped several layers deep, and the layer that
/// says something useful — "Refusing to connect to ::1: loopback" — is the innermost. The
/// desktop shows the outer `Debug`; a stranger reading its reply deserves the reason.
fn describe_error(err: &yaak_http::error::Error) -> String {
match err {
yaak_http::error::Error::Client(e) => {
let mut leaf: &dyn std::error::Error = e;
while let Some(next) = leaf.source() {
leaf = next;
}
let outer = e.to_string();
let inner = leaf.to_string();
if inner == outer { outer } else { format!("{outer}: {inner}") }
}
yaak_http::error::Error::RequestError(message) => format!("Request failed: {message}"),
other => other.to_string(),
}
}
struct DoneStats {
elapsed: u64,
content_length: u64,
content_length_compressed: u64,
}
fn to_wire_headers(headers: &[(String, String)]) -> Vec<HttpResponseHeader> {
headers
.iter()
.map(|(name, value)| HttpResponseHeader { name: name.clone(), value: value.clone() })
.collect()
}
async fn write_frame(frames: &FrameSender, frame: &Frame) -> Result<(), ()> {
let mut line = match serde_json::to_vec(frame) {
Ok(v) => v,
Err(e) => {
warn!("Failed to serialize response frame");
debug!("Frame serialization error: {e}");
return Err(());
}
};
line.push(b'\n');
frames.send(Ok(Bytes::from(line))).await.map_err(|_| ())
}
/// Request diagnostics are opt-in, including destinations and completion timing.
pub fn log_outcome(description: &str, started: Instant, outcome: &str) {
debug!("{description} -> {outcome} in {:?}", started.elapsed());
}

97
crates-server/yaak-web/src/wire.rs Normal file
View File

@@ -0,0 +1,97 @@
//! What crosses the wire between a tab and this server.
//!
//! One `POST /v1/http/send` carries a request the tab has already rendered —
//! templates resolved, inheritance applied — plus the send settings and the
//! cookies the send starts with. The reply is a stream of newline-delimited
//! JSON frames: timeline events as they happen, the response head as soon as
//! headers arrive, body chunks as they are read, and one terminal frame.
//!
//! Nothing here names a workspace, a request id, or a response id. The server
//! does not know what the tab will call this response; it only knows what came
//! back.
//!
//! The TypeScript side of this contract is generated from these types into
//! `bindings/` (`cargo test -p yaak-web`) and published to the tab as
//! `@yaakapp-internal/web`, so a change here is a type error there.
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use yaak_models::models::{
Cookie, HttpRequest, HttpResponseEventData, HttpResponseHeader, HttpSendSettings,
};
/// The body of `POST /v1/http/send`.
#[derive(Deserialize, Debug, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export, export_to = "gen_web.ts")]
pub struct SendRequest {
/// The request to send, in the desktop's own model shape but with every template already
/// rendered by the tab. The server builds the URL, headers and body from it exactly the way
/// the desktop does after rendering.
pub request: HttpRequest,
/// The resolved settings, values only. Where they came from is the tab's to record in
/// its timeline; the server only needs to obey them.
pub settings: HttpSendSettings,
/// The cookies to start with. `None` means no jar at all: nothing sent, nothing kept.
#[serde(default)]
pub cookies: Option<Vec<Cookie>>,
}
/// One line of the reply stream. Tags are snake_case like the timeline event tags; fields are
/// camelCase like every model the tab stores.
#[derive(Serialize, Debug, TS)]
#[serde(
tag = "type",
rename_all = "snake_case",
rename_all_fields = "camelCase"
)]
#[ts(export, export_to = "gen_web.ts")]
pub enum Frame {
/// A timeline event, in the same shape the desktop stores. Interleaved with everything
/// else in the order the engine produced it.
Event { event: HttpResponseEventData },
/// The response head. Sent once, as soon as the final hop's headers are in — before any of
/// the body — so the tab can show status and headers while the body streams.
Response {
status: u16,
status_reason: Option<String>,
/// The URL that answered, after redirects.
url: String,
remote_addr: Option<String>,
version: Option<String>,
headers: Vec<HttpResponseHeader>,
/// The headers that were actually sent on the final hop, cookies and all.
request_headers: Vec<HttpResponseHeader>,
/// `Content-Length` as declared by the server, if it declared one.
#[ts(type = "number | null")]
content_length: Option<u64>,
/// Milliseconds from the start of the send to the response head.
#[ts(type = "number")]
elapsed_headers: u64,
/// Milliseconds spent in DNS on the last lookup, or zero.
#[ts(type = "number")]
elapsed_dns: u64,
},
/// A piece of the response body, decompressed, base64-encoded.
Body { data: String },
/// The send finished. The last frame on a successful stream.
Done {
/// Milliseconds from the start of the send to the end of the body.
#[ts(type = "number")]
elapsed: u64,
/// Bytes of body relayed, after decompression.
#[ts(type = "number")]
content_length: u64,
/// Bytes on the wire as declared by the server, or the relayed size when unknown.
#[ts(type = "number")]
content_length_compressed: u64,
/// The jar as the send left it, for the tab to persist. `None` when the tab sent none.
cookies: Option<Vec<Cookie>>,
},
/// The send failed. The last frame on a failed stream. Cookies collected before the failure
/// still come back — the transaction may have set some before the hop that failed.
Error {
message: String,
cookies: Option<Vec<Cookie>>,
},
}

98
crates-server/yaak-web/tests/logging.rs Normal file
View File

@@ -0,0 +1,98 @@
use axum::Router;
use axum::body::{Body, to_bytes};
use axum::extract::{ConnectInfo, Request};
use axum::http::StatusCode;
use axum::routing::get;
use clap::Parser;
use std::io::{self, Write};
use std::net::SocketAddr;
use std::sync::{Arc, Mutex};
use tower::ServiceExt;
use yaak_web::{Config, DEFAULT_LOG_FILTER, router};
#[derive(Clone)]
struct LogCapture(Arc<Mutex<Vec<u8>>>);
impl Write for LogCapture {
fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
self.0.lock().unwrap().extend_from_slice(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
fn send_request(url: &str) -> Request {
let payload = serde_json::json!({
"request": { "url": url, "method": "GET" },
"settings": {
"validateCertificates": true,
"followRedirects": true,
"timeoutMs": 2000,
"sendCookies": true,
"storeCookies": true
}
});
Request::builder()
.method("POST")
.uri("/v1/http/send")
.header("content-type", "application/json")
.extension(ConnectInfo("192.0.2.42:1234".parse::<SocketAddr>().unwrap()))
.body(Body::from(payload.to_string()))
.unwrap()
}
#[tokio::test]
async fn default_logging_keeps_operational_warnings_without_request_details() {
let capture = LogCapture(Arc::new(Mutex::new(Vec::new())));
env_logger::Builder::new()
.parse_filters(DEFAULT_LOG_FILTER)
.target(env_logger::Target::Pipe(Box::new(capture.clone())))
.init();
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let upstream = tokio::spawn(async move {
axum::serve(
listener,
Router::new().route("/private-path", get(|| async { "secret body" })),
)
.await
.unwrap();
});
let mut config = Config::parse_from(["yaak-web"]);
config.allow_private_networks = true;
config.rate_limit_per_minute = 1;
let app = router(config.clone());
assert!(String::from_utf8_lossy(&capture.0.lock().unwrap()).contains("ALLOWED"));
capture.0.lock().unwrap().clear();
// Exercise the HTTP engine and DNS lookup as well as the proxy's own logs.
let url = format!("http://localhost:{port}/private-path?token=secret");
let response = app.clone().oneshot(send_request(&url)).await.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), 64 * 1024).await.unwrap();
assert!(String::from_utf8_lossy(&body).contains("\"type\":\"done\""));
assert_eq!(
app.oneshot(send_request(&url)).await.unwrap().status(),
StatusCode::TOO_MANY_REQUESTS
);
config.allow_private_networks = false;
let app = router(config.clone());
assert_eq!(
app.oneshot(send_request("http://127.0.0.1/private-path")).await.unwrap().status(),
StatusCode::FORBIDDEN
);
config.max_concurrent = 0;
assert_eq!(
router(config).oneshot(send_request(&url)).await.unwrap().status(),
StatusCode::SERVICE_UNAVAILABLE
);
upstream.abort();
let logs = capture.0.lock().unwrap();
assert!(logs.is_empty(), "Unexpected request logs: {}", String::from_utf8_lossy(&logs));
}

89
crates-server/yaak-web/tests/router.rs Normal file
View File

@@ -0,0 +1,89 @@
use axum::body::{Body, to_bytes};
use axum::extract::Request;
use axum::http::{HeaderValue, StatusCode, header};
use axum::middleware::{self, Next};
use axum::response::Response;
use std::path::PathBuf;
use tower::ServiceExt;
use yaak_web::{Config, router};
fn config(serve: Option<PathBuf>) -> Config {
Config {
host: "127.0.0.1".to_string(),
port: 0,
app_port: None,
serve,
allow_private_networks: false,
allowed_origins: vec!["*".into()],
max_request_bytes: 1024,
max_response_bytes: 2048,
max_timeout_secs: 5,
rate_limit_per_minute: 10,
max_concurrent: 2,
trust_forwarded_for: false,
}
}
async fn mark_response(request: Request, next: Next) -> Response {
let mut response = next.run(request).await;
response.headers_mut().insert("x-hosted-middleware", HeaderValue::from_static("yes"));
response
}
#[tokio::test]
async fn embedded_router_serves_api_without_static_files() {
let app = router(config(None)).layer(middleware::from_fn(mark_response));
let response = app
.clone()
.oneshot(Request::builder().uri("/v1/health").body(Body::empty()).unwrap())
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(response.headers()["x-hosted-middleware"], "yes");
let body = to_bytes(response.into_body(), 4096).await.unwrap();
let health: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(health["ok"], true);
assert_eq!(health["maxResponseBytes"], 2048);
assert_eq!(health["maxTimeoutSecs"], 5);
let response = app.oneshot(Request::new(Body::empty())).await.unwrap();
assert_eq!(response.status(), StatusCode::NOT_FOUND);
}
#[tokio::test]
async fn middleware_wraps_html_fallback_and_assets_alongside_api() {
let dir = tempfile::tempdir().unwrap();
std::fs::write(dir.path().join("index.html"), "<html>Yaak</html>").unwrap();
std::fs::create_dir(dir.path().join("assets")).unwrap();
std::fs::write(dir.path().join("assets/app-abc.js"), "// app").unwrap();
let app = router(config(Some(dir.path().into()))).layer(middleware::from_fn(mark_response));
for path in [
"/",
"/workspace/example",
"/assets/app-abc.js",
"/v1/health",
] {
let response = app
.clone()
.oneshot(Request::builder().uri(path).body(Body::empty()).unwrap())
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK, "{path}");
assert_eq!(response.headers()["x-hosted-middleware"], "yes", "{path}");
if path.starts_with("/assets/") {
assert_eq!(
response.headers()[header::CACHE_CONTROL],
"public, max-age=31536000, immutable"
);
} else if path == "/v1/health" {
assert_eq!(response.headers()[header::CONTENT_TYPE], "application/json");
} else {
assert_eq!(response.headers()[header::CACHE_CONTROL], "no-cache");
assert!(
response.headers()[header::CONTENT_TYPE].to_str().unwrap().starts_with("text/html")
);
assert_eq!(to_bytes(response.into_body(), 4096).await.unwrap(), "<html>Yaak</html>");
}
}
}